4. Software Development Related¶
4.1. Qt cross compilation environment support¶
Firefly released two Qt cross-compilation tool chains, suitable for the following environments:
Host: x86-64 / Ubuntu 18.04
Target: Firefly RK3568 RK3566 RK3399 RK3328 PX30 / Ubuntu 18.04 Minimal&Desktop
Host: x86-64 / Ubuntu 20.04
Target: Firefly RK3568 RK3566 / Ubuntu 20.04 Desktop
The tool chain fully supports wenEngine and backends such as EGLFS LinuxFB XCB.
See Qt5.1x.x_Release.md in tool-chain package for details
Note that the names of all paths in the document cannot be changed, otherwise it will cause compilation or running errors.
On the host side, enter the Qt project directory,
qmake && make.
Two test demos are provided in the tool chain, corresponding to EGLFS and LinuxFB Backend. After the deployment is completed, the user can build the demo on the host side and run the demo on the tartget side to test whether the deployment is successful.
After deciding which backend to use, you can modify the /etc/profile.d/target_qtEnv.sh in target, uncomment the environment variables of corresponding backend to keep it in effect.
# For example, using XCB, then uncomment the XCB part #XCB export QT_QPA_PLATFORM=XCB export QT_QPA_EGLFS_INTEGRATION=XCB_EGL
4.2. Video hardware codec support¶
The integrated VPU of RK3568/RK3566/RK3399/RK3288 has excellent video codec capabilities. Mpp is a set of video codec APIs provided by Rockchip for VPU, and is based on mpp. Rockchip provides a set of gstreamer codec plug-ins. Users can do video codec applications based on gstreamer according to their needs, or directly call mpp to achieve hardware codec acceleration.
The system provides a test video file located at
/usr/local/test.mp4, the test file is 1080P, 24Fps, H264 encoding, Mp4 format.
There are several ways to verify and develop video codec related applications.
Under Ubuntu 16.04, gstreamer has been installed in the
Under Ubuntu 18.04 and 20.04, gstreamer has been installed in the system.
/usr/local/bin/h264dec.sh Test hardware H264 decoding.
/usr/local/bin/h264enc.sh Test hardware H264 encoding.
Users can refer to these two scripts to configure their own gstreamer application.
The Mpv player provided by the system can directly call the rkmpp decoding plug-in.
FFmpeg only supports hardware decoding through Mpp for Rockchip temporarily, and there is no hardware encoding support for the time being.
Firefly Ubuntu has installed FFmpeg, and users can use it directly.
Confirm rkmpp decoder
$ ffmpeg -decoders | grep "rkmpp" V..... h264_rkmpp h264 (rkmpp) (codec h264) V..... hevc_rkmpp hevc (rkmpp) (codec hevc) V..... vp8_rkmpp vp8 (rkmpp) (codec vp8) V..... vp9_rkmpp vp9 (rkmpp) (codec vp9)
$ ffmpeg -y -c:v h264_rkmpp -i /usr/local/test.mp4 -an output.yuv
Special attention: FFmpeg h264_rkmpp decodes AV_PIX_FMT_DRM_PRIME, which is DRM frame data. If it is based on drm display, you can directly output the frame, otherwise, you need to use hwdownload to convert.
For more information, please refer to FFmpeg official website.
Under Ubunut system, mpp related dev packages have been installed in the system.
For more related information, please refer to the relevant documents under
RK3568/RK3566/RK3399/RK3288 supports OpenGL ES1.1/2.0/3.0/3.1.
Firefly Ubuntu has provided complete OpenGL-ES support.
$ sudo test_glmark2_normal.sh
In the Chromium browser, type in the address bar:
chrome://gpu to view the hardware acceleration support under Chromium.
EGL is an extension of OpenGL on the arm platform for the x window system, and its function is equivalent to the glx library under x86.
Because the Driver modesettings used by Xorg will load libglx.so by default (disabling glx will cause some applications that detect glx environment to fail to start), libglx.so will search for the dri implementation library in the system. However, RK3568/RK3566/RK3399/RK3288 Xorg 2D acceleration is implemented directly based on DRM and does not implement the dri library, so during the startup process, libglx.so will report the following error.
(EE) AIGLX error: dlopen of /usr/lib/aarch64-linux-gnu/dri/rockchip_dri.so failed
This has no effect on the operation of the system and does not need to be processed.
Based on the same reasoning, some applications will report the following errors during the startup process, and there is no need to deal with them, which will not affect the operation of the application.
libGL error: unable to load driver: rockchip_dri.so libGL error: driver pointer missing libGL error: failed to load driver: rockchip
In some versions of Ubuntu software previously released by Firefly, the loading of libglx.so is disabled by default. In some cases, the following errors will occur when running certain applications:
GdkGLExt-WARNING **: Window system doesn't support OpenGL.
The correction method is as follows:
Delete the following three lines of configuration in
Section "Module" Disable "glx" EndSection
Firefly Ubuntu has added opencl1.2 support, you can run the built-in
clinfo of the system to get platform opencl related parameters.
firefly@firefly:~$ clinfo Platform #0 Name: ARM Platform Version: OpenCL 1.2 v1.r14p0-01rel0-git(966ed26).f44c85cb3d2ceb87e8be88e7592755c3 Device #0 Name: Mali-T860 Type: GPU Version: OpenCL 1.2 v1.r14p0-01rel0-git(966ed26).f44c85cb3d2ceb87e8be88e7592755c3 Global memory size: 1 GB 935 MB 460 kB Local memory size: 32 kB Max work group size: 256 Max work item sizes: (256, 256, 256) …
4.5. TensorFlow Lite¶
RK3568/RK3566/RK3399 supports the neural network GPU acceleration solution LinuxNN, and Firefly Ubuntu has added LinuxNN support.
test.sh to test the Demo of the MobileNet model image classifier and the Target Detection Demo of the MobileNet-SSD model
firefly@firefly:/opt/tensorflowbin$ ./test.sh Loaded model mobilenet_ssd.tflite resolved reporter nn version: 1.0.0 findAvailableDevices filename:libarmnn-driver.so d_info:40432 d_reclen:40s [D][ArmnnDriver]: Register Service: armnn (version: 1.0.0)! first invoked time: 1919.17 ms invoked average time: 108.4 ms validCount: 26 car @ (546, 501) (661, 586) car @ (1, 549) (51, 618) person @ (56, 501) (239, 854) person @ (332, 530) (368, 627) person @ (391, 541) (434, 652) person @ (418, 477) (538, 767) person @ (456, 487) (602, 764) car @ (589, 523) (858, 687) person @ (826, 463) (1034, 873) bicycle @ (698, 644) (1128, 925) write out.jpg succ!
Platform limitation: Currently only 3399 and 3399PRO are supported.
ffmedia is a video codec framework developed based on Rockchip Mpp/RGA. Audio is currently not supported. ffmedia includes the following units:
Camera: Support UVC, Mipi CSI
File Reader：Support H264, H265 bare stream file reading
MppDec: Video decoding, support H264, H265, MJpeg
MppEnc: Video encoding, support H264, H265
RGA: Image scaling, cropping, format conversion
DRM Display: libdrm-based display module
Mp4 Enmux: The encoded data is output as a file in Mp4 format
4.6.2. Compile & Test¶
# Obtain the source code according to the operating system git clone https://gitlab.com/firefly-linux/ff_media -b ubuntu20.04 # Install the required environment apt install gcc g++ make cmake apt install libasound2-dev apt install libopencv-dev # Compile cd ff_media mkdir build; cd build cmake ../ make # Run test demo ## Run the demo to view the help information INFO: usage: Usage: ./demo <Input source> [Options] Options: -i, --input Input image size -o, --output Output image size, default same as input -a, --inputfmt Input image format, default MJPEG -b, --outputfmt Output image format, default NV12 -c, --count Instance count, default 1 -d, --drmdisplay Drm display, set display plane, set 0 to auto find plane, default disabled -e, --encodetype Encode encode, set encode type, default disabled -f, --file Enable save dec output data to file, set filename, default disabled -p, --port Enable rtsp stream, set push port, depend on encode enabled, default disabled -m, --enmux Enable save encode data to mp4 file, depend on encode enabled, default disabled -r, --rotate Image rotation degree, default 0 0: none 1: vertical mirror 2: horizontal mirror 90: 90 degree 180: 180 degree 270: 270 degree ## Demonstration: The input is an rtsp camera with a resolution of 1080p, the decoded image is scaled to 720p and rotated 90 degrees, and the output is displayed on the monitor. ./demo rtsp://admin:firstname.lastname@example.org -o 1280x720 -d 0 -r 90