Ubuntu application layer support The Firefly-RK3288 Ubuntu Desktop system is based on the Ubuntu 32bit system and currently maintains the Ubuntu 18.04 version. Has the following characteristics: The desktop environment uses LXDE, Lubuntu + Firefly custom theme, simple and beautiful. Xserver uses GPU + RGA for 2D acceleration, runs smoothly, and takes up less CPU resources. Streamline system services for embedded platforms. Provide OpenGL and OpenCL support based on Arm Mali GPU. Provide video hard codec support based on Rockchip VPU + Mpp. Adapt to QT, Docker, Electron and other development frameworks. Provide a series of interfaces to operate onboard resource equipment. Basic information User and password After the Firefly Linux Desktop system starts up, it automatically logs in to the firefly user. If a debugging serial port is connected, the serial terminal automatically logs in to the root user. firefly user password: firefly root user password: firefly Video hardware codec support RK3288 integrated VPU has excellent video encoding and decoding capabilities, Mpp is a set of video encoding and decoding APIs provided by Rockchip for VPU, and is based on mpp. Rockchip provides a set of gstreamer codec plug-ins. According to their own needs, users can use gstreamer to do video codec applications, or directly call mpp to achieve hardware codec acceleration. The system provides a test video file, located in "/usr/local/test.mp4", the test file is 1080P, 24Fps, H264 encoding, Mp4 format. There are several ways to verify and develop video codec related applications. Gstreamer Ubuntu 16.04, gstreamer 1.12 Already installed in the /opt/ directory Ubuntu 18.04, gstreamer 1.12 Already installed in the system "/usr/local/bin/h264dec.sh" Test H264 decoding "/usr/local/bin/h264enc.sh" Test H264 coding Users can refer to these two scripts to configure their own gstreamer application. Mpv The Mpv player provided by the system can directly call the rkmpp decoding plug-in. FFmpeg FFmpeg only supports hardware decoding through Mpp for Rockchip for the time being, there is no support for hardware encoding for the time being. The Ubuntu system released by Firefly has already installed FFmpeg, and users can use it directly. Confirm rkmpp decoder $ ffmpeg -decoders | grep "rkmpp" V..... h264_rkmpp h264 (rkmpp) (codec h264) V..... hevc_rkmpp hevc (rkmpp) (codec hevc) V..... vp8_rkmpp vp8 (rkmpp) (codec vp8) V..... vp9_rkmpp vp9 (rkmpp) (codec vp9) Test command $ ffmpeg -y -c:v h264_rkmpp -i test.mp4 -an -vf \ hwdownload,format=nv12,format=yuv420p output.yuv **pay attention:**FFmpeg h264_rkmpp decodes AV_PIX_FMT_DRM_PRIME, which is DRM frame data, if it is based on drm display, you can directly output the frame, otherwise, you need to use hwdownload to convert. More content can refer to FFmpeg official website . Mpp Under Ubuntu system, mpp related dev packages have been installed in the system. For more related information, please refer to the relevant documents under `linux-sdk/docs/Linux/Multimedia`. OpenGL-ES RK3288 support OpenGL ES1.1/2.0/3.0/3.1。 The Ubuntu system released by Firefly has provided complete OpenGL-ES support. Test command $ sudo test_glmark2_normal.sh webGL support In the Chromium browser, type: "chrome://gpu" in the address bar to see hardware acceleration support. Note: EGL is an extension of the OpenGL for the x window system on the arm platform, which is equivalent to the glx library under x86. Since the driver mode setting used by Xorg will load libglx.so by default (disabling glx will cause some applications failing to detect the glx environment), libglx.so will search the dri implementation library in the system. However, rk3288 Xorg 2D acceleration is directly based on DRM, and does not implement dri library, so libglx.so will report the following error during startup. (EE) AIGLX error: dlopen of /usr/lib/arm-linux-gnueabihf/dri/rockchip_dri.so failed This has no effect on system operation and does not need to be processed. Based on the same reason, some applications will report the following errors during startup, without processing, and will not affect the operation of the application. libGL error: unable to load driver: rockchip_dri.so libGL error: driver pointer missing libGL error: failed to load driver: rockchip Some versions of Ubuntu released by Firefly have turned off loading libglx.so by default. In some cases, some applications will run the following error. GdkGLExt-WARNING **: Window system doesn't support OpenGL. The method of correction is as follows： Delete /etc/X11/xorg.conf.d/20-modesetting.conf three-line configuration。 Section "Module" Disable "glx" EndSection OpenCL The Ubuntu system released by Firefly has added opencl1.2 support, and can run the system's built-in "clinfo" to get the platform opencl related parameters. firefly@firefly:~$ clinfo Number of platforms 1 Platform Name ARM Platform Platform Vendor ARM Platform Version OpenCL 1.2 v1.r14p0-01rel0-git(a79caef).004e51a1e8b44487ebc4c7bfea5f60ee Platform Profile FULL_PROFILE Platform Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atoy Platform Extensions function suffix ARM Platform Name ARM Platform Number of devices 1 Device Name Mali-T760 Device Vendor ARM Device Vendor ID 0x7501000 Device Version OpenCL 1.2 v1.r14p0-01rel0-git(a79caef).004e51a1e8b44487ebc4c7bfea5f60ee Driver Version 1.2 Device OpenCL C Version OpenCL C 1.2 v1.r14p0-01rel0-git(a79caef).004e51a1e8b44487ebc4c7bfea5f60ee Device Type GPU … Screen rotation The Ubuntu system released by Firefly, if you need to rotate the display direction of the system by default, you can modify the direction of the corresponding display device in "/etc/default/xrandr". firefly@firefly:~$ cat /etc/default/xrandr #!/bin/sh # Rotation can be one of 'normal', 'left', 'right' or 'inverted'. # xrandr --output HDMI-1 --rotate normal # xrandr --output LVDS-1 --rotate normal # xrandr --output EDP-1 --rotate normal # xrandr --output MIPI-1 --rotate normal # xrandr --output VGA-1 --rotate normal # xrandr --output DP-1 --rotate normal For platforms with a touch screen, if you need to rotate the direction of the touch screen, you can modify the SwapAxes / InvertX / InvertY values in "/etc/X11/xorg.conf.d/05-gslX680.conf". firefly@firefly:~$ cat /etc/X11/xorg.conf.d/05-gslX680.conf Section "InputClass" Identifier "gslX680" MatchIsTouchscreen "on" MatchProduct "gslX680" Driver "evdev" Option "SwapAxes" "off" # Invert the respective axis. Option "InvertX" "off" Option "InvertY" "off" EndSection Virtual keyboard In the Ubuntu system released by Firefly, you can execute the "onboard" at the menu to open a virtual keyboard. Sound setting The Firefly-RK3288 board generally has two or more audio devices. Headphone and HDMI are two common audio devices. Below is an example of audio settings for users to refer to. Specify audio devices in terminal Audio files in the system are in "/usr/share/sound/alsa", please check your sound card before play them: root@firefly:~# aplay -l **** List of PLAYBACK Hardware Devices **** card 0: rockchipfirefly [rockchip,firefly-codec], device 0: ff890000.i2s-ES8323 HiFi ES8323 HiFi-0 [] Subdevices: 1/1 Subdevice #0: subdevice #0 card 0: rockchipfirefly [rockchip,firefly-codec], device 1: ff890000.i2s-i2s-hifi i2s-hifi-1 [] Subdevices: 1/1 Subdevice #0: subdevice #0 And then specify a sound card to play an audio file. Among them, card0-device0 stands for headphone and card0-device1 stands for HDMI. We usually run command "aplay -Dhw:0,0 Fornt_Center.wav" to play an audio file, but in order to avoid playback failure because the audio files in the system are mono, so you can run as follow: #choose headphone to play root@firefly:/usr/share/sounds/alsa# aplay -Dplughw:0,0 Front_Center.wav Playing WAVE 'Front_Center.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono #choose HDMI to play root@firefly:/usr/share/sounds/alsa# aplay -Dplughw:0,1 Front_Center.wav Playing WAVE 'Front_Center.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono Start-up applications while booting up In the system, the application program can be set up automatically according to the user's needs. Select in turn from the System Menu Bar, "Preferences" -> "Default applications for LXSession", open the setting interface, and you can set the default opening mode of the application in "Launching applocations". In the "Autostart" column, you can select applications that you need to boot. For example, Bluetooth does not start by default. If it needs to start automatically, it can start Bluetooth automatically by ticking the "Blueman Applet". Serial port There are two types of serial ports on the development board: the normal ones that are built-in SoC UARTs, and the extended ones that are converted by SPI. The use of the converted serial port is exactly the same as of the normal one, but with different device file naming: # uart root@firefly:~# ls /dev/ttyS* ttyS0 ttyS1 ttyS2 ttyS3 # spi to uart root@firefly:~# ls /dev/ttysWK* ttysWK0 ttysWK1 ttysWK2 ttysWK3 Set baud rate Take "ttyS4" as an example below. To view the serial port baud rate setting, run: root@firefly:~# stty -F /dev/ttyS4 speed 9600 baud; line = 0; -brkint -imaxbel Set baud rate: # Set the baud rate to 115200 root@firefly:~# stty -F /dev/ttyS4 ospeed 115200 ispeed 115200 cs8 # Confirm if it has been modified root@firefly:~# stty -F /dev/ttyS4 speed 115200 baud; line = 0; -brkint -imaxbel Turn off echo Echo is useful in an interative serial terminal session but will affect the result of data transfer, for example, in a loopback test. To turn off echo, run: # Close echo root@firefly:~# stty -F /dev/ttyS4 -echo -echoe -echok # Check the configuration of all features and check if it is closed. The "-" means the feature is off root@firefly:~# stty -F /dev/ttyS4 -a | grep echo isig icanon iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc Send and receive raw data If the serial port is not in raw mode, there may be differences between the data sent and received (For example, the driver may do some line end converting, which is no good to binary data). Use "stty" to set it in "raw" mode to ensure that the same data is exactly sent and received: root@firefly:~# stty -F /dev/ttyS4 raw Operating mode The serial port has two operating modes: interrupt mode and DMA mode. Interrupt mode By default, the kernel configures the serial port into interrupt mode so there is no need to make any change. In interrupt mode, the transfer latency is low, but it tends to lose data in bulk data transfer. Therefore do not use interrupt mode in that case. DMA mode The DMA mode is mainly used when transferring a large amount of data. The kernel will provide a buffer space for the serial port to receive data to minimize the packet loss rate of the serial port transmission. WARNNING: The default size of the cache space is "8K". If the transfer exceeds the cache size, the packet will be lost. Therefore, if the DMA mode is used, the sender needs to send multiple packets. DTS setting: &uart4 { status = "okay"; + dmas = <&dmac_peri 8>, <&dmac_peri 9>; + dma-names = "tx", "rx"; }; DMA mode does not increase the transfer rate. Instead, the transfer rate is reduced because of the need for caching, so do not use DMA mode if you do not need to transfer large amounts of data. Flow control No matter whether it is interrupt mode or DMA mode, data transmission is not foolproof, because DDR, CPU frequency conversion or high occupancy may cause processing data not to be timely when transferring large amounts of data . At this time, it is necessary to use flow control. There are two types of flow control, one is software flow control and the other is hardware flow control. The following only describes the use of hardware flow control. Hardware support Hardware flow control requires hardware support. The "CTX" and "RTX" pins of the development board need to be connected to the device. WARNNING: Not all serial ports on the development board support hardware flow control. Please confirm the hardware support from the schematic. DTS setting uart4: serial@ff1c0000 { compatible = "rockchip,rk3288-uart", "snps,dw-apb-uart"; reg = <0x0 0xff1c0000 0x0 0x100>; interrupts = ; reg-shift = <2>; reg-io-width = <4>; clocks = <&cru SCLK_UART4>, <&cru PCLK_UART4>; clock-names = "baudclk", "apb_pclk"; pinctrl-names = "default"; - pinctrl-0 = <&uart4_xfer>; + pinctrl-0 = <&uart4_xfer &uart4_cts &uart4_rts>; status = "disabled"; }; WARNNING: Not all serial ports on the development board support hardware flow control. Please confirm the hardware support from the schematic. Application settings The application also needs to turn on flow control. Here is how to do that with "stty": # Turn on flow control root@firefly:~# stty -F /dev/ttyS4 crtscts # Check if the flow control is turned on, the "-" means the function has been turned off. root@firefly:~# stty -F /dev/ttyS4 -a | grep crtscts -parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal crtscts MIPI Camera (OV13850) The RK3288 board has added support for the MIPI camera "OV13850", and examples of cameras have also been added to the application. The following describes the relevant configuration. DTS setting Take "rk3288-firefly-aio.dts" as an example, these are the configurations that the camera needs to open. // power &vcc_mipi { status = "okay"; }; &dvdd_1v2 { status = "okay"; }; // mipi camera0 &ov13850 { status = "okay"; }; &mipi_phy_tx1rx1 { status = "okay"; }; // mipi camera1 /*&ov13850_1 { status = "okay"; }; &mipi_phy_rx0 { status = "okay"; };*/ // ISP &rkisp1 { status = "okay"; }; &isp_mmu { status = "okay"; }; Note: RK3288 only supports one ISP, so the two cameras cannot be used at the same time; for different RK3288 development boards, the corresponding GPIO properties may need to be modified. DEBUG Check whether the ISP driver is loaded successfully, and you can see the following device information: root@firefly:~# grep '' /sys/class/video4linux/video*/name /sys/class/video4linux/video0/name:rkisp1_mainpath /sys/class/video4linux/video1/name:rkisp1_selfpath /sys/class/video4linux/video2/name:rkisp1_dmapath /sys/class/video4linux/video3/name:rkisp1-statistics /sys/class/video4linux/video4/name:rkisp1-input-params root@firefly:~# ls /dev/video* video0 video1 video2 video3 video4 Check if the "OV13850" device is registered successfully. If successful, you can see the following print: root@firefly:~# dmesg |grep 13850 [ 1.113850] NET: Registered protocol family 2 [ 3.911130] ov13850 3-0010: driver version: 00.01.03 [ 3.911168] ov13850 3-0010: GPIO lookup for consumer reset [ 3.911171] ov13850 3-0010: using device tree for GPIO lookup [ 3.911187] of_get_named_gpiod_flags: parsed 'reset-gpios' property of node '/i2c@ff150000/ov13850@10[0]' - status (0) [ 3.911370] ov13850 3-0010: could not get default pinstate [ 3.911373] ov13850 3-0010: could not get sleep pinstate [ 3.911376] ov13850 3-0010: GPIO lookup for consumer pwdn [ 3.911379] ov13850 3-0010: using device tree for GPIO lookup [ 3.911403] of_get_named_gpiod_flags: parsed 'pwdn-gpios' property of node '/i2c@ff150000/ov13850@10[0]' - status (0) // Device ID successfully obtained [ 3.914401] ov13850 3-0010: Detected OV00d850 sensor, REVISION 0xb2 Test You can run the following script to test: #!/bin/bash export DISPLAY=:0 export XAUTHORITY=/home/firefly/.Xauthority WIDTH=640 HEIGHT=480 SINK=gtksink gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,width=${WIDTH},height=${HEIGHT}, framerate=30/1 ! videoconvert ! $SINK & wait The result is shown in the figure: Export root filesystem This section describes how to export and package the root file system on a development board into an image file. Suitable for copying the environment of a certain development board to other development boards, or making upgrade firmware. Steps Install ssh, rsync on the development board and PC: sudo apt-get install -y ssh rsync Set up ssh on the development board to allow root user access: Modify #PermitRootLogin prohibit-password in /etc/ssh/sshd_config file to PermitRootLogin yes Restart the ssh service: sudo /etc/init.d/ssh restart To synchronize the filesystem of the development board in the PC, please put the development board and the PC in the same local area network first. (Assuming the development board address is 192.168.100.100) mkdir rootfs sudo rsync -avx root@192.168.100.100:/ ./rootfs At this point, all the files in the filesystem of the development board are saved in the rootfs directory, and you can start to make the root filesystem image: dd if=/dev/zero of=linuxroot.img bs=1M count=4000 mkfs.ext4 linuxroot.img mkdir temp && sudo mount linuxroot.img ./temp sudo cp -rfp rootfs/* temp/ sudo umount temp/ e2fsck -p -f linuxroot.img resize2fs -M linuxroot.img Finally, linuxroot.img is the finished root filesystem image, which is consistent with the filesystem of your development board. Compile the universal device bus driver locally DEBIAN package download link: https://drive.google.com/drive/folders/1GBl2n66fARqJLVEZ- LeW3bfbs3tyOW3C?usp=sharing Install header files: sudo dpkg -i linux-headers-4.4.194_4.4.194-93_armhf.deb sudo dpkg -i linux-image-4.4.194_4.4.194-93_armhf.deb cd /usr/src/linux-headers-4.4.194 make headers_check make headers_install # make scripts may go wrong but it doesn't matter make scripts