Ubuntu application layer support¶
The Firefly-RK3288 Ubuntu Desktop system is based on the Ubuntu 32bit system and currently maintains the Ubuntu 18.04 version. Has the following characteristics:
The desktop environment uses LXDE, Lubuntu + Firefly custom theme, simple and beautiful.
Xserver uses GPU + RGA for 2D acceleration, runs smoothly, and takes up less CPU resources.
Streamline system services for embedded platforms.
Provide OpenGL and OpenCL support based on Arm Mali GPU.
Provide video hard codec support based on Rockchip VPU + Mpp.
Adapt to QT, Docker, Electron and other development frameworks.
Provide a series of interfaces to operate onboard resource equipment.
Basic information¶
User and password¶
After the Firefly Linux Desktop system starts up, it automatically logs in to the firefly user.
If a debugging serial port is connected, the serial terminal automatically logs in to the root user.
firefly user password: firefly
root user password: firefly
Video hardware codec support¶
RK3288 integrated VPU has excellent video encoding and decoding capabilities, Mpp is a set of video encoding and decoding APIs provided by Rockchip for VPU, and is based on mpp. Rockchip provides a set of gstreamer codec plug-ins. According to their own needs, users can use gstreamer to do video codec applications, or directly call mpp to achieve hardware codec acceleration.
The system provides a test video file, located in /usr/local/test.mp4
, the test file is 1080P, 24Fps, H264 encoding, Mp4 format.
There are several ways to verify and develop video codec related applications.
Gstreamer¶
Ubuntu 16.04, gstreamer 1.12 Already installed in the
/opt/
directoryUbuntu 18.04, gstreamer 1.12 Already installed in the system
/usr/local/bin/h264dec.sh
Test H264 decoding
/usr/local/bin/h264enc.sh
Test H264 coding
Users can refer to these two scripts to configure their own gstreamer application.
Mpv¶
The Mpv player provided by the system can directly call the rkmpp decoding plug-in.
FFmpeg¶
FFmpeg only supports hardware decoding through Mpp for Rockchip for the time being, there is no support for hardware encoding for the time being.
The Ubuntu system released by Firefly has already installed FFmpeg, and users can use it directly.
Confirm rkmpp decoder
$ ffmpeg -decoders | grep "rkmpp"
V..... h264_rkmpp h264 (rkmpp) (codec h264)
V..... hevc_rkmpp hevc (rkmpp) (codec hevc)
V..... vp8_rkmpp vp8 (rkmpp) (codec vp8)
V..... vp9_rkmpp vp9 (rkmpp) (codec vp9)
Test command
$ ffmpeg -y -c:v h264_rkmpp -i test.mp4 -an -vf \
hwdownload,format=nv12,format=yuv420p output.yuv
**pay attention:**FFmpeg h264_rkmpp decodes AV_PIX_FMT_DRM_PRIME, which is DRM frame data, if it is based on drm display, you can directly output the frame, otherwise, you need to use hwdownload to convert.
More content can refer toFFmpeg official website.
Mpp¶
Under Ubuntu system, mpp related dev packages have been installed in the system.
For more related information, please refer to the relevant documents under `linux-sdk/docs/Linux/Multimedia`.
OpenGL-ES¶
RK3288 support OpenGL ES1.1/2.0/3.0/3.1。
The Ubuntu system released by Firefly has provided complete OpenGL-ES support.
Test command
$ sudo test_glmark2_normal.sh
webGL support
In the Chromium browser, type: chrome://gpu
in the address bar to see hardware acceleration support.
Note:
EGL is an extension of the OpenGL for the x window system on the arm platform, which is equivalent to the glx library under x86.
Since the driver mode setting used by Xorg will load libglx.so by default (disabling glx will cause some applications failing to detect the glx environment), libglx.so will search the dri implementation library in the system. However, rk3288 Xorg 2D acceleration is directly based on DRM, and does not implement dri library, so libglx.so will report the following error during startup.
(EE) AIGLX error: dlopen of /usr/lib/arm-linux-gnueabihf/dri/rockchip_dri.so failed
This has no effect on system operation and does not need to be processed.
Based on the same reason, some applications will report the following errors during startup, without processing, and will not affect the operation of the application.
libGL error: unable to load driver: rockchip_dri.so libGL error: driver pointer missing libGL error: failed to load driver: rockchip
Some versions of Ubuntu released by Firefly have turned off loading libglx.so by default. In some cases, some applications will run the following error.
GdkGLExt-WARNING **: Window system doesn't support OpenGL.
The method of correction is as follows:
Delete
/etc/X11/xorg.conf.d/20-modesetting.conf
three-line configuration。Section "Module" Disable "glx" EndSection
OpenCL¶
The Ubuntu system released by Firefly has added opencl1.2 support, and can run the system’s built-in clinfo
to get the platform opencl related parameters.
firefly@firefly:~$ clinfo
Number of platforms 1
Platform Name ARM Platform
Platform Vendor ARM
Platform Version OpenCL 1.2 v1.r14p0-01rel0-git(a79caef).004e51a1e8b44487ebc4c7bfea5f60ee
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atoy
Platform Extensions function suffix ARM
Platform Name ARM Platform
Number of devices 1
Device Name Mali-T760
Device Vendor ARM
Device Vendor ID 0x7501000
Device Version OpenCL 1.2 v1.r14p0-01rel0-git(a79caef).004e51a1e8b44487ebc4c7bfea5f60ee
Driver Version 1.2
Device OpenCL C Version OpenCL C 1.2 v1.r14p0-01rel0-git(a79caef).004e51a1e8b44487ebc4c7bfea5f60ee
Device Type GPU
…
Screen rotation¶
The Ubuntu system released by Firefly, if you need to rotate the display direction of the system by default, you can modify the direction of the corresponding display device in /etc/default/xrandr
.
firefly@firefly:~$ cat /etc/default/xrandr
#!/bin/sh
# Rotation can be one of 'normal', 'left', 'right' or 'inverted'.
# xrandr --output HDMI-1 --rotate normal
# xrandr --output LVDS-1 --rotate normal
# xrandr --output EDP-1 --rotate normal
# xrandr --output MIPI-1 --rotate normal
# xrandr --output VGA-1 --rotate normal
# xrandr --output DP-1 --rotate normal
For platforms with a touch screen, if you need to rotate the direction of the touch screen, you can modify the SwapAxes / InvertX / InvertY values in /etc/X11/xorg.conf.d/05-gslX680.conf
.
firefly@firefly:~$ cat /etc/X11/xorg.conf.d/05-gslX680.conf
Section "InputClass"
Identifier "gslX680"
MatchIsTouchscreen "on"
MatchProduct "gslX680"
Driver "evdev"
Option "SwapAxes" "off"
# Invert the respective axis.
Option "InvertX" "off"
Option "InvertY" "off"
EndSection
Virtual keyboard¶
In the Ubuntu system released by Firefly, you can execute the onboard
at the menu to open a virtual keyboard.
Sound setting¶
The Firefly-RK3288 board generally has two or more audio devices. Headphone and HDMI are two common audio devices. Below is an example of audio settings for users to refer to.
Specify audio devices in terminal¶
Audio files in the system are in /usr/share/sound/alsa
, please check your sound card before play them:
root@firefly:~# aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: rockchipfirefly [rockchip,firefly-codec], device 0: ff890000.i2s-ES8323 HiFi ES8323 HiFi-0 []
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: rockchipfirefly [rockchip,firefly-codec], device 1: ff890000.i2s-i2s-hifi i2s-hifi-1 []
Subdevices: 1/1
Subdevice #0: subdevice #0
And then specify a sound card to play an audio file. Among them, card0-device0 stands for headphone and card0-device1 stands for HDMI. We usually run command aplay -Dhw:0,0 Fornt_Center.wav
to play an audio file, but in order to avoid playback failure because the audio files in the system are mono, so you can run as follow:
#choose headphone to play
root@firefly:/usr/share/sounds/alsa# aplay -Dplughw:0,0 Front_Center.wav
Playing WAVE 'Front_Center.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono
#choose HDMI to play
root@firefly:/usr/share/sounds/alsa# aplay -Dplughw:0,1 Front_Center.wav
Playing WAVE 'Front_Center.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono
Start-up applications while booting up¶
In the system, the application program can be set up automatically according to the user’s needs.
Select in turn from the System Menu Bar, Preferences
-> Default applications for LXSession
, open the setting interface, and you can set the default opening mode of the application in Launching applocations
.
In the Autostart
column, you can select applications that you need to boot. For example, Bluetooth does not start by default. If it needs to start automatically, it can start Bluetooth automatically by ticking the Blueman Applet
.
Serial port¶
There are two types of serial ports on the development board: the normal ones that are built-in SoC UARTs, and the extended ones that are converted by SPI. The use of the converted serial port is exactly the same as of the normal one, but with different device file naming:
# uart
root@firefly:~# ls /dev/ttyS*
ttyS0 ttyS1 ttyS2 ttyS3
# spi to uart
root@firefly:~# ls /dev/ttysWK*
ttysWK0 ttysWK1 ttysWK2 ttysWK3
Set baud rate¶
Take ttyS4
as an example below. To view the serial port baud rate setting, run:
root@firefly:~# stty -F /dev/ttyS4
speed 9600 baud; line = 0;
-brkint -imaxbel
Set baud rate:
# Set the baud rate to 115200
root@firefly:~# stty -F /dev/ttyS4 ospeed 115200 ispeed 115200 cs8
# Confirm if it has been modified
root@firefly:~# stty -F /dev/ttyS4
speed 115200 baud; line = 0;
-brkint -imaxbel
Turn off echo¶
Echo is useful in an interative serial terminal session but will affect the result of data transfer, for example, in a loopback test. To turn off echo, run:
# Close echo
root@firefly:~# stty -F /dev/ttyS4 -echo -echoe -echok
# Check the configuration of all features and check if it is closed. The "-" means the feature is off
root@firefly:~# stty -F /dev/ttyS4 -a | grep echo
isig icanon iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt
echoctl echoke -flusho -extproc
Send and receive raw data¶
If the serial port is not in raw mode, there may be differences between the data sent and received (For example, the driver may do some line end converting, which is no good to binary data). Use stty
to set it in raw
mode to ensure that the same data is exactly sent and received:
root@firefly:~# stty -F /dev/ttyS4 raw
Operating mode¶
The serial port has two operating modes: interrupt mode and DMA mode.
Interrupt mode¶
By default, the kernel configures the serial port into interrupt mode so there is no need to make any change. In interrupt mode, the transfer latency is low, but it tends to lose data in bulk data transfer. Therefore do not use interrupt mode in that case.
DMA mode¶
The DMA mode is mainly used when transferring a large amount of data. The kernel will provide a buffer space for the serial port to receive data to minimize the packet loss rate of the serial port transmission.
WARNNING: The default size of the cache space is “8K”. If the transfer exceeds the cache size, the packet will be lost. Therefore, if the DMA mode is used, the sender needs to send multiple packets.
DTS setting:
&uart4 {
status = "okay";
+ dmas = <&dmac_peri 8>, <&dmac_peri 9>;
+ dma-names = "tx", "rx";
};
DMA mode does not increase the transfer rate. Instead, the transfer rate is reduced because of the need for caching, so do not use DMA mode if you do not need to transfer large amounts of data.
Flow control¶
No matter whether it is interrupt mode or DMA mode, data transmission is not foolproof, because DDR, CPU frequency conversion or high occupancy may cause processing data not to be timely when transferring large amounts of data . At this time, it is necessary to use flow control. There are two types of flow control, one is software flow control and the other is hardware flow control. The following only describes the use of hardware flow control.
Hardware support¶
Hardware flow control requires hardware support. The CTX
and RTX
pins of the development board need to be connected to the device.
WARNNING: Not all serial ports on the development board support hardware flow control. Please confirm the hardware support from the schematic.
DTS setting¶
uart4: serial@ff1c0000 {
compatible = "rockchip,rk3288-uart", "snps,dw-apb-uart";
reg = <0x0 0xff1c0000 0x0 0x100>;
interrupts = <GIC_SPI 59 IRQ_TYPE_LEVEL_HIGH>;
reg-shift = <2>;
reg-io-width = <4>;
clocks = <&cru SCLK_UART4>, <&cru PCLK_UART4>;
clock-names = "baudclk", "apb_pclk";
pinctrl-names = "default";
- pinctrl-0 = <&uart4_xfer>;
+ pinctrl-0 = <&uart4_xfer &uart4_cts &uart4_rts>;
status = "disabled";
};
WARNNING: Not all serial ports on the development board support hardware flow control. Please confirm the hardware support from the schematic.
Application settings¶
The application also needs to turn on flow control. Here is how to do that with stty
:
# Turn on flow control
root@firefly:~# stty -F /dev/ttyS4 crtscts
# Check if the flow control is turned on, the "-" means the function has been turned off.
root@firefly:~# stty -F /dev/ttyS4 -a | grep crtscts
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal crtscts
MIPI Camera (OV13850)¶
The RK3288 board has added support for the MIPI camera OV13850
, and examples of cameras have also been added to the application. The following describes the relevant configuration.
DTS setting¶
Take rk3288-firefly-aio.dts
as an example, these are the configurations that the camera needs to open.
// power
&vcc_mipi {
status = "okay";
};
&dvdd_1v2 {
status = "okay";
};
// mipi camera0
&ov13850 {
status = "okay";
};
&mipi_phy_tx1rx1 {
status = "okay";
};
// mipi camera1
/*&ov13850_1 {
status = "okay";
};
&mipi_phy_rx0 {
status = "okay";
};*/
// ISP
&rkisp1 {
status = "okay";
};
&isp_mmu {
status = "okay";
};
Note: RK3288 only supports one ISP, so the two cameras cannot be used at the same time; for different RK3288 development boards, the corresponding GPIO properties may need to be modified.
DEBUG¶
Check whether the ISP driver is loaded successfully, and you can see the following device information:
root@firefly:~# grep '' /sys/class/video4linux/video*/name
/sys/class/video4linux/video0/name:rkisp1_mainpath
/sys/class/video4linux/video1/name:rkisp1_selfpath
/sys/class/video4linux/video2/name:rkisp1_dmapath
/sys/class/video4linux/video3/name:rkisp1-statistics
/sys/class/video4linux/video4/name:rkisp1-input-params
root@firefly:~# ls /dev/video*
video0 video1 video2 video3 video4
Check if the OV13850
device is registered successfully. If successful, you can see the following print:
root@firefly:~# dmesg |grep 13850
[ 1.113850] NET: Registered protocol family 2
[ 3.911130] ov13850 3-0010: driver version: 00.01.03
[ 3.911168] ov13850 3-0010: GPIO lookup for consumer reset
[ 3.911171] ov13850 3-0010: using device tree for GPIO lookup
[ 3.911187] of_get_named_gpiod_flags: parsed 'reset-gpios' property of node '/i2c@ff150000/ov13850@10[0]' - status (0)
[ 3.911370] ov13850 3-0010: could not get default pinstate
[ 3.911373] ov13850 3-0010: could not get sleep pinstate
[ 3.911376] ov13850 3-0010: GPIO lookup for consumer pwdn
[ 3.911379] ov13850 3-0010: using device tree for GPIO lookup
[ 3.911403] of_get_named_gpiod_flags: parsed 'pwdn-gpios' property of node '/i2c@ff150000/ov13850@10[0]' - status (0)
// Device ID successfully obtained
[ 3.914401] ov13850 3-0010: Detected OV00d850 sensor, REVISION 0xb2
Test¶
You can run the following script to test:
#!/bin/bash
export DISPLAY=:0
export XAUTHORITY=/home/firefly/.Xauthority
WIDTH=640
HEIGHT=480
SINK=gtksink
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,width=${WIDTH},height=${HEIGHT}, framerate=30/1 ! videoconvert ! $SINK &
wait
The result is shown in the figure:
Export root filesystem¶
This section describes how to export and package the root file system on a development board into an image file. Suitable for copying the environment of a certain development board to other development boards, or making upgrade firmware.
Steps¶
Install
ssh
,rsync
on the development board and PC:sudo apt-get install -y ssh rsync
Set up ssh on the development board to allow root user access:
Modify
#PermitRootLogin prohibit-password
in/etc/ssh/sshd_config
file toPermitRootLogin yes
Restart the ssh service:
sudo /etc/init.d/ssh restart
To synchronize the filesystem of the development board in the PC, please put the development board and the PC in the same local area network first. (Assuming the development board address is 192.168.100.100)
mkdir rootfs sudo rsync -avx root@192.168.100.100:/ ./rootfs
At this point, all the files in the filesystem of the development board are saved in the rootfs directory, and you can start to make the root filesystem image:
dd if=/dev/zero of=linuxroot.img bs=1M count=4000 mkfs.ext4 linuxroot.img mkdir temp && sudo mount linuxroot.img ./temp sudo cp -rfp rootfs/* temp/ sudo umount temp/ e2fsck -p -f linuxroot.img resize2fs -M linuxroot.img
Finally, linuxroot.img is the finished root filesystem image, which is consistent with the filesystem of your development board.
Compile the universal device bus driver locally¶
DEBIAN package download link: https://drive.google.com/drive/folders/1GBl2n66fARqJLVEZ-LeW3bfbs3tyOW3C?usp=sharing
Install header files:
sudo dpkg -i linux-headers-4.4.194_4.4.194-93_armhf.deb
sudo dpkg -i linux-image-4.4.194_4.4.194-93_armhf.deb
cd /usr/src/linux-headers-4.4.194
make headers_check
make headers_install
# make scripts may go wrong but it doesn't matter
make scripts