1. Docker¶

1.1. Docker support¶

Firefly normal firmware generally does not meet Docker operation requirements. If there is a need, you can use SDK to enable related kernel configs and rebuild the kernel to support Docker.

(RK356X v1.2.4a and later, RK3588 support Docker by default, you can skip straight to the installation step)

The following case is based on Firefly Ubuntu 20.04, and the kernel configuration part is generic!

1.1.1. Check Kernel Configuration¶

First use script to see which configuration is needed by Docker but missing in current kernel. You can get the check script from GitHub.

After get the script, begin to check:

#Copy the script to SDK/kernel/
cp check-config.sh PathToSDK/kernel/
cd PathToSDK/kernel
chmod +x check-config.sh

#Get current kernel configuration
make ARCH=arm64 firefly_linux_defconfig

#Check
./check-config.sh .config

The result looks like this, mainly two parts:

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
......

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
......

Generally Necessary: All features in this part need to be enabled in kernel.

Optional Features: Enable them as you needed.

1.1.2. Enable Needed Features¶

From the result you can find out which features are need to be enabled, then use make ARCH=arm64 menuconfig to enter config menu, search and enable them. Please read the instructions in the config menu, if some features can’t be selected, check the dependency.

Enabled all the necessary features and some optional features, remember to save:

make ARCH=arm64 savedefconfig
mv defconfig arch/arm64/configs/firefly_linux_defconfig

Compile the kernel:

#Back to SDK dir
cd ..
#Compile
./build.sh kernel

1.1.3. Install Docker¶

After burning new kernel, install Docker on device (This method works on PC as well)

Step1：Quick install

# Here only introduces the quick-install using script
apt-get update
wget -qO- https://get.docker.com/ | sh

You should see the Docker version information after the installation is successful.

Step2：Check docker root dir (This step is only for PC docker installing)

If you are installing docker in Firefly device, please skip this step.

# execute
docker info | grep -i dir
# result
 Docker Root Dir: /var/lib/docker

The result shows docker root dir, this location may be different on different PC.

Images and containers need large disk space, if the default location does not have enough space, you need to change the default location to one with sufficient space.

Emphasize again：This step is only for PC, in Firefly device, change this dir will cause docker error, please go to next step.

# stop docker service first
sudo systemctl stop docker

# modify /lib/systemd/system/docker.service
sudo vim /lib/systemd/system/docker.service

# add your destination dir at the end of ExecStart line: --graph /home/firefly/docker/data
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --graph /home/firefly/docker/data

# start docker service
sudo systemctl daemon-reload
sudo systemctl start docker

# check if the location is updated
docker info | grep -i dir
 Docker Root Dir: /home/firefly/docker/data

Step3：add your user to docker group

sudo usermod -a -G docker firefly
# reboot
sudo reboot

Step4：After reboot, run demo to test docker:

firefly@firefly:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
93288797bd35: Pull complete
Digest: sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

1.2. Build a distributed compilation environment with Docker¶

distcc is a program used for distributed compilation of C, C++, Objective-C or Objective-C++ code through several machines on the network. distcc does not require all machines to share the file system, have synchronized clocks, or have the same libraries and header files installed, as long as the server machine has a suitable compilation tool to compile. This example uses Docker technology to deploy the distcc distributed compilation service on two Firefly-RK3399 devices (arm64) and one PC (x86_64), and then realizes the use of distcc’s distributed compilation features on one of the Firefly-RK3399 devices To speed up the kernel compilation process.

1.2.1. Ready to work¶

Two Firefly-RK3399 devices;
Router, network cable;
PC machine.

Connect the two devices and the PC to the same LAN. The corresponding IP address after connection is:

PC: 192.168.1.100
Device A: 192.168.1.101
Device B: 192.168.1.102

1.2.2. PC deployment Docker¶

Install Docker using script:

wget -qO- https://get.docker.com/ | sh

In order to enable current ordinary users to execute docker related commands, the current user needs to be added to the dokcer user group:

sudo groupadd docker # add docker user group
sudo gpasswd -a $USER docker # Add the current user to the docker user group
newgrp docker # Update docker user group

Start the Docker background service:

sudo service docker start

Create a new Dockerfile_distcc.x86_64 file. The contents of the file are as follows:

FROM ubuntu:bionic
MAINTAINER Firefly <service@t-firefly.com>

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests\
gcc-aarch64-linux-gnu distcc\
    && apt-get clean \
    && rm -rf /var/lib/apt/lists

RUN ln -s aarch64-linux-gnu-gcc /usr/bin/gcc &&\
ln -s aarch64-linux-gnu-gcc /usr/bin/cc

EXPOSE 3632

ENTRYPOINT ["/usr/bin/distccd"]
CMD ["--no-detach", "--log-stderr", "--log-level", "debug", "--allow", "0.0.0.0/0"]

Generate Docker image:

docker build -t distcc_server:x86_64 -f Dockerfile_distcc.x86_64.

You can use the command docker images to view the generated Docker image:

docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
distcc_server x86_64 138c0b7e3801 9 minutes ago 66.1MB

Start the Docker container with the newly created image, and provide the distcc service on the 3632 TCP port of the host network:

docker run -d -p 3632:3632 distcc_server:x86_64

Use the command docker ps to view the running container:

docker ps
CONTAINER ID    IMAGE                  COMMAND                  CREATED         STATUS          PORTS                    NAMES
fa468d068185    distcc_server:x86_64   "/usr/bin/distccd --…"   9 minutes ago   Up 9 minutes    0.0.0.0:3632->3632/tcp   epic_chatterjee

1.2.3. Device deployment Docker¶

apt-get update
wget -qO- https://get.docker.com/ | sh

Add the current user to the dokcer user group:

sudo groupadd docker
sudo gpasswd -a $USER docker
newgrp docker # Update docker

Start the Docker background service:

service docker start

Create a new Dockerfile_distcc.arm64 file with the following content:

FROM ubuntu:bionic
MAINTAINER Firefly <service@t-firefly.com>

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
gcc distcc\
    && apt-get clean \
    && rm -rf /var/lib/apt/lists

EXPOSE 3632

ENTRYPOINT ["/usr/bin/distccd"]
CMD ["--no-detach", "--log-stderr", "--log-level", "debug", "--allow", "0.0.0.0/0"]

Generate Docker image:

docker build -t distcc_server:arm64 -f Dockerfile_distcc.arm64.

Start the Docker container with the newly created image, and provide the distcc service on the 3632 TCP port of the host network:

docker run -d -p 3632:3632 distcc_server:arm64

Use the docker save command to export the newly created image:

docker save -o distcc_server.tar distcc_server:arm64

Copy the generated distcc_server.tar file to another device, and then import the image:

docker load -i distcc_server.tar

After importing the image, there is also a distcc_server:arm64 image on this device, and then you can run a container:

docker run -d -p 3632:3632 distcc_server:arm64

Tip: If the user has registered a Docker Hub account, he can use docker push to push the image to the remote warehouse, and use docker pull on another device to pull the image of the remote warehouse. Users can understand related operations by themselves.

1.2.4. The client compiles the kernel with distcc¶

So far, all three machines have deployed a distributed compilation environment. You can choose any machine as the client, and the remaining two machines as the server. Choose one of the devices as the client.

Create a Dockerfile_compile.arm64 file in the client to generate a Docker image for compiling the kernel. The content of the file is as follows:

FROM ubuntu:bionic
MAINTAINER Firefly <service@t-firefly.com>

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
    && apt-get install -y --no-install-recommends --no-install-suggests \
bc make python sed libssl-dev binutils build-essential distcc\
    liblz4-tool gcc \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists

Then generate a mirror:

docker build -t compile:arm64 -f Dockerfile_compile.arm64.

Pull the kernel file to the client before starting the container. Then create the /etc/distcc/hosts file, its content is to list all the IP addresses of the machines that provide distcc services. The content is as follows:

# As described in the distcc manpage, this file can be used for a global
# list of available distcc hosts.
#
# The list from this file will only be used, if neither the
# environment variable DISTCC_HOSTS, nor the file $HOME/.distcc/hosts
# contains a valid list of hosts.
#
# Add a list of hostnames in one line, seperated by spaces, here.
192.168.1.100
192.168.1.101
193.168.1.102

In order to test the results more accurately, first clear the cache on the client device:

echo 3> /proc/sys/vm/drop_caches

Start a Docker container with the compile:arm64 image, and at the same time mount the host’s current kernel directory to the /mnt directory in the container, and mount the host’s /etc/distcc/hosts file to the container/etc /distcc/hosts and use the parameter -it to start the container in interactive mode:

docker run -it --rm -v $(pwd):/mnt -v /etc/distcc/hosts:/etc/distcc/hosts compile:arm64 /bin/bash
root@f4415264351b:/# ls
bin boot dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
root@f4415264351b:/# cd mnt/

Enter the /mnt directory in the container, and then start to compile the kernel with distcc, add the time command to view the execution time of the compilation command, the CC parameter indicates to use distcc to compile:

time make ARCH=arm64 rk3399-firefly.img -j32 CC="distcc"

Note: If you use a PC as the client, you need to use the following command to compile:

time make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- rk3399-firefly.img -j32 CC="distcc aarch64-linux-gnu-gcc"

During the compilation process, you can use the command distccmon-text 1 to view the compilation status in a new window in the container used for compilation:

distccmon-text 1
                    ...
Compile perf_regs.c 192.168.1.101[0]
Compile fork.c 192.168.1.101[2]
Compile dma-mapping.c 192.168.1.101[3]
Compile signal32.c 192.168.1.100[0]
Compile open.c 192.168.1.100[2]
Compile traps.c 192.168.1.100[3]
Compile arm64ksyms.c 192.168.1.102[0]
Compile mempool.c 192.168.1.102[2]
Compile filemap.c 192.168.1.102[3]
Preprocess localhost[0]
Preprocess localhost[1]
Preprocess localhost[2]
                    ...

After the final compilation command is completed, you can see the time taken for compilation:

real 15m44.809s
user 16m0.029s
sys 6m22.317s

The following is the time it takes to compile the kernel with a single device:

real 23m33.002s
user 113m2.615s
sys 9m29.348s

By comparison, it can be seen that the distributed compilation implemented by distcc can effectively improve the compilation speed.

note:

Different platforms require different compilers. For example, on the x86_64 platform, you need to install the corresponding cross-compilation tool gcc-aarch64-linux-gnu. On the arm64 platform, you only need to install the gcc compilation tool. Users need to specify and install the correct tool in the corresponding Dockerfile file according to the actual situation.
If you use a PC as a client, you must use the parameter CC="distcc aarch64-linux-gnu-gcc" to specify the use of aarch64-linux-gnu-gcc when compiling the kernel in the container used for compilation Compile with cross-compiler tools.