docker containerd 交叉编译 移植到arm64v8
可以把docker移植到嵌入式的aarch64 linux系统中，比如hi3559等芯片
一. docker编译
docker以及所依赖的containerd、runc等可能没法自己交叉编译，没有相关文档，并且东西太多太复杂，不过好在docker官方为不同平台提供编译好的程序，其中包括：
containerd、ctr、docker-init、dockerd、containerd-shim、docker、docker-proxy、runc
相关说明：
https://docs.docker.com/engine/install/binaries/
其中包括下载地址：
https://download.docker.com/linux/static/stable/aarch64/
二. 移植iptable
如果系统中没有iptable命令，则需要移植
0.打开内核选项：
Networking support ---> Networking options ---> Network packet filtering framework (Netfilter)
选中Core Netfilter Configuration以及下面所有的，IP: Netfilter Configuration以及下面所有的。
iptable又依赖libmnl、libnftnl，需要按顺序编译：
1.libmnl
git clone https://git.netfilter.org/libmnl/
./configure CC=aarch64-himix100-linux-gcc --host=aarch64-himix100-linux --prefix=$PWD/bin --enable-static
make
make install2.libnftnl
git clone https://git.netfilter.org/libnftnl/
LIBMNL_CFLAGS="-I/home/a/share/libmnl/bin/include" \
LIBMNL_LIBS="-L/home/a/share/libmnl/bin/lib -lmnl" \
./configure CC=aarch64-himix100-linux-gcc --host=aarch64-himix100-linux --prefix=$PWD/bin --enable-static
make
make install3.iptables
git clone https://git.netfilter.org/iptables/
libmnl_CFLAGS="-I/home/a/share/libmnl/bin/include" \
libmnl_LIBS="-L/home/a/share/libmnl/bin/lib -lmnl" \
libnftnl_CFLAGS="-I/home/a/share/libnftnl/bin/include" \
libnftnl_LIBS="-L/home/a/share/libnftnl/bin/lib -lnftnl" \
./configure CC=aarch64-himix100-linux-gcc --host=aarch64-himix100-linux --prefix=$PWD/bin --enable-static --without-cli
make
make install最后把 iptables/bin 目录下生成的东西放到板子上。
三. 内核配置
内核需要打开很多选项，具体见此文章最后的docker依赖的内核选项列表。
此列表是用这个脚本检测并生成的 https://github.com/moby/moby/blob/master/contrib/check-config.sh
四. 运行服务
1.配置eth0和lo：
ifconfig lo 127.0.0.1 netmask 255.255.255.0
2.挂载cgroup并重启
vi /etc/fstab
在最后一行增加以下配置信息：
none /sys/fs/cgroup cgroup defaults 0 0
3. 把下载的docker安装包里的文件都放在 /usr/bin 下
4. 先执行 containerd &，然后执行 dockerd &
五. 运行镜像
1.下载镜像
一般要在单板上用 docker pull arm64v8/ubuntu 来下载镜像，还要配置好 /etc/docker/daemon.json，如果pull有问题的话可以先用另外的方法下载镜像：
- 找一个正式的ubuntu系统通过上面的命令下载镜像
- 用命令 docker save arm64v8/ubuntu > ubuntu_image.tar 来保存镜像，把ubuntu_image.tar放到单板上
- 在单板上用 docker load < ./ubuntu_image.tar 来导入镜像，这时候 docker image ls 就能看到这个镜像了
2.执行 docker run -it arm64v8/ubuntu /bin/sh 就能运行容器，并进入这个容器的shell环境
附：docker依赖的内核选项:
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: missing
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: missing
- CONFIG_POSIX_MQUEUE: enabled
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
(cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: missing
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled (as module)
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
- "overlay":
- CONFIG_VXLAN: enabled (as module)
- CONFIG_BRIDGE_VLAN_FILTERING: enabled
Optional (for encrypted networks):
- CONFIG_CRYPTO: enabled
- CONFIG_CRYPTO_AEAD: enabled
- CONFIG_CRYPTO_GCM: enabled
- CONFIG_CRYPTO_SEQIV: enabled
- CONFIG_CRYPTO_GHASH: enabled
- CONFIG_XFRM: enabled
- CONFIG_XFRM_USER: enabled
- CONFIG_XFRM_ALGO: enabled
- CONFIG_INET_ESP: enabled (as module)
- CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- "ipvlan":
- CONFIG_IPVLAN: missing
- "macvlan":
- CONFIG_MACVLAN: enabled (as module)
- CONFIG_DUMMY: enabled (as module)
- "ftp,tftp client in container":
- CONFIG_NF_NAT_FTP: enabled (as module)
- CONFIG_NF_CONNTRACK_FTP: enabled (as module)
- CONFIG_NF_NAT_TFTP: enabled (as module)
- CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
- "aufs":
- CONFIG_AUFS_FS: missing
- "btrfs":
- CONFIG_BTRFS_FS: enabled (as module)
- CONFIG_BTRFS_FS_POSIX_ACL: enabled
- "devicemapper":
- CONFIG_BLK_DEV_DM: enabled (as module)
- CONFIG_DM_THIN_PROVISIONING: enabled (as module)
- "overlay":
- CONFIG_OVERLAY_FS: enabled (as module)
- "zfs":
- /dev/zfs: missing
- zfs command: missing
- zpool command: missing