Ubuntu20.04使用zfs系统后以rootless方式安装docker测试

一、安装

从https://download.docker.com/linux/static/stable/下载静态二进制文件，选择对应的硬件平台链接，下载与要安装的Docker引擎版本相关的.tgz文件。

可选择下载最新的docker-xx.xx.x.tgz和docker-rootless-extras-xx.xx.x.tgz，下载后，创建/opt/docker/bin目录，将docker-xx.xx.x.tgz压缩包内的docker/目录下的文件和docker-rootless-extras-xx.xx.x.tgz压缩包内docker-rootless-extras/目录下的文件都解压到/opt/docker/bin目录中。

如果需要使用docker-compose进行容器编排，可从https://github.com/docker/compose/releases下载最新版本的docker-compose，下载后的docker-compose-Linux-x86_64文件放入/opt/docker/bin目录并重命名为docker-compose。

给/opt/docker/bin下的所有文件增加可执行权限：

chmod +x /opt/docker/bin/*

rootless模式需要在主机上安装newuidmap和newgidmap。这些命令由大多数发行版上的uidmap包提供：

sudo apt-get install -y uidmap

进入/opt/docker/bin目录并执行安装命令：

$ ./dockerd-rootless-setuptool.sh install
[INFO] Creating /home/liqf/.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
+ sleep 3
+ systemctl --user --no-pager --full status docker.service
...（省略）
+ systemctl --user enable docker.service
Created symlink /home/liqf/.config/systemd/user/default.target.wants/docker.service → /home/liqf/.config/systemd/user/docker.service.
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger liqf`
​
[INFO] Make sure the following environment variables are set (or add them to ~/.bashrc):
​
export PATH=/opt/docker/bin:$PATH
export DOCKER_HOST=unix:///run/user/1000/docker.sock

根据提示增加默认环境变量：

$ vi ~/.profile
...(使用Shift+g转到文件最后，Shift+a增加如下内容)
if [ -d "/opt/docker/bin" ] ; then
    PATH="/opt/docker/bin:$PATH"
    DOCKER_HOST=unix:///run/user/1000/docker.sock
    export DOCKER_HOST
fi

验证 docker info：

$ docker info
Client:
 Context:    default
 Debug Mode: false
​
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.3
 Storage Driver: zfs
  Zpool: rpool
  Zpool Health: ONLINE
  Parent Dataset: rpool/USERDATA/yourname_nt483j
  Space Used By Parent: 69000417280
  Space Available: 408242704384
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 5.8.0-44-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.655GiB
 Name: PCName
 ID: 2GBY:KKYB:BC4V:R3RT:ZIH5:4GW2:LMG4:QVGK:YXFJ:JEUR:WUHV:X4OB
 Docker Root Dir: /home/liqf/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
​
WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

以上输出可以看出，docker默认使用了zfs存储驱动。

二、运行hello-world测试及zfs问题分析

拉取并运行hello-world报如下错误：

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Extracting  2.529kB/2.529kB
docker: failed to register layer: exit status 1: "/sbin/zfs fs create -o mountpoint=legacy rpool/USERDATA/yourname_nt483j/1ef941359fdac8141843a758f4d3460884adb96269e911b2b27e5440cc885755" => cannot create 'rpool/USERDATA/yourname_nt483j/1ef941359fdac8141843a758f4d3460884adb96269e911b2b27e5440cc885755': permission denied
.
See 'docker run --help'.

根据提示错误为使用zfs创建文件系统时缺少权限，查看rpool/USERDATA/yourname_nt483j目录权限情况：

$ zfs allow rpool/USERDATA/liqf_nt483j
---- Permissions on rpool/USERDATA/liqf_nt483j -----------------------
Local+Descendent permissions:
    user liqf create,destroy,mount

显示当前用户liqf对rpool/USERDATA/liqf_nt483j目录具有create,mount权限，查看错误中显示的命令，除create外还需要mountpoint权限。为用户增加mountpoint权限：

$ sudo zfs allow liqf create,destroy,mount,snapshot,promote,clone,readonly,mountpoint rpool/USERDATA/liqf_nt483j

再次运行hello-world报如下错误：

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Extracting  2.529kB/2.529kB
docker: failed to register layer: error creating zfs mount: mount rpool/USERDATA/liqf_nt483j/dc631fa5915301d19ed17eb9fb554c0f8c8e4410c6169547d9dd3503ff4576cf:/home/liqf/.local/share/docker/zfs/graph/dc631fa5915301d19ed17eb9fb554c0f8c8e4410c6169547d9dd3503ff4576cf: operation not permitted.
See 'docker run --help'.

对zfs的测试表明，Ubunu默认不允许非root用户使用mount命令进行挂载：

$ zfs create rpool/USERDATA/liqf_nt483j/test
filesystem successfully created, but it may only be mounted by root
$ mount rpool/USERDATA/liqf_nt483j/test /mnt
mount: 只有 root 能执行该操作
$ zfs destroy rpool/USERDATA/liqf_nt483j/test

在ZFS文件系统上以rootless模式运行docker测试失败，在官网安装Docker storage drivers中，找到如下描述：

Supported backing filesystems

With regard to Docker, the backing filesystem is the filesystem where /var/lib/docker/ is located. Some storage drivers only work with specific backing filesystems.

STORAGE DRIVER	SUPPORTED BACKING FILESYSTEMS
overlay2,overlay	xfswith ftype=1,ext4
fuse-overlayfs	any filesystem
aufs	xfs,ext4
devicemapper	direct-lvm
btrfs	btrfs
zfs	zfs
vfs	any filesystem

在官网Run the Docker daemon as a non-root user (Rootless mode)中，有以下描述：

Known limitations

• Only the following storage drivers are supported:

  • overlay2 (only if running with kernel 5.11 or later, or Ubuntu-flavored kernel, or Debian-flavored kernel)

  • fuse-overlayfs (only if running with kernel 4.18 or later, and fuse-overlayfs is installed)

  • vfs

因此，在使用zfs文件系统时，可选的存储驱动只有fuse-overlayfs和vfs两种，其中vfs不具有分层存储功能，空间占用较大。

三、使用fuse-overlayfs存储驱动并运行hello-world

存储驱动fuse-overlayfs是为rootless容器开发的开源存储驱动，官网：https://github.com/containers/fuse-overlayfs，可下载代码并使用buildah编译。在ubuntu 20.04中，可使用apt安装：

sudo apt install fuse-overlayfs

要修改docker使用fuse-overlayfs存储驱动，先停止docker服务：

$ systemctl --user stop docker.service

然后在/home/liqf/.config/docker下创建daemon.json并增加配置：

$ vi /home/liqf/.config/docker/daemon.json
{
  "storage-driver":"fuse-overlayfs"
}

启动docker服务：

$ systemctl --user start docker.service

验证 docker info：

$ docker info
Client:
 Context:    default
 Debug Mode: false
​
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.3
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 5.8.0-44-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.655GiB
 Name: FyPC
 ID: 2GBY:KKYB:BC4V:R3RT:ZIH5:4GW2:LMG4:QVGK:YXFJ:JEUR:WUHV:X4OB
 Docker Root Dir: /home/liqf/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
​
WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

以上输出可以看出，docker已经使用了fuse-overlayfs存储驱动。

拉取并运行hello-world成功：

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b8dfde127a29: Pull complete 
Digest: sha256:308866a43596e83578c7dfa15e27a73011bdd402185a84c5cd7f32a88b501a24
Status: Downloaded newer image for hello-world:latest
​
Hello from Docker!
This message shows that your installation appears to be working correctly.
​
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.
​
To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash
​
Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/
​
For more examples and ideas, visit:
 https://docs.docker.com/get-started/