Docker-使用 overlayfs 作为后端存储遇到的问题

问题描述

在一个测试环境中，发现容器中无法删除镜像中只读层的文件，现象：

问题原因

参考 https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html

相关知识点

1、XFS 文件系统作为 overlayfs 底层时，该文件系统创建时必须使用 -n ftype=1 标志，使用 xfs_info 命令查看 xfs 文件系统的标志：

$ xfs_info /var/lib/docker
meta-data=/dev/mapper/centos_k8s--master-root isize=256    agcount=4, agsize=4653056 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=18612224, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=9088, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ xfs_info /var/lib/docker | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

在一台正常的机器上：

$ xfs_info /var/lib/docker
meta-data=/dev/mapper/cl-root    isize=512    agcount=4, agsize=2956800 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=11827200, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=5775, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ xfs_info /var/lib/docker |grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

2、通过 man mkfs.xfs 命令，查看 -n ftype 的含义：是否允许将 inode 类型存储在目录结构中，以便 readdir，getdents 不需要查找 inode 就可知道 inode 类型，默认为0，不存在目录结构中。

3、在 docker-1.13+ 版本中，使用 docker info 命令可以看到对 xfs 的 ftype 特性是否支持的信息 https://github.com/moby/moby/pull/27433

如果supports d_type为false，会有如下告警信息

解决办法

使用独立的 LV 并使用 mkfs.xfs -n ftype=1 命令格式化新的分区，然后挂载到 /var/lib/docker
如果 /var/lib/docker 已经使用了单独的 LV，需要先删除原来的 LV，再重新创建一个：

$ systemctl stop docker && systemctl disable docker
$ rm -rf /var/lib/docker/*
$ umount /dev/mapper/centos_k8s--node1-var_lib_docker
$ lvremove /dev/mapper/centos_k8s--node1-var_lib_docker
$ lvcreate -L 180G -n docker-lv centos_k8s-node1
$ mkfs.xfs -n ftype=1 /dev/centos_k8s-node1/docker-lv
$ mount /dev/mapper/centos_k8s--node1-docker--lv /var/lib/docker
$ sed '/centos_k8s--node1-var_lib_docker/d' /etc/fstab
$ echo "/dev/mapper/centos_k8s--node1-docker--lv  /var/lib/docker  xfs     defaults        0 0" >> /etc/fstab
$ systemctl start docker && systemctl enable docker

如果 /var/lib/docker 没有使用单独的 LV，推荐使用新的 LV 挂载到 /var/lib/docker 目录：

$ systemctl stop docker && systemctl disable docker
$ rm -rf /var/lib/docker/*
# 为虚拟机添加一块新磁盘 /dev/sdb 大小 50GB
$ fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x50e1bb21.
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-104857599, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-104857599, default 104857599):
Using default value 104857599
Partition 1 of type Linux and of size 50 GiB is set
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'
Command (m for help): p
Disk /dev/sdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x50e1bb21
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   104857599    52427776   8e  Linux LVM
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
# 强制内核写入新的分区表
$ partprobe
# 创建PV
$ pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created.
$ pvs
  PV         VG                Fmt  Attr PSize  PFree
  /dev/sda2  centos_k8s-master lvm2 a--  98.88g  4.00m
  /dev/sdb1                    lvm2 ---  50.00g 50.00g
# 创建 VG
$ vgcreate docker-vg /dev/sdb1
  Volume group "docker-vg" successfully created
$ vgs
  VG                #PV #LV #SN Attr   VSize  VFree
  centos_k8s-master   1   3   0 wz--n- 98.88g  4.00m
  docker-vg           1   0   0 wz--n- 50.00g 50.00g
# 创建 LV
$ lvcreate -L 45G -n docker-lv docker-vg
  Logical volume "docker-lv" created.
$ lvs
  LV        VG                Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home      centos_k8s-master -wi-ao---- 20.00g                                                   
  root      centos_k8s-master -wi-ao---- 71.00g                                                   
  swap      centos_k8s-master -wi-ao----  7.88g                                                   
  docker-lv docker-vg         -wi-a----- 45.00g
  
###### 使用 mkfs.xfs -n ftype=1 格式化新的 LV
$ mkfs.xfs -n ftype=1 /dev/docker-vg/docker-lv
meta-data=/dev/docker-vg/docker-lv isize=512    agcount=4, agsize=2949120 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=11796480, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=5760, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
$ [[ ! -d /var/lib/docker ]] && mkdir /var/lib/docker
# 将 /dev/docker-vg/docker-lv 挂载到 /var/lib/docker
$ mount /dev/docker-vg/docker-lv /var/lib/docker
# 查看 /var/lib/docker 的 ftype 参数值
$ xfs_info /var/lib/docker |grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
# 设置开机自动挂载
$ echo "/dev/mapper/docker--vg-docker--lv   /var/lib/docker         xfs     defaults        0 0" >> /etc/fstab
$ systemctl start docker && systemctl enable docker

现在重新测试一下：

$ docker run --rm -it centos:7.3.1611
Unable to find image 'centos:7.3.1611' locally
7.3.1611: Pulling from library/centos
45a2e645736c: Pull complete
Digest: sha256:c577af3197aacedf79c5a204cd7f493c8e07ffbce7f88f7600bf19c688c38799
Status: Downloaded newer image for centos:7.3.1611
[root@325a7b4bc2d8 /]# ls
anaconda-post.log  bin  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@325a7b4bc2d8 /]# rm -f anaconda-post.log
[root@325a7b4bc2d8 /]# ll
total 0
lrwxrwxrwx.   1 root root   7 Dec 14 14:57 bin -> usr/bin
drwxr-xr-x.   5 root root 360 May 18 04:42 dev
drwxr-xr-x.   1 root root  66 May 18 04:42 etc
drwxr-xr-x.   2 root root   6 Nov  5  2016 home
lrwxrwxrwx.   1 root root   7 Dec 14 14:57 lib -> usr/lib
lrwxrwxrwx.   1 root root   9 Dec 14 14:57 lib64 -> usr/lib64
drwx------.   2 root root   6 Dec 14 14:57 lost+found
drwxr-xr-x.   2 root root   6 Nov  5  2016 media
drwxr-xr-x.   2 root root   6 Nov  5  2016 mnt
drwxr-xr-x.   2 root root   6 Nov  5  2016 opt
dr-xr-xr-x. 243 root root   0 May 18 04:42 proc
dr-xr-x---.   2 root root 114 Dec 14 14:59 root
drwxr-xr-x.  10 root root 130 Dec 14 14:59 run
lrwxrwxrwx.   1 root root   8 Dec 14 14:57 sbin -> usr/sbin
drwxr-xr-x.   2 root root   6 Nov  5  2016 srv
dr-xr-xr-x.  13 root root   0 May 18 04:37 sys
drwxrwxrwt.   7 root root 132 Dec 14 14:59 tmp
drwxr-xr-x.  13 root root 155 Dec 14 14:57 usr
drwxr-xr-x.  18 root root 238 Dec 14 14:59 var
[root@325a7b4bc2d8 /]#

现在一些都正常了。

参考资料：

1、https://github.com/moby/moby/issues/27358

2、https://github.com/moby/moby/issues/31283

3、https://github.com/moby/moby/issues/31445

4、https://github.com/moby/moby/pull/27433

5、https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html