【Docker】docker Overlay2 文件系统原理

概述

overlayFS是被称为联合文件系统的其中一个解决方案。在2014年，发布了第一个版本并且合并到了Linux的内核3.18版本中，此时，在docker被称为是overlay文件驱动。后来在Linux 内核4.0 版本中进行了改进，称为overlay2。（overlay存在诸多性能和不稳定的问题，不推荐使用overlay，直接使用默认的overlay2即可）

overlay2工作原理

如下图。overlayfs 通过三个目录：lower 目录、upper 目录、以及 work 目录实现，其中 lower 目录可以是多个，work 目录为工作基础目录，挂载后内容会被清空，且在使用过程中其内容用户不可见，最后联合挂载完成给用户呈现的统一视图称为为 merged 目录。

• lowerdir对应底层文件系统，是能被上层文件系统upperdir所共享的只读层
• workdir则可以理解为overlay2运作的一个工作目录，用于完成copy-on-write等操作
• overlay2运作时（也就是容器启动时），会将lowerdir、upperdir和workdir联合挂载到merged目录，为使用者提供一个“统一视图”

在具体使用中。可以查看 /var/lib/docker/overlay2/容器ID目录结构，并使用mount | grep overlay查看overlay的挂载情况。如下，确实此目录下的link、lower、work、diff等目录是通过挂载多个目录后，合并显示在一起的。

[root@k8s-m2 ~]# ls /var/lib/docker/overlay2/
0b7652536a0b18637a8326008b4dd50a577562d1f3d5a62eebe02564845fd9c0       
0b7652536a0b18637a8326008b4dd50a577562d1f3d5a62eebe02564845fd9c0-init    
......
[root@k8s-m2 ~]# mount |grep overlay|more
overlay on /var/lib/docker/overlay2/0b7652536a0b18637a8326008b4dd50a577562d1f3d5a62eebe02564845fd9c0/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/SG5
OKVILBSA6FA4TG6QC3RW4CC:/var/lib/docker/overlay2/l/3SHYUCXLHK7TDFYIIMBGZMHY6J:/var/lib/docker/overlay2/l/67UXAPNT4CKASMUI7FGMJS2UOE:/var/lib/docker/overlay2/l/XDCHJ3X76XZLAA
43KZOHSUO5ZP,upperdir=/var/lib/docker/overlay2/0b7652536a0b18637a8326008b4dd50a577562d1f3d5a62eebe02564845fd9c0/diff,workdir=/var/lib/docker/overlay2/0b7652536a0b18637a83260
08b4dd50a577562d1f3d5a62eebe02564845fd9c0/work)
......

overlay2 是如何存储文件的？

1、环境准备
为了更好的说明，在测试时使用一个干净的环境：没有任何镜像和容器，/var/lib/docker/overlay2目录开始就backingFsBlockDev和l 两个目录。

[root@kube1 ~]# sudo yum install -y yum-utils device-mapper-persistent-data lvm2
[root@kube1 ~]# yum install yum-utils -y
[root@kube1 ~]# sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@kube1 ~]# sudo yum makecache fast
[root@kube1 ~]# yum install -y docker-ce
[root@kube1 ~]# systemctl start docker
[root@kube1 overlay2]# ll
total 0
brw------- 1 root root 253, 0 Jan 27 22:04 backingFsBlockDev
drwx------ 2 root root      6 Jan 27 22:04 l

2、 镜像拉取
拉取一个nginx镜像用于测试，观察拉取过程：可以看到目前版本的Nginx镜像一共被分为7层拉取。

[root@kube1 overlay2]# docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
2f44b7a888fa: Pull complete 
8b7dd3ed1dc3: Pull complete 
35497dd96569: Pull complete 
36664b6ce66b: Pull complete 
2d455521f76c: Pull complete 
dc9c4fdb83d6: Pull complete 
8056d2bcf3b6: Pull complete 
Digest: sha256:4c0fdaa8b6341bfdeca5f18f7837462c80cff90527ee35ef185571e1c327beac
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

3、观察存储目录情况
可以发现/var/lib/docker/overlay2/ 目录下比刚开始多了7个文件夹。

[root@kube1 overlay2]# ls
10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912d139868afd7835ff83503  backingFsBlockDev
2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1  c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0
410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f865757584  cb9d1b97aade9bd3e3a56268a8608622a86a7ac0d0ade9bbb2b0679f4c0ed620
48305d303096d54e0eea026f3366357e2c485c7d5b7c5939e5a062cc36bd03b2  l
a593596d98137556641a7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894

4、首先来查看一下l目录，可以看到l目录是一堆软连接，把一些较短的随机串软连到镜像层的 diff 文件夹下，这样做是为了避免达到mount命令参数的长度限制

[root@kube1 overlay2]# ls -al l/
total 4
drwx------  2 root root  244 Jan 27 22:12 .
drwx--x--- 10 root root 4096 Jan 27 22:12 ..
lrwxrwxrwx  1 root root   72 Jan 27 22:12 D6T76WE7CR4JMMPUM4NTO55AXG -> ../48305d303096d54e0eea026f3366357e2c485c7d5b7c5939e5a062cc36bd03b2/diff
lrwxrwxrwx  1 root root   72 Jan 27 22:12 GTRRPSN3NBPKHCIMECN7X7BJEP -> ../10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912d139868afd7835ff83503/diff
lrwxrwxrwx  1 root root   72 Jan 27 22:12 JMSMKE6MVY4KJ5CB5ZDDIDVPW2 -> ../a593596d98137556641a7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/diff
lrwxrwxrwx  1 root root   72 Jan 27 22:12 LHNRUYWVMA3S5JGMBYRKW4BWHI -> ../c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0/diff
lrwxrwxrwx  1 root root   72 Jan 27 22:12 SX6PROXGAB4ZJPH7HTPYRQUMH2 -> ../cb9d1b97aade9bd3e3a56268a8608622a86a7ac0d0ade9bbb2b0679f4c0ed620/diff
lrwxrwxrwx  1 root root   72 Jan 27 22:12 TMLMOWKXKUJ4WAWMNDCJVVLMBG -> ../2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1/diff
lrwxrwxrwx  1 root root   72 Jan 27 22:12 WCBZTXQX2YLYEZWCUYGFHKTAST -> ../410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f865757584/diff

5、docker image inspect nginx 查看nginx镜像的信息，每个镜像都会有一个GraphDriver.Data信息，这个信息指示了镜像是怎么存的

"GraphDriver": {
    "Data": {
        "LowerDir": "/var/lib/docker/overlay2/48305d303096d54e0eea026f3366357e2c485c7d5b7c5939e5a062cc36bd03b2/diff:/var/lib/docker/overlay2/cb9d1b97aade9bd3e3a56268a8608622a86a7ac0d0ade9bbb2b0679f4c0ed620/diff:/var/lib/docker/overlay2/10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912d139868afd7835ff83503/diff:/var/lib/docker/overlay2/410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f865757584/diff:/var/lib/docker/overlay2/a593596d98137556641a7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/diff:/var/lib/docker/overlay2/c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0/diff",
        "MergedDir": "/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1/merged",
        "UpperDir": "/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1/diff",
        "WorkDir": "/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1/work"
    },
    "Name": "overlay2"
},
"RootFS": {
    "Type": "layers",
    "Layers": [
        "sha256:571ade696b261f0ff46e3cdac4635afc009c4ed3429950cb95cd7e5f70ba0a07",
        "sha256:b6c2a8d6f0ac89ef77e161532f3d9d0dc5dfe0a5f20042e0afc0ad14288405eb",
        "sha256:b61d4b2cd2daf06047984c5876a35338c2beb5ae3f6bef479d25f05772a6a482",
        "sha256:eddcd06e5ef9b91677526f6c55fa01a7d6963c435d5cf2bfb488d91aaa72d4a8",
        "sha256:b4ad478450363f0a8020bb5552641fe6077e78fca48da4d77a979724a3ad2a72",
        "sha256:fbcc9bc44d3e165e7e4f56fb189a05ea5c562a733985ec00d5e3fad309eb63cc",
        "sha256:009507b8560964795eab5126f6363cb2b7403596adf370c9e95d4648c43e771f"
    ]
},
"Metadata": {
    "LastTagTime": "0001-01-01T00:00:00Z"
}

}

6、将这7个文件夹全部展开，可以看到目录结构几乎都是一致的，需要重点关注的是diff文件夹和lower文件。

[root@kube1 overlay2]# ls */
10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912d139868afd7835ff83503/:
committed  diff  link  lower  work

2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1/:
diff  link  lower  work

410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f865757584/:
committed  diff  link  lower  work

48305d303096d54e0eea026f3366357e2c485c7d5b7c5939e5a062cc36bd03b2/:
committed  diff  link  lower  work

a593596d98137556641a7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/:
committed  diff  link  lower  work

c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0/:
committed  diff  link

cb9d1b97aade9bd3e3a56268a8608622a86a7ac0d0ade9bbb2b0679f4c0ed620/:
committed  diff  link  lower  work

l/:
D6T76WE7CR4JMMPUM4NTO55AXG  JMSMKE6MVY4KJ5CB5ZDDIDVPW2  SX6PROXGAB4ZJPH7HTPYRQUMH2  WCBZTXQX2YLYEZWCUYGFHKTAST
GTRRPSN3NBPKHCIMECN7X7BJEP  LHNRUYWVMA3S5JGMBYRKW4BWHI  TMLMOWKXKUJ4WAWMNDCJVVLMBG

可以看到c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0文件夹中不存在lower文件，说明它是最底层的，相当于是根镜像，即docker pull时下载的第一层。
同时，下层diff文件夹下的文件，正是Linux文件目录结构。说明在nginx的dockerfile中，肯定有FROM 的操作。

实例说明：Dockerfile的每一个命令都可能引起了系统的变化，它的每一个变化都会记录一层diff文件。

容器怎么存储的？

1、当前环境只有一个nginx镜像，没有容器，/var/lib/docker/overlay2/ 目录下只有镜像层的存储目录。

2、docker run 启动一个容器

[root@kube1 overlay2]# docker run -id nginx
76cd5f0f5a3f8969d3061dfc94107858f4ae4e9c5634320347c8650528434e87
[root@kube1 overlay2]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS     NAMES
76cd5f0f5a3f   nginx     "/docker-entrypoint.…"   3 seconds ago   Up 2 seconds   80/tcp    festive_lederberg
[root@kube1 overlay2]# ll
total 0
drwx--x--- 4 root root     72 Jan 27 22:12 10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912d139868afd7835ff83503
drwx--x--- 4 root root     72 Jan 28 01:10 2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1
drwx--x--- 4 root root     72 Jan 27 22:12 410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f865757584
drwx--x--- 4 root root     72 Jan 27 22:12 48305d303096d54e0eea026f3366357e2c485c7d5b7c5939e5a062cc36bd03b2
drwx--x--- 4 root root     72 Jan 27 22:12 a593596d98137556641a7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894
brw------- 1 root root 253, 0 Jan 27 22:04 backingFsBlockDev
drwx--x--- 3 root root     47 Jan 27 22:12 c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0
drwx--x--- 5 root root     69 Jan 28 01:10 cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb
drwx--x--- 4 root root     72 Jan 28 01:10 cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb-init
drwx--x--- 4 root root     72 Jan 27 22:12 cb9d1b97aade9bd3e3a56268a8608622a86a7ac0d0ade9bbb2b0679f4c0ed620

3、查看/var/lib/docker/overlay2/ 目录，发现新增了两个目录：其中带-init的目录是只读的；没有init的容器目录才是容器的读写目录

4、link和lower文件与镜像层的功能一致，link文件内容为该容器层的短 ID，lower文件为该层的所有父层镜像的短 ID。diff目录为容器的读写层，容器内修改的文件都会在diff中出现，merged目录为分层文件联合挂载后的结果，也是容器内的工作目录。

5、根据 docker inspect nginx获取到的GraphDriver.Data数据显示，merged目录，是lowerDir各个目录合并UpperDir各个目录后的结果。

[root@kube1 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS     NAMES
76cd5f0f5a3f   nginx     "/docker-entrypoint.…"   26 minutes ago   Up 26 minutes   80/tcp    festive_lederberg
[root@kube1 ~]# docker inspect 76
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb-init/diff:/var/lib/docker/overlay2/2839c8c9efc53d0d0a89611b08d2ecead4717031e2458efe8721e2ec0007d7b1/diff:/var/lib/docker/overlay2/48305d303096d54e0eea026f3366357e2c485c7d5b7c5939e5a062cc36bd03b2/diff:/var/lib/docker/overlay2/cb9d1b97aade9bd3e3a56268a8608622a86a7ac0d0ade9bbb2b0679f4c0ed620/diff:/var/lib/docker/overlay2/10dd8ff4345f0387e720d5fdd1df6c6d2b73547d912d139868afd7835ff83503/diff:/var/lib/docker/overlay2/410fc9512d1e11b714b6643ce2dcd987df56cc0025b4133afb4645f865757584/diff:/var/lib/docker/overlay2/a593596d98137556641a7376b14a4d0b00f6f2d6f05bdffe05b01d740a65e894/diff:/var/lib/docker/overlay2/c2f6ab870d00b7709f1745e0373fa6eb7f90cec78348d36b66ba57c3108045a0/diff",
                "MergedDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb/merged",
                "UpperDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb/diff",
                "WorkDir": "/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb/work"
            },
            "Name": "overlay2"

6、当我们进入容器创建一个文件，然后再宿主机上相应的merged目录也会出现这个文件。

[root@kube1 ~]# docker exec -it 76 /bin/bash
root@76cd5f0f5a3f:/# ls
bin  boot  dev  docker-entrypoint.d  docker-entrypoint.sh  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@76cd5f0f5a3f:/# touch a.txt
root@76cd5f0f5a3f:/# 
exit
[root@kube1 merged]# pwd
/var/lib/docker/overlay2/cb03d50e1baaf7e40f3c981c428213cef7a3b375d7d9134cc3db932fd94b9bcb/merged
[root@kube1 merged]# ls a.txt 

7、查看某个磁盘量使用大的容器
通过如下可以看到目前来说在当前服务器上，k8s_nginx-gateway这个容器磁盘使用量最大。

[root@k8s-m3 ~]# du -s /var/lib/docker/overlay2/*/merged | sort -rn | head
2245124 /var/lib/docker/overlay2/eacbb4f09241f94054700abe1385477ffbdfb80ec8e12cbae3e2309307f84e48/merged
191388  /var/lib/docker/overlay2/2aaa6391b869059c4b7273688daa3380eea523c4940fa9ef677b81c3933d59d6/merged
118368  /var/lib/docker/overlay2/10a01f4c8eda3b5a397a3f883bb4d7f88154803accc0e201717584b5f4b2fad4/merged
110528  /var/lib/docker/overlay2/259b5e08903acc921fcd6f84d7ba95bd797118152f8a63c0543d41c806834902/merged
101548  /var/lib/docker/overlay2/c48167ccceed1301bcb25ed4c1888e0df566929b5400958af683900d1bed86a7/merged
47656   /var/lib/docker/overlay2/fa40f1b425689bde2573b77f31e21d8392a0be9f9243afbb0e93ce0fdc392ab5/merged
47656   /var/lib/docker/overlay2/f8dcd1cce200d1c018f597dd55d75e42c593165e2526d9aa6a6e380ab4a7e285/merged
47656   /var/lib/docker/overlay2/c2d775727d77d6f03c259d864714e3a03e8fefe4cf363ee1561bd0e5587cd5d3/merged
47656   /var/lib/docker/overlay2/4b0c49ba53bd03c2a4c81cd6ad899dfdcf859e115f6f6f204c4437b34b734050/merged
47656   /var/lib/docker/overlay2/43db1fd015f0f14b592f10f93d036fa2bde383c3f8da0a77acfc93f2faaac76d/merged

[root@k8s-m3 ~]# docker ps -q | xargs docker inspect --format '{{.State.Pid}}, {{.Id}}, {{.Name}}, {{.GraphDriver.Data.WorkDir}}' | grep  eacbb4f09241f94054700abe1385477ffbdfb80ec8e12cbae3e2309307f84e48
15110, 808f1771db3776c15251657d1bdf90702cd5470fc70d760d365ee56390c4f9ec, /k8s_nginx-gateway_my-nginx-7ff446c4f4-79wzx_default_2b11ff88-7c7b-4c73-b207-cb0b1d79924f_0, /var/lib/docker/overlay2/eacbb4f09241f94054700abe1385477ffbdfb80ec8e12cbae3e2309307f84e48/work

结论： overlay2将镜像层和容器层都放在单独的目录，并且有唯一 ID，每一层仅存储发生变化的文件，最终使用联合挂载技术将容器层和镜像层的所有文件统一挂载到容器中，使得容器中看到完整的系统文件。

更多关于docker的知识分享，请前往博客主页。编写过程中，难免出现差错，敬请指出