Bluemix虚拟机Docker使用direct-lvm存储方式运行容器

最近在Bluemix的虚拟机上玩docker，但是发现默认安装使用的是loop-lvm的模式做后端存储，这个肯对对后面的实验会造成影响，而且docker官方也不建议在生产环境下使用loop-lvm，下图为docker官方给出的存储方案优劣对比：

所以今天开题分享一下如何更改docker的devicemapper存储方式为direct-lvm。

docker 最先是跑在ubuntu和debian上的,使用aufs存储器. 由于docker越来越流行,许多公司希望在RHEL上使用,但是上游内核中没有包括aufs,所以rhel不能使用aufs. 最终开发者们开发了一个新的后端存储引擎devicemapper,基于已有的Device Mapper技术,并且使docker支持可插拔,现在全世界有很多真实案例在生产环境使用devicemapper。Device Mapper是Linux系统中基于内核的高级卷管理技术框架。Docker的devicemapper存储驱动就是基于该框架的精简置备和快照功能来实现镜像和容器的管理。

devicemapper是Red Hat Enterprise Linux下Docker Engine的默认存储驱动，它有两种配置模式:loop-lvm和direct-lvm，loop-lvm是默认的模式，但如果是在生产环境的部署Docker，官方不推荐使用该模式。我们使用docker info命令可以看到以下警告:

 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `–storage-opt dm.thinpooldev` or use `–storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.

direct-lvm是Docker推荐的生产环境的推荐模式，他使用块设备来构建精简池来存放镜像和容器的数据。

本文的操作系统是Centos7.2，使用docker版本为1.12，devicemapper版本为device-mapper-1.02.107-5.el7.x86_64。

1.停止Docker服务

首先运行docker info 查看当前系统的docker配置：

# root at bastion.shanker in ~ [15:03:10]
# docker info
Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 40
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: docker-253:0-11668417-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 3.641 GB
 Data Space Total: 107.4 GB
 Data Space Available: 103.7 GB
 Metadata Space Used: 5.972 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.142 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2015-10-14)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.7.9-200.fc24.x86_64
Operating System: Fedora 24 (Twenty Four)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 8
Total Memory: 14.99 GiB
Name: bastion.shanker
ID: SWHZ:KSZ3:CQMS:W5HN:F33Z:HWU2:2GUE:OQII:7BSE:J62P:6EMX:RHL4
Username: shanker
Registry: https://index.docker.io/v1/
Registries: docker.io (secure)

发现Storage Driver是Devicemapper，Data File和Metadata File都是loop设备，下面我们将docker停掉：

#systemctl stop docker

2. 添加磁盘并创建thin-pool lv

查看新添加的磁盘

fdisk -l /dev/sdb

Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

创建pv

# root at model.shanker in ~ [14:36:01]
# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created

创建vg

# root at model.shanker in ~ [14:37:49]
# vgcreate docker /dev/sdb
  Volume group "docker" successfully created

创建data lv

# root at model.shanker in ~ [14:41:00]
# lvcreate --wipesignatures y -n thinpool docker -l 95%VG 
  Logical volume "thinpool" created.

创建metadata lv

# root at model.shanker in ~ [14:41:06]
# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
  Logical volume "thinpoolmeta" created.

注意作为meta的pool大小不能超过16GB！！！

将pool 转换为thin-pool

将 thinpool lv 的 chunksize 改为 512KB,并且将前 4KB 字节清零。

 root at model.shanker in ~ [14:42:31]
# lvconvert -y --zero n -c 512k --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
  WARNING: Converting logical volume docker/thinpool and docker/thinpoolmeta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted docker/thinpool to thin pool.

创建一个thinpool的profile

# root at model.shanker in /etc/lvm/profile [14:45:42]
# vi docker-thinpool.profile

# root at model.shanker in /etc/lvm/profile [14:45:50]
# pwd
/etc/lvm/profile

# root at model.shanker in /etc/lvm/profile [14:45:53]
# cat docker-thinpool.profile 
activation {

thin_pool_autoextend_threshold=80

thin_pool_autoextend_percent=20

}

应用配置

# root at model.shanker in /etc/lvm/profile [14:45:56]
# lvchange --metadataprofile docker-thinpool docker/thinpool
  Logical volume "thinpool" changed.

注意： docker-thinpool 即刚才创建的 profile 文件名的前缀，不需要加.profile,而且要在
/etc/lvm/profile 目录下运行此命令。 执行完毕后不要mount，不要格式化 lv。

3. 查看lv状态

# root at model.shanker in /etc/lvm/profile [14:46:50]
# lvs -o+seg_monitor
  LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Monitor  
  thinpool docker twi-a-t--- 19.00g             0.00   0.03                             monitored
  root     rhel   -wi-ao---- 27.46g                                                              
  swap     rhel   -wi-ao----  2.00g                                                              

看到有Meta 和Data下面都有数字，代表刚才创建的thinpool创建成功。

4. 配置docker

添加如下参数到docker服务的启动项里

ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt
dm.thinpooldev=/dev/mapper/docker-thinpool –storage-opt dm.fs=xfs --storage-opt
dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true -D -H
tcp://0.0.0.0:5256 -H unix:///var/run/docker.sock --pidfile=/var/run/docker.pid<

完整的配置如下

# cat /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
#ExecStart=/usr/bin/dockerd
ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt
dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true --storageopt dm.use_deferred_deletion=true -D -H
tcp://0.0.0.0:5256 -H unix:///var/run/docker.sock --pidfile=/var/run/docker.pid
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]

5. 清除Graphdriver

在启动docker之前，需要将之前残留的docker文件删除掉，要不然会有以下报错：

Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem
verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed

rm -rf /var/lib/docker/*

6. 启动docker

#systemctl daemon-reload
#systemctl start docker
# docker info
Containers: 6
Running: 0
Paused: 0
Stopped: 6
Images: 1
Server Version: 1.12.2
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 100.7 MB
Data Space Total: 20.4 GB
Data Space Available: 20.3 GB
Metadata Space Used: 114.7 kB
Metadata Space Total: 213.9 MB
Metadata Space Available: 213.8 MB
Thin Pool Minimum Free Space: 2.039 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true

Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge overlay null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: RHEV
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.797 GiB
Name: model.shanker
ID: 3RCI:APCJ:5NE3:JJPN:HW7P:SIND:P6KW:2J26:4XPG:VJRG:UTNW:TQY6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 24
System Time: 2016-10-21T17:12:54.422486318+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0/8

查看 devicemapper 的资源,发现 docker-thinpool 与 docker info 显示的 Pool Name 一致,代表启用
direct-lvm 成功。

# dmsetup ls
rhel-swap (253:1)
rhel-root (253:0)
docker-253:0-35330063-pool (253:5)
docker-thinpool_tdata (253:3)
docker-thinpool_tmeta (253:2)
docker-thinpool (253:4)