使用内核态的RBD

使用内核态的RBD模块（正常情况）

• 创建测试RBD Image：

[root@ceph-node1 tmp]# rbd create rbd/bench-image --size 1G
[root@ceph-node1 tmp]# rbd ls -p rbd
bench-image
test-image
[root@ceph-node1 tmp]# rbd info rbd/bench-image
rbd image 'bench-image':
	size 1024 MB in 256 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.5ddff6b8b4567
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags:

• 在一台虚拟机中使用这个Image：

[root@ceph-node1 ~]# rbd map bench-iamge
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (2) No such file or directory

• 查看系统Log：

[root@ceph-node1 ~]# dmesg | tail
Oct 18 02:30:24 ceph-node1 kernel: libceph: client374111 fsid cbc99ef9-fbc3-41ad-a726-47359f8d84b3
Oct 18 02:30:24 ceph-node1 kernel: rbd: image bench-image: image uses unsupported features: 0x38

从日志可以看出，是RBD Image的有些feature，当前内核不支持，所以需要disable这些feature，rbd支持disable功能。

• 内核及系统版本

[root@ceph-node1 ~]# uname -r
3.10.0-514.26.2.el7.x86_64
[root@ceph-node1 ~]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)

• Disable RBD Image feature

[root@ceph-node1 ~]# rbd feature disable bench-image fast-diff
[root@ceph-node1 ~]# rbd feature disable bench-image object-map
[root@ceph-node1 ~]# rbd feature disable bench-image exclusive-lock
[root@ceph-node1 ~]# rbd feature disable bench-image deep-flatten

上面这些feature也可以通过一条命令直接disable，不过需要注意feature先后顺序。

• 查看当前Image的feature

[root@ceph-node1 ~]# rbd info bench-image
rbd image 'bench-image':
	size 1024 MB in 256 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.5ddff6b8b4567
	format: 2
	features: layering
	flags:

• 继续使用RBD Image

[root@ceph-node1 ~]# rbd map bench-image --id admin
/dev/rbd0

使用RBD内核模块 (通过上面的disable feature，依然不成功，可以尝试如下方法)

• 在另外一个Ceph集群中，通过上述方法，依然不能使用RBD内核模块

[root@nova10 ~]# uname -r
3.10.0-327.el7.x86_64
[root@nova10 ~]# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

[root@storage04 ~]# rbd info volumes/bench-image
rbd image 'bench-image':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.2adef66b8b4567
	format: 2
	features: layering
	flags: 

[root@nova10 ~]# rbd map volumes/bench-image --id admin
^C

[root@nova10 ~]# tailf /var/log/messages
Oct 18 15:04:41 nova10 kernel: libceph: mon1 192.168.34.12:6789 feature set mismatch, my 102b84a842a42 < server's 40102b84a842a42, missing 400000000000000
Oct 18 15:04:41 nova10 kernel: libceph: mon1 192.168.34.12:6789 missing required protocol features
Oct 18 15:07:24 nova10 kernel: libceph: mon0 192.168.34.11:6789 feature set mismatch, my 102b84a842a42 < server's 40102b84a842a42, missing 400000000000000
Oct 18 15:07:24 nova10 kernel: libceph: mon0 192.168.34.11:6789 missing required protocol features

• Google大神搜索，发现社区有人也遇到过这种情况，大致说法就是crush tunable指定的版本太高了，低版本内核不支持，具体查看如下链接：http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client

• 但由于该Ceph集群是生产环境，我不可能调整这个crush tunable，内核升级基本也不太可能，所以选择了使用rbd nbd模块来使用。

使用rbd-nbd模块

• 安装rbd-nbd

[root@ceph-node1 ~]# yum install rbd-nbd
Installed:
  rbd-nbd.x86_64 1:10.2.9-2.el7.centos                                                                                                                                                  
Complete!

• 使用rbd-nbd

[root@ceph-node1 ~]# rbd-nbd map test-image
modprobe: FATAL: Module nbd not found.
rbd-nbd: failed to find unused device

也不成功，报错：没有nbd模块

• 我们只能自己编译nbd模块，编译过程在后面会详细说明。
• 当成功加了nbd模块之后，使用rbd-nbd

[root@ceph-node1 ~]# rbd-nbd map test-image
/dev/nbd0

[root@ceph-node1 ~]# rbd-nbd list-mapped
/dev/nbd0

[root@ceph-node1 ~]# rbd showmapped
id pool image       snap device    
0  rbd  bench-image -    /dev/rbd0

编译nbd模块

• 查看当前系统内核

[root@ceph-node1 ~]# uname -r
3.10.0-514.26.2.el7.x86_64

• 下载内核源码包

[root@ceph-node1 ~]# wget http://vault.centos.org/7.3.1611/os/Source/SPackages/kernel-3.10.0-514.el7.src.rpm

Centos的内核包地址：（注意替换相应的系统版本） http://vault.centos.org/7.3.1611/os/Source/SPackages/

• 安装依赖包

yum install rpm-build m4 net-tools bc xmlto asciidoc hmaccalc newt-devel perl pesign elfutils-devel binutils-devel bison audit-libs-devel numactl-devel pciutils-devel ncurses-devel libtiff perl-ExtUtils-Embed python-devel java-devel -y

• 安装内核源码包：

[root@ceph-node1 ~]$ rpm -ivh kernel-3.10.0-514.el7.src.rpm

• 准备编译

[root@ceph-node1 ~]$ echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros
[root@ceph-node1 ~]$ cd ~/rpmbuild/SPECS
[root@ceph-node1 ~]$ rpmbuild -bp --target=$(uname -m) kernel.spec

• 编译

  • 在内核源码中选择编译nbd模块

    [root@ceph-node1 ~]$ cd ~/rpmbuild/BUILD/kernel-3.10.0-514.el7/linux-3.10.0-514.el7.centos.x86_64
    [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ make menuconfig
      在弹出的窗口中选择Device Driver -> Block devices -> Network block device support，并在“Network block device support”中输入“M”，保存退出

  • 编译

    [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ make prepare && make modules_prepare && make
    [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ make M=drivers/block -j4

  • 问题

    编译的时候遇到如下错误：

    drivers/block/nbd.c: 在函数‘__nbd_ioctl’中:
    drivers/block/nbd.c:619:19: 错误：‘REQ_TYPE_SPECIAL’未声明(在此函数内第一次使用)
    sreq.cmd_type = REQ_TYPE_SPECIAL;
                   ^
    drivers/block/nbd.c:619:19: 附注：每个未声明的标识符在其出现的函数内只报告一次
    make[1]: *** [drivers/block/nbd.o] 错误 1
    make: *** [_module_drivers/block] 错误 2

    解决：

    [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ vim drivers/block/nbd.c
    将sreq.cmd_type = REQ_TYPE_SPECIAL;最后那个宏是7，可以改成： sreq.cmd_type = 7; 然后继续编译

• 加载nbd内核模块
  • 复制nbd.ko到内核模块目录下

    [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ cp drivers/block/nbd.ko /lib/modules/3.10.0-514.26.2.el7.x86_64/extra/

  • 加载nbd模块

    [root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]$ depmod -a && modprobe nbd

• 查看nbd模块

[root@ceph-node1 linux-3.10.0-514.el7.centos.x86_64]# modinfo nbd
  filename:       /lib/modules/3.10.0-514.26.2.el7.x86_64/extra/nbd.ko
  license:        GPL
  description:    Network Block Device
  rhelversion:    7.3
  srcversion:     7CFF86AC0ED31FD6EC44EA5
  depends:        
  vermagic:       3.10.0 SMP mod_unload modversions 
  parm:           nbds_max:number of network block devices to initialize (default: 16) (int)
  parm:           max_part:number of partitions per device (default: 0) (int)
  parm:           debugflags:flags for controlling debug output (int)

附

• 原来的crush tunables

[root@ceph-node1 ~]# ceph osd crush show-tunables
{
    "choose_local_tries": 0,
    "choose_local_fallback_tries": 0,
    "choose_total_tries": 50,
    "chooseleaf_descend_once": 1,
    "chooseleaf_vary_r": 1,
    "chooseleaf_stable": 0,
    "straw_calc_version": 1,
    "allowed_bucket_algs": 22,
    "profile": "firefly",
    "optimal_tunables": 0,
    "legacy_tunables": 0,
    "minimum_required_version": "firefly",
    "require_feature_tunables": 1,
    "require_feature_tunables2": 1,
    "has_v2_rules": 0,
    "require_feature_tunables3": 1,
    "has_v3_rules": 0,
    "has_v4_buckets": 0,
    "require_feature_tunables5": 0,
    "has_v5_rules": 0
}

• 调整crush tunables为jewel

[root@ceph-node1 ~]# ceph osd crush tunables jewel
adjusted tunables profile to jewel
[root@ceph-node1 ~]# ceph osd crush show-tunables
{
    "choose_local_tries": 0,
    "choose_local_fallback_tries": 0,
    "choose_total_tries": 50,
    "chooseleaf_descend_once": 1,
    "chooseleaf_vary_r": 1,
    "chooseleaf_stable": 1,
    "straw_calc_version": 1,
    "allowed_bucket_algs": 54,
    "profile": "jewel",
    "optimal_tunables": 1,
    "legacy_tunables": 0,
    "minimum_required_version": "jewel",
    "require_feature_tunables": 1,
    "require_feature_tunables2": 1,
    "has_v2_rules": 0,
    "require_feature_tunables3": 1,
    "has_v3_rules": 0,
    "has_v4_buckets": 0,
    "require_feature_tunables5": 1,
    "has_v5_rules": 0
}

• 创建rbd image

[root@ceph-node1 ~]# rbd create crush-jewel --size 10G

[root@ceph-node1 ~]# rbd info crush-jewel
rbd image 'crush-jewel':
	size 10240 MB in 2560 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.6c6f16b8b4567
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags:

• 直接使用rbd-nbd map

[root@ceph-node1 ~]# rbd-nbd map crush-jewel
/dev/nbd1

[root@ceph-node1 ~]# mkfs.xfs /dev/nbd1
meta-data=/dev/nbd1              isize=512    agcount=4, agsize=655360 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0