添加ceph osd，创建rbd pool及rbd设备，配置cinder后端

1.   为ceph新增OSD节点

 

新增两个OSD，使用ceph-deploy创建的osd每个只有5G大小，调试起来肯定不方便

原来的OSD是：

 

osd.0

osd.1

 

现在增加osd.2、osd.3：

 

Osd编号	所在osd节点	对应磁盘	大小
Osd.0	10.33.41.136/192.168.11.5	/dev/vdb	5G
Osd.1	10.33.41.139/192.168.11.6	/dev/vdb	5G
Osd.2	10.33.41.136/192.168.11.5	/dev/vdc	20G
Osd.3	10.33.41.139/192.168.11.6	/dev/vdc	20G

 

（1）扩展10.33.41.55上的cinder-volumes大小

对cinder-volumes扩展128G内存

#pvcreate /dev/sda3

#vgextend cinder-volumes /dev/sda3

（2）新增两个云硬盘，挂载到10.33.41.136、10.33.41.139上

（3）新增osd.2（10.33.41.136）、osd.3（10.33.41.139）

增加过程参考：

http://docs.ceph.org.cn/start/quick-ceph-deploy/

 

我的添加过程：

在10.33.41.136：

（1）先把vdc分区，做一个vdc1

vdc   253:32   0    20G 0 disk

└─vdc1 253:33   0    20G 0 part

（2）然后将/dev/vdc1格式化为ext4

# mkfs.ext4 /dev/vdc1

（3）然后将/dev/vdc1挂载到/var/vdc1

# mount -t ext4 /dev/vdc1 /var/osd2

（4）然后prepare

# ceph-deploy  osd prepare lxpnode2:/var/osd2

（5）激活osd

# ceph-deploy  osd activate lxpnode2:/var/osd2

（6）查看

# ceph -s

   cluster 2a5f361c-2b64-4512-b710-719409f05f10

    health HEALTH_OK

    monmap e1: 1 mons at {lxpnode1=192.168.11.4:6789/0}

           election epoch 2, quorum 0 lxpnode1

    osdmap e23: 3 osds: 3 up, 3 in

     pgmap v1908: 252 pgs, 3 pools, 12914 kB data, 7 objects

           8067 MB used, 47944 MB / 59080 MB avail

                 252 active+clean

recovery io 1700 kB/s, 1 keys/s, 0objects/s

 

# ceph osd stat

    osdmap e23: 3 osds: 3 up, 3 in

#ceph osd dump

epoch 23

fsid 2a5f361c-2b64-4512-b710-719409f05f10

created 2016-04-28 09:32:07.631465

modified 2016-05-04 13:49:31.018699

flags

pool 0 'rbd' replicated size 2 min_size 1crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flagshashpspool stripe_width 0

pool 1 'rbd_pool' replicated size 2min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 60 pgp_num 60last_change 10 flags hashpspool stripe_width 0

pool 2 'images' replicated size 2 min_size1 crush_ruleset 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 15flags hashpspool stripe_width 0

       removed_snaps [1~1]

max_osd 3

 

同样的方法新增10.33.41.139上的/dev/vdc为新的osd3

 

操作之后：

root@lxpnode1:/etc/ceph# ceph -s

   cluster 2a5f361c-2b64-4512-b710-719409f05f10

    health HEALTH_OK

    monmap e1: 1 mons at {lxpnode1=192.168.11.4:6789/0}

           election epoch 2, quorum 0 lxpnode1

     osdmap e33: 4 osds: 4up, 4 in

     pgmap v1941: 252 pgs, 3 pools, 12914 kB data, 7 objects

           13233 MB used, 61767 MB / 79110 MB avail

                 252 active+clean

recovery io 1756 kB/s, 0 objects/s

 

然后创建新的rbd dev，大小为10G，作为cinder的后端

 

 

 

2.   创建rbd pool和rbd blk设备

# ceph osd pool create cinder 100

# ceph osd lspools

0 rbd,1 rbd_pool,2 images,3 cinder,

# rbd create --size 18432 cinder/myblk  这里保守一点，使用18G

# rbd ls cinder

myblk

# rbd map myblk -p cinder

# lsblk

NAME  MAJ:MIN RM   SIZE RO TYPEMOUNTPOINT

vda   253:0    0    20G 0 disk

├─vda1 253:1    0  19.5G 0 part /

├─vda2 253:2    0     1K 0 part

└─vda5 253:5    0   510M 0 part [SWAP]

rbd1  251:0    0    4G  0 disk

rbd2   250:0    0   18G  0 disk

 

3.   将rbd配置为cinder后端

（1）拷贝/etc/ceph到openstack环境

然后openstack和ceph节点中的hosts都改一下

192.168.11.8 ubuntu

192.168.11.4 lxpnode1

192.168.11.5 lxpnode2

192.168.11.6 lxpnode3

~                     

（2）cinder.conf

[DEFAULT]

enabled_backends = ceph

 

[ceph]

volume_driver =cinder.volume.drivers.rbd.RBDDriver

rbd_pool = cinder    //上面创建的rbd pool

rbd_ceph_conf = /etc/ceph/ceph.conf

rbd_flatten_volume_from_snapshot = false

rbd_max_clone_depth = 5

rbd_store_chunk_size = 4

rados_connect_timeout = -1

glance_api_version = 2

（3）重启cinder-volume服务