ceph rbd

Before we begin, some notes:

Current CEPH version: 0.67 (“dumpling”).
OS: Centos 6.4 x86_64 (running some VMs on KVM, basic CentOS qemu packages, nothing custom)

Since CEPH RBD module was first introduced with kernel 2.6.34 (and current RHEL/CentOS kernel is 2.6.32) – that means we need a newer kernel.

So, one of the options for the new kernel is, 3.x from elrepo.org:

rpm –import http://elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm
yum –enablerepo=elrepo-kernel install kernel-ml # will install 3.11.latest, stable, mainline
# or
yum –enablerepo=elrepo-kernel install kernel-lt # will install 3.0.latest, long term supported
If you want that new kernel to boot by default, edit /etc/grub.conf, and change the Default=1 to Default=0, and reboot.

That’s for the Kernel side.

If you want to use qemu with RBD, you will also need newer qemu packages – CEPH does provide some “untested”, “use it at your own risk” packages, that are rhel/centos based verions, just with RBD support added – well, running them on productional cloud, and all seems fine so far for me…

wget http://ceph.com/packages/ceph-extras/rpm/rhel6/x86_64/qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.x86_64.rpm
wget http://ceph.com/packages/ceph-extras/rpm/rhel6/x86_64/qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.x86_64.rpm
wget http://ceph.com/packages/ceph-extras/rpm/rhel6/x86_64/qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.x86_64.rpm

After done, in order to actually install those CEPH-provided versions of binaries, we need to download and install CEPH, that is the required libraries librados and librbd:

rpm --import 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'
rpm -Uvh http://ceph.com/rpm-dumpling/el6/noarch/ceph-release-1-0.el6.noarch.rpm
yum install ceph

If all fine (will not talk here about creating the /etc/ceph.conf file and keyring for the client – maybe another post), then we are ready to install CEPH version of qemu packages:

First, we need to remove the original packages from Centos 6, then install the new ones:

rpm -e --nodeps qemu-img
rpm -e --nodeps qemu-kvm
rpm -e --nodeps qemu-kvm-tools
rpm -Uvh qemu-*

And… that’s it. If you run the qemu-img | grep “Supported formats” you will see RBD as being one od those.
You may shutdown and restart all your existing VMs, in order to load the new qemu binaries.

There should be rbd codes in the running kernel. Check it by "cat
/boot/config-`uname -r` | grep CONFIG_BLK_DEV_RBD". Result should be
"CONFIG_BLK_DEV_RBD=m" or "CONFIG_BLK_DEV_RBD=y".

[root@newosd1 kernel]# cat /boot/config-3.10.0-123.el7.x86_64 |grep RBD
# CONFIG_BLK_DEV_DRBD is not set
# CONFIG_BLK_DEV_RBD is not set

[root@newosd1 3.10.0-123.el7.x86_64]# yum install kernel-devel-3.10.0-123.el7.x86_64

http://ceph.com/packages/

编译linux内核：$make && make modules && make modules_install && make install && reboot,参考：http://blog.csdn.net/polisan/article/details/5624207

http://cephnotes.ksperis.com/blog/2013/09/12/using-ceph-rbd-with-libvirt-on-debian-wheezy

Use

Create a pool dans associated key on ceph admin host :

ceph osd pool create vmimages 100 100
ceph auth get-or-create client.vmimages mon 'allow r' osd 'allow rwx pool=vmimages'

keep the key returned.

Create a secret file for libvirt for this user :

echo "
    <secret ephemeral='no' private='no'>
   <usage type='ceph'>
     <name>client.vmimages secret</name>
   </usage>
</secret>" > secret.xml

virsh secret-define --file secret.xml

Define the key value : first uuid is the value return by the previous command follow by the key return by ceph auth get-or-create

virsh secret-set-value  76e3a541-b997-58ac-f7bd-77dd7d4347cb AQAREH1QkNDNCBaac03ZICi/CePnRDS+vGyrqQ==

Create a device:

(on ceph admin host if you do not want to specify the user, the key, and monitors) with rbd : rbd create vmimages/ubuntu-newdrive --size=2048 with qemu (you need to add support for that) : qemu-img create -f rbd rbd:vmimages/ubuntu-newdrive 2G

Attach on existing host (even if running) :

(192.168.0.100, 192.168.0.101, 192.168.0.102 is the mon host.)

echo "
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw'/>
      <auth username='vmimages'>
        <secret type='ceph' uuid='76e3a541-b997-58ac-f7bd-77dd7d4347cb'/>
      </auth>
      <source protocol='rbd' name='vmimages/ubuntu-newdrive'>
        <host name='192.168.0.100' port='6789'/>
        <host name='192.168.0.101' port='6789'/>
        <host name='192.168.0.102' port='6789'/>
      </source>
      <target dev='vdz' bus='virtio'/>
    </disk>
" > device.xml

virsh attach-device ubuntu device.xml --persistent