kubernetes的ceph RBD volume(1)：使用Ceph RBD作为后端Volume
Kubernetes使用Ceph RBD作为后端Volume。 Kubernetes的官方源码 的examples/volumes/rbd目录下，就有一个使用cephrbd作为kubernetes pod volume的例子，我们可以参考
1. 当ceph集群安装完成以后，我们就要创建相应的rbd块用于kubernetes存储。创建块设备之前，需要先创建存储池
ceph osd pool create kube 256 256 #后面两个256分别为pg-num和pgp-num
ceph osd pool ls detail
pool 4 'kube' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 2410 flags hashpspool stripe_width 0
removed_snaps [1~3]
2. 在kube存储池创建一个映像文件，就叫vol50，该映像文件的大小为50GB：
rbd create kube/vol50 --size 50000
rbd -p kube info vol50
rbd image 'vol50':
size 51200 MB in 12800 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.754829.238e1f29
format: 1
3. 创建用户client.kube 用admin用户
ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read class-write object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring
通常我们在ceph install时在ceph.conf中使用默认的安全验证协议 cephx – The Ceph authentication protocol 了。
4. 生成secret
得到key(base64)
grep key /etc/ceph/ceph.client.kube.keyring |awk '{printf "%s", $NF}'|base64
QVFCK0l4RlpqK0xDTkJBQTRKYVBPTWx6WkFIVVhLK2ozM2lQdUE9PQo=
写secret.yaml：
[root@testnew rbd]# cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
type: "kubernetes.io/rbd"
data:
key: QVFCK0l4RlpqK0xDTkJBQTRKYVBPTWx6WkFIVVhLK2ozM2lQdUE9PQo=
copy /etc/ceph/ceph.client.kube.keyring和ceph.conf到kubernetes的所有节点
kubectl create -f ceph-secret.yaml
[root@testnew ~]# kubectl get secret
NAME TYPE DATA AGE
ceph-secret kubernetes.io/rbd 1 13d
5. 格式化一个空image
格式化一个空image那样对其进行格式化了，这里格成ext4文件系统（格式化这一步可以不需要）
rbd map kube/vol50
rbd info kube/vol50
mkfs.ext4 /dev/rbd0
rbd unmap /dev/rbd0
6. 创建pod with RBD
[root@testnew kube]# cat frontend-rbd-controller.yaml
apiVersion: v1
kind: ReplicationController
metadata:
name: frontendrbd1
labels:
name: frontendrbd1
spec:
replicas: 1
selector:
name: frontendrbd1
template:
metadata:
labels:
name: frontendrbd1
spec:
containers:
- name: frontendrbd1
image: kubeguide/guestbook-php-frontend
env:
- name: GET_HOSTS_FROM
value: env
ports:
- containerPort: 80
volumeMounts:
- mountPath: /mnt/rbd
name: rbdpb
volumes:
- name: rbdpb
rbd:
monitors:
- 10.0.200.11:6789
- 10.0.200.13:6789
- 10.0.200.14:6789
pool: kube
image: vol50
user: kube
secretRef:
name: ceph-secret
fsType: ext4
readOnly: false
kubectl create -f frontend-rbd-controller.yaml
[root@testnew ~]# kubectl get rc
NAME DESIRED CURRENT READY AGE
frontendrbd1 1 1 1 13d
[root@testnew ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
frontendrbd1-h9z78 1/1 Running 1 13d
7. 验证volume在container里。
[root@testnew ~]# kubectl exec frontendrbd1-h9z78 -it bash
root@frontendrbd1-h9z78:/var/www/html# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/docker-253:1-530097-861967a5b3b1a5f40b4880db1921a52af2656a10bf5ce9d1727c40845a4aa9c2 10474496 623084 9851412 6% /
tmpfs 4087712 0 4087712 0% /dev
tmpfs 4087712 0 4087712 0% /sys/fs/cgroup
/dev/rbd0 51474912 16840936 31996152 35% /mnt/rbd
/dev/vda1 19593296 15144980 3577460 81% /etc/hosts
shm 65536 0 65536 0% /dev/shm