K8S：以sotrageclass方式部署cephfs为集群后端存储
一、事前准备：
①、K8S集群运行正常。
[root@k8s01 rbac]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s01 Ready <none> 10d v1.16.6
k8s02 Ready <none> 10d v1.16.6
k8s03 Ready <none> 10d v1.16.6
②、ceph集群运行正常，并配置好了cephfs。
[root@k8s01 rbac]# ceph -s
cluster:
id: b5f36dec-8faa-4efa-b08d-cbcd8305ae63
health: HEALTH_WARN
1 monitors have not enabled msgr2
services:
mon: 3 daemons, quorum k8s01,k8s03,k8s04 (age 2d)
mgr: k8s01(active, since 118m)
mds: cephfs:1 {0=k8s01=up:active}
osd: 3 osds: 3 up (since 2d), 3 in (since 3d)
task status:
scrub status:
mds.k8s01: idle
data:
pools: 3 pools, 81 pgs
objects: 29 objects, 161 KiB
usage: 3.2 GiB used, 237 GiB / 240 GiB avail
pgs: 81 active+clean
③、社区提供的cephfs-provisioner，地址：https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs
官方没有cephfs动态卷支持
使用社区提供的cephfs-provisioner
#pwd
opt/k8s/work/kubernetes/cluster/addons/ceph-storage-class/external-storage/ceph/cephfs/deploy/rbac
[root@k8s01 rbac]# ls -l
total 40
-rw-r--r-- 1 root root 279 Jul 27 14:40 ceph-deployment.yaml
-rw-r--r-- 1 root root 148 Jul 27 14:38 ceph-secret.yaml
-rw-r--r-- 1 root root 283 Jul 27 14:14 clusterrolebinding.yaml
-rw-r--r-- 1 root root 652 Jul 27 14:14 clusterrole.yaml
-rw-r--r-- 1 root root 708 Jul 27 14:14 deployment.yaml
-rw-r--r-- 1 root root 263 Jul 27 14:14 rolebinding.yaml
-rw-r--r-- 1 root root 316 Jul 27 14:14 role.yaml
-rw-r--r-- 1 root root 93 Jul 27 14:14 serviceaccount.yaml
-rw-r--r-- 1 root root 499 Jul 27 14:35 test-pvc.yaml
-rw-r--r-- 1 root root 294 Jul 27 14:47 test.yaml
二、部署cephfs（storageclass）
先下载社区提供的cephfs-provisioner到本地，或可以通过百度网盘下载！
①、创建cephfs-provisioner
#kubectl create ns cephfs
#创建namespace
#kubectl -n apply -f ./
②、创建ceph-secret
在创建k8s pv前,由于ceph 集群是开启了cephx认证的,于是首先需要k8s 创建secret资源，k8s的secret资源采用的是base64加密。用下列命令在ceph monitor上提取key:
[root@k8s01 rbac]# ceph auth get-key client.admin |base64
QVFCL3E1ZGIvWFdxS1JBQTUyV0ZCUkxldnRjQzNidTFHZXlVYnc9PQ==
[root@k8s01 rbac]# cat ceph-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret #secret名称
namespace: cephfs #名称空间
data:
key: QVFBeVFCbGZCOS9uTlJBQTdvWDlBZEtxcDBReG10VTgrVmN6aUE9PQ==
[root@k8s01 rbac]# kubectl apply -f ceph-secret.yaml
③、创建storageclass
[root@k8s01 rbac]# cat ceph-deployment.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cephfs
namespace: cephfs
provisioner: ceph.com/cephfs
parameters:
monitors: 172.16.1.11:6789,172.16.1.13:6789,172.16.1.14:6789 #ceph服务列表
adminId: admin #认证用户
adminSecretName: ceph-secret #admin的secret
adminSecretNamespace: cephfs
[root@k8s01 rbac]# kubectl apply -f ceph-deployment.yaml
④、测试cephfs作为后端存储
[root@k8s01 rbac]# cat test-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: claim #PVC名称 pod 关联改名称
namespace: cephfs
spec:
accessModes:
- ReadWriteMany
storageClassName: cephfs
resources:
requests:
storage: 1Gi
---
kind: Pod
apiVersion: v1
metadata:
name: test-pod
namespace: cephfs
spec:
containers:
- name: test-pod
image: ikubernetes/myapp:v4
volumeMounts:
- name: pvc
mountPath: "/data/cephfs" #pod 内挂载路径
volumes:
- name: pvc
persistentVolumeClaim:
claimName: claim
⑤、验证
[root@k8s01 rbac]# kubectl get storageclass
NAME PROVISIONER AGE
cephfs ceph.com/cephfs 82m
[root@k8s01 rbac]# kubectl get pvc
No resources found in default namespace.
[root@k8s01 rbac]# kubectl get pvc -n cephfs
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
claim Bound pvc-d2bf4d1c-0f9b-409e-b902-bcd0b6087658 1Gi RWX cephfs 67m
[root@k8s01 rbac]# kubectl get pod -n cephfs
NAME READY STATUS RESTARTS AGE
cephfs-provisioner-7b77478cb8-687rv 1/1 Running 0 95m
test-pod 1/1 Running 0 63m
资源：
参考：https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs
参考博客：https://blog.csdn.net/weixin_42715413/article/details/89513385
参考博客：https://blog.csdn.net/ywq935/article/details/103850257
下载速度慢，已上传至百度网盘。
链接：https://pan.baidu.com/s/1PNali5-YwQd_K_77Yos5RQ
提取码：vmi5