ceph块存储和K8S集成报错汇总

1. kubectl get pvc一直处于pending状态

$ kubectl get pvc
NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
testclaim   Pending                                      fast-rbd       5m23s

解决思路:
查看testclaim的具体描述:

$  kubectl describe pvc testclaim
Name:          testclaim
Namespace:     default
StorageClass:  fast-rbd
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type     Reason                Age                  From                                                                               Message
  ----     ------                ----                 ----                                                                               -------
  Normal   Provisioning          17s (x5 over 4m5s)   ceph.com/rbd_rbd-provisioner-c968dcb4b-cm45p_fa9ff5b1-8091-11ea-8e48-6efb4972793f  External provisioner is provisioning volume for claim "default/testclaim"
  Warning  ProvisioningFailed    17s (x5 over 4m5s)   ceph.com/rbd_rbd-provisioner-c968dcb4b-cm45p_fa9ff5b1-8091-11ea-8e48-6efb4972793f  failed to provision volume with StorageClass "fast-rbd": failed to get admin secret from ["kube-system"/"ceph-secret"]: secrets "ceph-secret" not found
  Normal   ExternalProvisioning  10s (x17 over 4m5s)  persistentvolume-controller                                                        waiting for a volume to be created, either by external provisioner "ceph.com/rbd" or manually created by system administrator

看到报错原因

 failed to get admin secret from ["kube-system"/"ceph-secret"]: secrets "ceph-secret" not found

解决办法

# 1. 将用户更改为ceph用户， 然后执行ceph auth -get-key client.admin
$ su cephadm
$ ceph auth get-key client.admin
AQDaMZVeREAwBxAA2nDczMFt3E98kDqbWTio3w==

# 2. 在k8s中创建管理的秘钥
$ kubectl create secret generic ceph-secret \
     --type="kubernetes.io/rbd" \
     --from-literal=key='AQDaMZVeREAwBxAA2nDczMFt3E98kDqbWTio3w==' \
     --namespace=kube-system

# 3. 删除旧的pvc重新创建
$ kubectl delete -f ceph-RBD-PVC.yaml
$ kubectl create -f Ceph-RBD-PVC.yaml

成功解决问题！