Kubernetes ceph-csi 环境搭建

在kubernetes上搭建ceph csi环境

本文旨在说明如何在kubernetes集群中使用ceph-csi (版本 v2.0)做kubernetes的存储，官方的搭建的说明ceph-csi环境搭建
值得注意的是(官方的说明文档里面有)：ceph-csi使用的是RBD的默认kernel模块，这个模块可能不支持所有的Ceph CRUSH或者RBD镜像的特征。

需要的环境

• kubernetes (实验中使用的是1.15版本)
• ceph 集群(实验中使用的是nautious版本)
  • ceph luminous集群也可以，但是要修改部分命令

搭建过程

创建一个ceph pool

[root@node1 ~]# ceph osd pool create kubernetes
[root@node1 ~]# rbd pool init kubernetes

编辑CEPH-CSI

为kubernetes和ceph-csi创建一个新的用户

[root@node1 ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
    key = AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==

• 注意：这里key后面对应的只是一个例子，实际配置中要以运行命令后产生的结果为准
• 这里的key使用user的key，后面配置中是需要用到的
• 如果是ceph luminous版本的集群，那么命令应该是ceph auth get-or-create client.kubernetes mon 'allow r' osd 'allow rwx pool=kubernetes' -o ceph.client.kubernetes.keyring

生成ceph-csi的kubernetes configmap

[root@node1 ~]# ceph mon dump
dumped monmap epoch 1
epoch 1
fsid 82e65362-72f7-4206-b94d-04e2c5e2d70d
last_changed 2020-05-29 05:26:42.313889
created 2020-05-29 05:26:42.313889
min_mon_release 14 (nautilus)
0: [v2:10.80.0.41:3300/0,v1:10.80.0.41:6789/0] mon.node1

• 注意，这里一共有两个需要使用的信息，第一个是fsid(可以称之为集群id)，第二个是监控节点信息0: [v2:10.80.0.41:3300/0,v1:10.80.0.41:6789/0] mon.node1(可能有多个，实验中只配置了1个)
• 另外，目前的ceph-csi只支持V1版本的协议，所以监控节点那里我们只能用v1的那个IP和端口号

用以上的的信息生成configmap：

[root@node1 ~]#  cat < csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "82e65362-72f7-4206-b94d-04e2c5e2d70d",
        "monitors": [
          "10.80.0.41:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
EOF

在kubernetes集群上，将此configmap存储到集群
kubectl apply -f csi-config-map.yaml

生成ceph-csi cephx的secret

[root@node1 ~]#  cat < csi-rbd-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
stringData:
  userID: kubernetes
  userKey: AQD9o0Fd6hQRChAAt7fMaSZXduT3NWEqylNpmg==
EOF

• 这里就用到了之前生成的用户的用户id(kubernetes)和key

将此配置存储到kubernetes中
kubectl apply -f csi-rbd-secret.yaml

配置ceph-csi插件(kubernetes上的rbac和提供存储功能的容器)

• rbac部分

  kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-provisioner-rbac.yaml
  kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-nodeplugin-rbac.yaml
  

• provisioner部分

  $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
  $ kubectl apply -f csi-rbdplugin-provisioner.yaml
  $ wget https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml
  $ kubectl apply -f csi-rbdplugin.yaml
  

  • 这里的两个yaml文件默认都会去拉取镜像quay.io/cephcsi/cephcsi:canary，如果要是用其他版本，请自行修改yaml文件

使用ceph块儿设备

创建storageclass

$ cat < csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: 82e65362-72f7-4206-b94d-04e2c5e2d70d
   imageFeatures: layering
   pool: kubernetes
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
mountOptions:
   - discard
EOF
$ kubectl apply -f csi-rbd-sc.yaml

• 第一，这里的clusterID对应之前的步骤中的fsid
• 第二，这里新增了一项，imageFeatures，这个是用来确定创建的image的特征的，这个默认是用的rbd内核中的特征列表，但是我们用的linux内核不一定支持所有的特征
• storageclass的yaml文件比较全的文件csi-ceph-sc.yaml

  ---
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
       name: csi-rbd-sc
    provisioner: rbd.csi.ceph.com
    # If topology based provisioning is desired, delayed provisioning of
    # PV is required and is enabled using the following attribute
    # For further information read TODO
    # volumeBindingMode: WaitForFirstConsumer
    parameters:
       # String representing a Ceph cluster to provision storage from.
       # Should be unique across all Ceph clusters in use for provisioning,
       # cannot be greater than 36 bytes in length, and should remain immutable for
       # the lifetime of the StorageClass in use.
       # Ensure to create an entry in the config map named ceph-csi-config, based on
       # csi-config-map-sample.yaml, to accompany the string chosen to
       # represent the Ceph cluster in clusterID below
       clusterID: 
       # If you want to use erasure coded pool with RBD, you need to create
       # two pools. one erasure coded and one replicated.
       # You need to specify the replicated pool here in the `pool` parameter, it is
       # used for the metadata of the images.
       # The erasure coded pool must be set as the `dataPool` parameter below.
       # dataPool: ec-data-pool
       pool: rbd
    
       # RBD image features, CSI creates image with image-format 2
       # CSI RBD currently supports only `layering` feature.
       imageFeatures: layering
    
       # The secrets have to contain Ceph credentials with required access
       # to the 'pool'.
       csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
       csi.storage.k8s.io/provisioner-secret-namespace: default
       csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
       csi.storage.k8s.io/controller-expand-secret-namespace: default
       csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
       csi.storage.k8s.io/node-stage-secret-namespace: default
       # Specify the filesystem type of the volume. If not specified,
       # csi-provisioner will set default as `ext4`.
       csi.storage.k8s.io/fstype: ext4
       # uncomment the following to use rbd-nbd as mounter on supported nodes
       # mounter: rbd-nbd
    
       # Prefix to use for naming RBD images.
       # If omitted, defaults to "csi-vol-".
       # volumeNamePrefix: "foo-bar-"
    
       # Instruct the plugin it has to encrypt the volume
       # By default it is disabled. Valid values are "true" or "false".
       # A string is expected here, i.e. "true", not true.
       # encrypted: "true"
    
       # Use external key management system for encryption passphrases by specifying
       # a unique ID matching KMS ConfigMap. The ID is only used for correlation to
       # config map entry.
       # encryptionKMSID: 
    
       # Add topology constrained pools configuration, if topology based pools
       # are setup, and topology constrained provisioning is required.
       # For further information read TODO
       # topologyConstrainedPools: |
       #   [{"poolName":"pool0",
       #     "dataPool":"ec-pool0" # optional, erasure-coded pool for data
       #     "domainSegments":[
       #       {"domainLabel":"region","value":"east"},
       #       {"domainLabel":"zone","value":"zone1"}]},
       #    {"poolName":"pool1",
       #     "dataPool":"ec-pool1" # optional, erasure-coded pool for data
       #     "domainSegments":[
       #       {"domainLabel":"region","value":"east"},
       #       {"domainLabel":"zone","value":"zone2"}]},
       #    {"poolName":"pool2",
       #     "dataPool":"ec-pool2" # optional, erasure-coded pool for data
       #     "domainSegments":[
       #       {"domainLabel":"region","value":"west"},
       #       {"domainLabel":"zone","value":"zone1"}]}
       #   ]
    
    reclaimPolicy: Delete
    allowVolumeExpansion: true
    mountOptions:
       - discard
  

• 这里有必要讲一下ceph的feature
  • 实验所使用的ceph集群是基于centos7.5的，其linux内核版本为3.10，默认只支持ceph layering 特征
    在没有指定imageFeature时，去申请pvc，会产生如下bug

    Warning  FailedMapVolume         0s (x4 over 5s)  kubelet, k8s-1           MapVolume.SetUp failed for volume "pvc-5340f32e-fa2e-4f53-af1d-7154430af7a2" : rpc error: code = Internal desc = rbd: map failed exit status 6, rbd output: rbd: sysfs write failed
    RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
    In some cases useful info is found in syslog - try "dmesg | tail".
    rbd: map failed: (6) No such device or address
    

    ceph nautious默认的image需要支持的特征有features: layering, exclusive-lock, object-map, fast-diff, deep-flatten; linux内核3.10版本只支持layering，所以这里有必要限制一下，当然我们也可以升级linux内核版本，不过这个代价就比较大了

创建PVC

$ cat < raw-block-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF
$ kubectl apply -f raw-block-pvc.yaml

创建使用PVC的Pod

$ cat < raw-block-pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
    - name: fc-container
      image: fedora:26
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc
EOF
$ kubectl apply -f raw-block-pod.yaml

基于Ceph filesystem的pvc和pod

• pvc

  $ cat < pvc.yaml
  ---
  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: rbd-pvc
  spec:
    accessModes:
      - ReadWriteOnce
    volumeMode: Filesystem
    resources:
      requests:
        storage: 1Gi
    storageClassName: csi-rbd-sc
  EOF
  $ kubectl apply -f pvc.yaml
  

• pod

  $ cat < pod.yaml
  ---
  apiVersion: v1
  kind: Pod
  metadata:
    name: csi-rbd-demo-pod
  spec:
    containers:
      - name: web-server
        image: nginx
        volumeMounts:
          - name: mypvc
            mountPath: /var/lib/www/html
    volumes:
      - name: mypvc
        persistentVolumeClaim:
          claimName: rbd-pvc
          readOnly: false
  EOF
  $ kubectl apply -f pod.yaml
  

后续遇到的问题

ceph-csi 2.1 3.0版本部署时遇到的新问题

• 新版本的ceph-csi中需要一个ConfigMap，其名为ceph-csi-encryption-kms-config
• 解决方案：
  • 找到需要用到它的yaml，注释掉所有相关引用
    • 引用了该ConfigMap的yaml文件
      deploy/rbd/kubernetes/csi-rbdplugin-provisioner.yaml
      deploy/rbd/kubernetes/csi-rbdplugin.yaml
  • 部署一个
    • 样例路径ceph-csi/examples/kms/vault/kms-config.yaml

ceph-csi 使用过程中出现的"driver name rbd.csi.ceph.com not found in the list of registered CSI drivers"

• 遇到的问题描述
  部署完成ceph-csi之后，携带的例子也跑通了，继续部署对master节点亲和性100%的pod，结果pod无法mount通过ceph-csi申请的PV
• 问题原因
  ceph-csi官方的deploy目录下的daemonset的配置中，默认是不允许在master节点上部署pod的，这样导致，master节点上通过ceph-csi申请volume的pod，可以申请到PV但却无法挂载到pod
• 解决方案
  在ceph-csi/deploy/rbd/kubernetes/csi-rbdplugin.yaml中的DaemonSet的spec中添加如下配置：

  tolerations:
     - key: node-role.kubernetes.io/master
       effect: NoSchedule
  

  示例如下：