说明
官方的部署文档中说明，使用cinder csi需要OpenStack作为Kubernetes Provider（这点还是很限制cinder在k8s中的使用）。因为对OpenStack不了解（也对“需要OpenStack作为Kubernetes Provider”不太理解），我直接在自己的k8s测试集群中部署cinder csi，csi驱动的容器都能正常启，创建pvc也可以动态绑定pv（cinder 的volume可以创建成功），但是创建pod时，csi controller attach这个volume到node上时失败了。原因是：自己本地搭建k8s的话，它申请的vloume资源 openstack没办法挂载到k8s服务器。下面记录下OpenStack作为Provider的k8s如何使用cinder。
环境
可以按官方文档搭建OpenStack和k8s集群。我用OpenStack是虚拟化团队创建的。k8s版本v1.18.3，CSI Cinder driver release.1.17
部署cinder csi
- clonecloud-provider-openstack项目
[root@kubernetes-master home]# git clone -b release-1.17 https://github.com/kubernetes/cloud-provider-openstack.git
[root@kubernetes-master home]# cd cloud-provider-openstack/manifests/cinder-csi-plugin
[root@kubernetes-master cinder-csi-plugin]# ls -lrt
total 28
-rw-r--r--. 1 root root 199 Jul 14 10:11 csi-cinder-driver.yaml
-rw-r--r--. 1 root root 3467 Jul 14 10:11 cinder-csi-nodeplugin.yaml
-rw-r--r--. 1 root root 694 Jul 14 10:11 cinder-csi-nodeplugin-rbac.yaml
-rw-r--r--. 1 root root 3545 Jul 14 10:11 cinder-csi-controllerplugin.yaml
-rw-r--r--. 1 root root 5555 Jul 14 10:11 cinder-csi-controllerplugin-rbac.yaml
-rw-r--r--. 1 root root 800 Jul 14 10:20 csi-secret-cinderplugin.yaml
- 根据openstack接口信息配置cloud.conf, 放到cat /etc/kubernetes/cloud.conf
[root@kubernetes-master cinder-csi-plugin]# cat /etc/kubernetes/cloud.conf
[Global]
username = admin
password = ZE0LNthHkgfS7M13KjA6aBdZ4A1x3w3nd
domain-name = default
auth-url = http://10.145.69.61:35357/v3
tenant-id = 6441367ec4ae4ce39e5f265dd
region = bjcc_region_2
- 修改csi-secret-cinderplugin.yam
- 在OpenStack中，compute实例使用配置驱动器或元数据服务来检索特定于实例的数据，所以按照官方文档我们将[Metadata]节加到cloud.conf（为了给csi驱动使用，所以不需要更改我们之前创建的/etc/kubernetes/cloud.conf）
[root@kubernetes-master home]# cat cloud.conf
[Global]
username = admin
password = ZE0LNthHkgfS7M13KjA6aBdZ4A1x3w3nd
domain-name = default
auth-url = http://10.145.69.61:35357/v3
tenant-id = 6441367ec4ae4ce39e5f265dd
region = bjcc_region_2
[Metadata]
search-order = configDrive,metadataService
- 对cloud.conf做base64
[root@kubernetes-master home]# cat cloud.conf | base64 -w 0
W0dsb2JhbF0KdXNlcm5hbWUgPSBhZG1pbgpwYXNzd29yZCA9IFpFMExOdGhIa2dmUzdNMTNLakE2YUJkWjRBMXgzdzNuZE5VS2x1WU8KZG9tYWluLW5hbWUgPSBkZWZhdWx0CmF1dGgtdXJsID0gaHR0cDovLzEwLjE0NS42OS42OjM1MzU3L3YzCnRlbmFudC1pZCA9IDY0NDEzNjdlYzRhZTRjZTM5ZTVmMjY1ZGQwMWRiY2IxCnJlZ2lvbiA9IGJqY2NfcmVnaW9uXzIKCltNZXRhZGF0YV0Kc2VhcmNoLW9yZGVyID0gY29uZmlnRHJpdmUsbWV0YWRhdGFTZXJ2aWNlCg==
- 修改csi-secret-cinderplugin.yaml
[root@kubernetes-master cinder-csi-plugin]# cat csi-secret-cinderplugin.yaml
# This YAML file contains secret objects,
# which are necessary to run csi cinder plugin.
kind: Secret
apiVersion: v1
metadata:
name: cloud-config
namespace: kube-system
data:
#cloud.conf: W0dsb2JhbF0KdXNlcm5hbWUgPSBhZG1pbgpwYXNzd29yZCA9IG5vbW9yZXNlY3JldApkb21haW4tbmFtZSA9IGRlZmF1bHQKYXV0aC11cmwgPSBodHRwOi8vMTkyLjE2OC4yMDAuOS9pZGVudGl0eQp0ZW5hbnQtaWQgPSBjYzM0YjExZmY5NWQ0MjMwOTA4MWQwYmQ0NmMwZmY4OQpyZWdpb24gPSBSZWdpb25PbmUK
cloud.conf: W0dsb2JhbF0KdXNlcm5hbWUgPSBhZG1pbgpwYXNzd29yZCA9IFpFMExOdGhIa2dmUzdNMTNLakE2YUJkWjRBMXgzdzNuZE5VS2x1WU8KZG9tYWluLW5hbWUgPSBkZWZhdWx0CmF1dGgtdXJsID0gaHR0cDovLzEwLjE0NS42OS42OjM1MzU3L3YzCnRlbmFudC1pZCA9IDY0NDEzNjdlYzRhZTRjZTM5ZTVmMjY1ZGQwMWRiY2IxCnJlZ2lvbiA9IGJqY2NfcmVnaW9uXzIKCltNZXRhZGF0YV0Kc2VhcmNoLW9yZGVyID0gY29uZmlnRHJpdmUsbWV0YWRhdGFTZXJ2aWNlCg==
- 修改k8s master参数
- 修改kube-controller-manager参数，并重启
# 启动参数添加两行
--cloud-provider=openstack \
--cloud-config=/etc/kubernetes/cloud.conf
如果是kubeadm 以pod的形式启动的kube-controller-manager，修改/etc/kubernetes/manifests/kube-controller-manager.yaml文件就会自动重建pod
注： 官方已经不推荐修改kube-controller-manager的cloud-provider选项了，以后会废弃。而是直接修改kubelet，由于本人没有测试过，大家可以按照官网跑一把，写的很清楚https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/
- 部署csi
/home/cloud-provider-openstack/manifests/cinder-csi-plugin
[root@kubernetes-master cinder-csi-plugin]# kubectl apply -f .
查看csi pod
[root@kubernetes-master cinder-csi-plugin]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-65dbdb44db-56l6t 1/1 Running 0 13h
csi-cinder-controllerplugin-0 5/5 Running 0 152m
csi-cinder-nodeplugin-6j2l5 2/2 Running 0 152m
csi-cinder-nodeplugin-fx9zq 2/2 Running 0 152m
csi-cinder-nodeplugin-pz2dl 2/2 Running 0 152m
dashboard-metrics-scraper-545bbb8767-5ch8h 1/1 Running 0 13h
kube-flannel-ds-amd64-5qkhq 1/1 Running 8 14h
kube-flannel-ds-amd64-5tmqv 1/1 Running 8 14h
kube-flannel-ds-amd64-z9ntk 1/1 Running 8 14h
kubernetes-dashboard-65665f84db-dgk6g 1/1 Running 0 13h
metrics-server-869ffc99cd-f2b8v 1/1 Running 0 13h
csi driver
[root@kubernetes-master cinder-csi-plugin]# kubectl get csidrivers.storage.k8s.io
NAME ATTACHREQUIRED PODINFOONMOUNT MODES AGE
cinder.csi.openstack.org true true Persistent,Ephemeral 154m
到此cinder csi已经部署好了，下面测试一下
测试
- 使用官方的examples
[root@kubernetes-master cinder-csi-plugin]# kubectl -f examples/cinder-csi-plugin/nginx.yaml create
[root@kubernetes-master cinder-csi-plugin]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 135m
- 查看pvc是否bound
[root@kubernetes-master cinder-csi-plugin]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-cinderplugin Bound pvc-18b3ef69-e2ef-4be8-a579-f2b37e1e53f2 1Gi RWO csi-sc-cinderplugin 146m
pvc-snapshot-demo Bound pvc-ed0de1a3-9a64-4984-912b-0a81d4a3c8e4 1Gi RWO csi-sc-cinderplugin 114m
snapshot-demo-restore Bound pvc-f6e8e9b2-da5c-45a1-ba33-a230ee9cc5ba 1Gi RWO csi-sc-cinderplugin 110m
- 计入容器查看挂载情况
[root@kubernetes-master cinder-csi-plugin]# kubectl exec -it nginx2 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
root@nginx2:/# mount | grep sdb
/dev/sdb on /var/lib/www/html type ext4 (rw,relatime,seclabel,data=ordered)
root@nginx2:/# ls /var/lib/www/html/
index.html lost+found test
root@nginx2:/# echo "test cinder" > /var/lib/www/html/newfile
root@nginx2:/# ls /var/lib/www/html/
index.html lost+found newfile test
root@nginx2:/#
快照等一些列测试请参考官方文档，文末附链接
参考
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-cinder-csi-plugin.md
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-openstack-cloud-controller-manager.md#steps
https://stackoom.com/question/3zJL9/%E5%A6%82%E4%BD%95%E5%B0%86Openstack-Cinder%E5%8D%B7%E7%94%A8%E4%BD%9CKubernetes%E6%8C%81%E4%B9%85%E5%8D%B7
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/