kubernetes对接cinder创建pv报错：Failed to provision volume with StorageClass standard
1、环境信息
这个环境用于验证kubernetes使用cinder作为provider提供pv
kubernetes本身运行在openstack集群的虚拟机中，通过router打通和openstack管理面的链接
1.1、openstack 集群虚拟机列表
$ nova list
+--------------------------------------+-----------+--------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------+--------+------------+-------------+-----------------------------------------------------------------------------------------------------------------------------------+
| ccec3aee-15e5-4710-8e4c-5a4db6826022 | k8s-m1 | ACTIVE | - | Running | clu_mgt=180.0.0.3; clu_net2=172.0.2.139, 192.168.32.198 |
| 9eacafc1-8ce7-4073-a8c2-73270cffae31 | k8s-m2 | ACTIVE | - | Running | clu_mgt=180.0.0.206; clu_net2=172.0.2.146, 192.168.32.186 |
| 8128acd7-fd05-4367-985d-a36023f87de8 | k8s-m3 | ACTIVE | - | Running | clu_mgt=180.0.0.8; clu_net2=172.0.2.234, 192.168.32.159
1.2、kubernetes节点列表
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-m1 Ready master 4d22h v1.17.0 172.0.2.139 CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://19.3.6
k8s-m2 Ready master 4d22h v1.17.0 172.0.2.146 CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://19.3.6
k8s-m3 Ready master 4d22h v1.17.0 172.0.2.234 CentOS Linux 7 (Core) 3.10.0-1062.el7.x86_64 docker://19.3.6
1.3、cloud-config文件
其中：192.168.32.90为openstack internal api地址，k8s虚拟机通过float ip与该地址对接
# cat ../cloud-config
[Global]
username=admin
password=keystone
auth-url=http://192.168.32.90:5000/v3
tenant-id=d57f7ea9b3b94d9a87321807ec453ca8
domain-id=default
2、创建pv过程
创建过程同
k8s+virtual-kubelet部署带动态pv的pod到openstack-zun
3、创建pvc出现问题
3.1、pvc处于pending状态
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cinder-pvc Pending
3.2、查看pvc详细信息，显示和这个地址不通：172.255.0.113
# kubectl describe pvc cinder-pvc
Name: cinder-pvc
Namespace: default
StorageClass: standard
Status: Pending
Volume:
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"cinder-pvc","namespace":"default"},"spec":{"accessM...
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Mounted By:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 8s (x2 over 16s) persistentvolume-controller Failed to provision volume with StorageClass "standard": failed to create a 1 GB volume: Post https://172.255.0.113:8776/v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes: dial tcp 172.255.0.113:8776: connect: no route to host
3.3、172.255.0.113这个地址是openstack的public api地址
但是我们需要的是 internal api地址
$ openstack endpoint list | grep keystone
| 0fc44a1959dd4888a9a3125c9199dc2c | RegionOne | keystone | identity | True | internal | https://192.168.32.90:5000/v3 |
| 3e39dce895e64d2794c022eb3d6cb85e | RegionOne | keystone | identity | True | public | https://172.255.0.113:5000/v3 |
| 427000db07dd4feda52c04d121e24ca7 | RegionOne | keystone | identity | True | admin | https://192.168.166.113:35357/v3 |
4、解决问题
4.1、初步分析
pvc详细信息中，https://172.255.0.113:8776/v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes
是openstack cinder的public API endpoint，该endpoint是通过调用keystone endpoint查询接口获取
keystone endpoint查询接口有个参数，可以指定获取public、internal还是admin的endpoint
推测：kubernetes默认获取的是openstack public endpoint
4.2、分析kubernetes获取openstack endpoint代码
这里就略去代码分析过程直接贴最终代码了
代码路径：
gophercloud\openstack\client.go
gophercloud\endpoint_search.go
调用链
AuthenticatedClient->Authenticate->v3auth->NewIdentityV3->ApplyDefaults
ApplyDefaults代码
func (eo *EndpointOpts) ApplyDefaults(t string) {
if eo.Type == "" {
eo.Type = t
}
if eo.Availability == "" {
eo.Availability = AvailabilityPublic
}
}
可以看到，gophercloud在ApplyDefaults中将默认值设置为了public
4.3、为什么默认获取public api
从应用场景上来看，k8s要对接IAAS集群，一般情况下都是通过外部对接，所以默认设置获取public api也是合理的
而我的验证环境恰恰是将k8s部署在了openstack集群上，且需要对接openstack集群，走的是internal api，属于少数应用场景
4.4、解决问题
找到问题源头了，解决就比较简单了
可以直接修改代码如下：
func (eo *EndpointOpts) ApplyDefaults(t string) {
if eo.Type == "" {
eo.Type = t
}
if eo.Availability == "" {
eo.Availability = AvailabilityInternal
}
}
当然，更好的解决方法是将该变量作为可配参数引出来