yanwei2020

基于K8S和OpenStack日志分析K8S动态使用Cinder PV流程

1、环境说明

环境信息同：
OpenStack虚拟机部署K8S集群，使用Cinder提供PV报错：mount: special device does not exist

2、K8S动态使用Cinder PV流程

2.1、创建带Cinder PV的POD

# kubectl apply -f busybox-cinder-dym.yaml
pod/testpvcpod-cinder created
persistentvolumeclaim/cinder-pvc created

2.2、查看kube-controller-manager日志发起创建volume请求

在k8s-m1节点上查看

# kubectl logs kube-controller-manager-k8s-m2 -n kube-system
I0414 17:03:35.985804       1 openstack_volumes.go:481] Created volume 2bf910e5-b3aa-4761-acc4-4ff846e522bf in Availability Zone: nova Region:  Ignore volume AZ: false
I0414 17:03:35.990185       1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cinder-pvc", UID:"03336a91-a44b-492c-a374-77705348eeb2", APIVersion:"v1", ResourceVersion:"300084", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-03336a91-a44b-492c-a374-77705348eeb2 using kubernetes.io/cinder

2.3、查看cinder 日志执行volume创建

在OpenStack 控制节点上查看
cinder-api日志

# cat /var/log/cinder/api.log
2020-04-14 17:11:13.612 15117 INFO cinder.api.v3.volumes [None req-ac3e03df-4e5e-4108-b115-01e69b2e6735 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Create volume of 1 GB

cinder-volume日志

# cat /var/log/cinder/volume.log |grep 5dc55f61-aa8d-4550-8da7-dd9d656e7994
2020-04-14 17:11:14.223 30848 INFO cinder.volume.flows.manager.create_volume [None req-ac3e03df-4e5e-4108-b115-01e69b2e6735 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Volume 5dc55f61-aa8d-4550-8da7-dd9d656e7994: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-5dc55f61-aa8d-4550-8da7-dd9d656e7994'}
2020-04-14 17:11:14.800 30848 INFO cinder.volume.flows.manager.create_volume [None req-ac3e03df-4e5e-4108-b115-01e69b2e6735 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Volume volume-5dc55f61-aa8d-4550-8da7-dd9d656e7994 (5dc55f61-aa8d-4550-8da7-dd9d656e7994): created successfully

2.4、查看kube-controller-manager日志发起attach volume请求

# kubectl logs kube-controller-manager-k8s-m2 -n kube-system
I0414 15:15:20.834686       1 attacher.go:166] Attach operation successful: volume "848ca614-9ec9-4d72-8922-576fd9c819e6" attached to instance "c8713bd5-5d53-4def-ad35-f9222b4f0f6f".
I0414 15:15:20.936175       1 operation_generator.go:359] AttachVolume.Attach succeeded for volume "testpv" (UniqueName: "kubernetes.io/cinder/848ca614-9ec9-4d72-8922-576fd9c819e6") from node "k8s-n1"
I0414 15:15:20.936301       1 event.go:281] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"testpvcpod-cinder", UID:"6fed1fa3-6330-44bf-ab29-f75f45915a8c", APIVersion:"v1", ResourceVersion:"282459", FieldPath:""}): type: 'Normal' reason: 'SuccessfulAttachVolume' AttachVolume.Attach succeeded for volume "testpv"

2.5、调用nova api接口执行attach volume操作

首先根据虚拟机名k8s-n1获取虚拟机ID

/var/log/httpd/access_log:192.168.32.99 - - [14/Apr/2020:15:20:03 +0800] "GET /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/detail?name=%5Ek8s-n1%24 HTTP/1.0" 200 1976 "-" "gophercloud/2.0.0"
/var/log/nova/nova-api-wsgi.log:2020-04-14 15:16:04.045 8958 INFO nova.api.openstack.requestlog [req-65f041e1-aaa0-41c0-9fcb-563f749c8bcf af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] 192.168.32.99 "GET /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/detail?name=%5Ek8s-n1%24" status: 200 len: 1976 microversion: 2.1 time: 0.629235

然后调用nova attach接口attach volume

$ sudo grep -r "c8713bd5-5d53-4def-ad35-f9222b4f0f6f" /var/log/httpd/
/var/log/httpd/access_log:192.168.32.99 - - [14/Apr/2020:15:10:29 +0800] "POST /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/c8713bd5-5d53-4def-ad35-f9222b4f0f6f/os-volume_attachments HTTP/1.0" 200 194 "-" "gophercloud/2.0.0"

/var/log/nova/nova-api-wsgi.log:2020-04-14 15:10:33.607 8956 INFO nova.api.openstack.requestlog [req-46d2d2f3-f49b-463e-bc88-d879875e0969 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] 192.168.32.99 "POST /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/c8713bd5-5d53-4def-ad35-f9222b4f0f6f/os-volume_attachments" status: 200 len: 194 microversion: 2.1 time: 4.400781
/var/log/nova/nova-api-wsgi.log:[pid: 8956|app: 0|req: 318/1600] 192.168.32.99 () {62 vars in 1359 bytes} [Tue Apr 14 15:10:29 2020] POST /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/c8713bd5-5d53-4def-ad35-f9222b4f0f6f/os-volume_attachments => generated 194 bytes in 4403 msecs (HTTP/1.0 200) 9 headers in 357 bytes (1 switches on core 0)

2.6、nova compute执行cinder volume attach操作

在k8s-n1虚拟机所在的OpenStack计算节点上执行

# cat /var/log/nova/nova-compute.log
2020-04-14 17:11:18.839 36494 INFO nova.compute.manager [req-84de2684-0c6c-427c-9d52-c2f12820df67 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] [instance: c8713bd5-5d53-4def-ad35-f9222b4f0f6f] Attaching volume 5dc55f61-aa8d-4550-8da7-dd9d656e7994 to /dev/vdb
2020-04-14 17:11:21.438 36494 INFO os_brick.initiator.connectors.iscsi [req-84de2684-0c6c-427c-9d52-c2f12820df67 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Trying to connect to iSCSI portal 10.43.166.42:3260

2.7、动态Cinder PV挂载成功

cinder 卷挂载成功

$ openstack volume list
+--------------------------------------+-------------------------------------------------------------+-----------+------+-----------------------------------------------------------------+
| ID                                   | Name                                                        | Status    | Size | Attached to                                                     |
+--------------------------------------+-------------------------------------------------------------+-----------+------+-----------------------------------------------------------------+
| 5dc55f61-aa8d-4550-8da7-dd9d656e7994 | kubernetes-dynamic-pvc-ce961d70-ea7f-4f23-9cf7-7de782951136 | in-use    |    1 | Attached to k8s-n1 on /dev/vdb

K8S查看Cinder PV信息

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
nfs-pv                                     1Gi        RWX            Recycle          Bound    default/nfs-pvc      slow                    26h
pvc-ce961d70-ea7f-4f23-9cf7-7de782951136   1Gi        RWO            Delete           Bound    default/cinder-pvc   standard                7m46s
# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cinder-pvc   Bound    pvc-ce961d70-ea7f-4f23-9cf7-7de782951136   1Gi        RWO            standard       7m49s
nfs-pvc      Bound    nfs-pv                                     1Gi        RWX            slow           26h

3、卸载动态Cinder PV流程

3.1、kubectl delete操作

# kubectl delete -f busybox-cinder-dym.yaml
pod "testpvcpod-cinder" deleted
persistentvolumeclaim "cinder-pvc" deleted

3.2、kube-controller-manager 发起detach请求

K8S Master节点上查看controller Pod日志

# kubectl logs kube-controller-manager-k8s-m2 -n kube-system
I0414 15:24:31.615277       1 reconciler.go:229] attacherDetacher.DetachVolume started for volume "testpv" (UniqueName: "kubernetes.io/cinder/848ca614-9ec9-4d72-8922-576fd9c819e6") on node "k8s-n1"
I0414 15:24:31.617730       1 operation_generator.go:1297] Verified volume is safe to detach for volume "testpv" (UniqueName: "kubernetes.io/cinder/848ca614-9ec9-4d72-8922-576fd9c819e6") on node "k8s-n1"
I0414 15:24:39.600416       1 attacher.go:404] detached volume "848ca614-9ec9-4d72-8922-576fd9c819e6" from node "k8s-n1"
I0414 15:24:39.600457       1 operation_generator.go:470] DetachVolume.Detach succeeded for volume "testpv" (UniqueName: "kubernetes.io/cinder/848ca614-9ec9-4d72-8922-576fd9c819e6") on node "k8s-n1"

3.3、kubelet执行umount操作

K8S Node节点上查看kubelet日志
/var/log/messages

Apr 14 13:25:44 k8s-n1 kubelet: I0414 13:25:44.686060   22356 reconciler.go:183] operationExecutor.UnmountVolume started for volume "default-token-lrww8" (UniqueName: "kubernetes.io/secret/d07e19c7-74bf-4674-b6ce-cdeec5efba66-default-token-lrww8") pod "d07e19c7-74bf-4674-b6ce-cdeec5efba66" (UID: "d07e19c7-74bf-4674-b6ce-cdeec5efba66")
Apr 14 13:25:44 k8s-n1 kubelet: I0414 13:25:44.696646   22356 operation_generator.go:713] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07e19c7-74bf-4674-b6ce-cdeec5efba66-default-token-lrww8" (OuterVolumeSpecName: "default-token-lrww8") pod "d07e19c7-74bf-4674-b6ce-cdeec5efba66" (UID: "d07e19c7-74bf-4674-b6ce-cdeec5efba66"). InnerVolumeSpecName "default-token-lrww8". PluginName "kubernetes.io/secret", VolumeGidValue ""
Apr 14 13:25:44 k8s-n1 kubelet: I0414 13:25:44.786480   22356 reconciler.go:303] Volume detached for volume "default-token-lrww8" (UniqueName: "kubernetes.io/secret/d07e19c7-74bf-4674-b6ce-cdeec5efba66-default-token-lrww8") on node "k8s-n1" DevicePath ""
Apr 14 13:25:52 k8s-n1 kubelet: I0414 13:25:52.708793   22356 reconciler.go:303] Volume detached for volume "pvc-ce961d70-ea7f-4f23-9cf7-7de782951136" (UniqueName: "kubernetes.io/cinder/d07e19c7-74bf-4674-b6ce-cdeec5efba66-pvc-ce961d70-ea7f-4f23-9cf7-7de782951136") on node "k8s-n1" DevicePath ""
Apr 14 13:27:01 k8s-n1 kubelet: E0414 13:27:01.412361   22356 kubelet.go:1680] Unable to attach or mount volumes for pod "testpvcpod-cinder_default(d07e19c7-74bf-4674-b6ce-cdeec5efba66)": unmounted volumes=[testpv-cinder default-token-lrww8], unattached volumes=[testpv-cinder default-token-lrww8]: timed out waiting for the condition; skipping pod
Apr 14 13:27:01 k8s-n1 kubelet: E0414 13:27:01.412396   22356 pod_workers.go:191] Error syncing pod d07e19c7-74bf-4674-b6ce-cdeec5efba66 ("testpvcpod-cinder_default(d07e19c7-74bf-4674-b6ce-cdeec5efba66)"), skipping: unmounted volumes=[testpv-cinder default-token-lrww8], unattached volumes=[testpv-cinder default-token-lrww8]: timed out waiting for the condition

3.4、nova api执行detach操作

OpenStack控制节点查看httpd日志核nova api日志
/var/log/httpd/access_log
/var/log/nova/nova-api-wsgi.log

/var/log/httpd/access_log:192.168.32.99 - - [14/Apr/2020:15:19:57 +0800] "DELETE /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/c8713bd5-5d53-4def-ad35-f9222b4f0f6f/os-volume_attachments/848ca614-9ec9-4d72-8922-576fd9c819e6 HTTP/1.0" 202 - "-" "gophercloud/2.0.0"
/var/log/nova/nova-api-wsgi.log:2020-04-14 15:19:58.441 8956 INFO nova.api.openstack.requestlog [req-70590210-3db2-4f0e-975f-73907ded6b78 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] 192.168.32.99 "DELETE /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/c8713bd5-5d53-4def-ad35-f9222b4f0f6f/os-volume_attachments/848ca614-9ec9-4d72-8922-576fd9c819e6" status: 202 len: 0 microversion: 2.1 time: 0.735860
/var/log/nova/nova-api-wsgi.log:[pid: 8956|app: 0|req: 322/1620] 192.168.32.99 () {58 vars in 1420 bytes} [Tue Apr 14 15:19:57 2020] DELETE /compute/v2.1/d57f7ea9b3b94d9a87321807ec453ca8/servers/c8713bd5-5d53-4def-ad35-f9222b4f0f6f/os-volume_attachments/848ca614-9ec9-4d72-8922-576fd9c819e6 => generated 0 bytes in 738 msecs (HTTP/1.0 202) 9 headers in 361 bytes (1 switches on core 0)

3.5、nova compute 执行detach操作

OpenStack计算节点上查看nova-compute日志

# cat /var/log/nova/nova-compute.log
2020-04-14 17:21:12.481 36494 INFO nova.compute.manager [req-38462e83-b32d-40a5-8f4a-683fabce74e7 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] [instance: c8713bd5-5d53-4def-ad35-f9222b4f0f6f] Detaching volume 5dc55f61-aa8d-4550-8da7-dd9d656e7994
2020-04-14 17:21:12.755 36494 INFO nova.virt.block_device [req-38462e83-b32d-40a5-8f4a-683fabce74e7 af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] [instance: c8713bd5-5d53-4def-ad35-f9222b4f0f6f] Attempting to driver detach volume 5dc55f61-aa8d-4550-8da7-dd9d656e7994 from mountpoint /dev/vdb

OpenStack控制节点上查看cinder-volume日志

# cat /var/log/cinder/volume.log
2020-04-14 17:21:17.551 30848 INFO cinder.volume.targets.tgt [req-38462e83-b32d-40a5-8f4a-683fabce74e7 req-955d6e59-3c32-49d8-8b64-a8f79a80585c af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Removing iscsi_target for Volume ID: 5dc55f61-aa8d-4550-8da7-dd9d656e7994
2020-04-14 17:21:18.567 30848 INFO cinder.volume.targets.iscsi [None req-f416db01-4184-459b-a33d-3ebdf5a2c75f af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Skipping remove_export. No iscsi_target is presently exported for volume: 5dc55f61-aa8d-4550-8da7-dd9d656e7994

3.6、cinder 执行删除卷操作

cinder-api日志

# cat /var/log/cinder/api.log
/var/log/cinder/api.log:2020-04-14 17:21:18.185 15123 INFO cinder.api.openstack.wsgi [None req-f416db01-4184-459b-a33d-3ebdf5a2c75f af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] DELETE https://10.43.166.113:8776/v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes/5dc55f61-aa8d-4550-8da7-dd9d656e7994
/var/log/cinder/api.log:2020-04-14 17:21:18.186 15123 INFO cinder.api.v3.volumes [None req-f416db01-4184-459b-a33d-3ebdf5a2c75f af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] Delete volume with id: 5dc55f61-aa8d-4550-8da7-dd9d656e7994
/var/log/cinder/api.log:2020-04-14 17:21:18.306 15123 INFO cinder.api.openstack.wsgi [None req-f416db01-4184-459b-a33d-3ebdf5a2c75f af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] https://10.43.166.113:8776/v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes/5dc55f61-aa8d-4550-8da7-dd9d656e7994 returned with HTTP 202
/var/log/cinder/api.log:2020-04-14 17:21:18.307 15123 INFO eventlet.wsgi.server [None req-f416db01-4184-459b-a33d-3ebdf5a2c75f af2a52fe5cce4fc6b99b1c2b7c01ba4e d57f7ea9b3b94d9a87321807ec453ca8 - default default] 192.168.32.99 "DELETE /v3/d57f7ea9b3b94d9a87321807ec453ca8/volumes/5dc55f61-aa8d-4550-8da7-dd9d656e7994 HTTP/1.1" status: 202  len: 270 time: 0.1272171