kubernetes pod NodeLost 分析
问题描述
[root@master-64 ~]# kubectl get pods -n kube-system -owide| grep -v Running
NAME READY STATUS RESTARTS AGE IP NODE
calico-node-cnrvm 2/2 NodeLost 0 48d 10.39.1.44 proxy-44
filebeat-k7svm 1/1 NodeLost 0 15d 192.168.159.0 proxy-44
kube-proxy-oop2g 1/1 NodeLost 0 62d 10.39.1.44 proxy-44
kubectl-qlj27 1/1 NodeLost 0 15d 192.168.159.3 proxy-44
prometheus-tim-3864503240-ng4h0 0/1 CrashLoopBackOff 485 1d 192.168.88.70 slave-95
service-proxy-clq2t 1/1 NodeLost 0 2h 10.39.1.44 proxy-44[root@master ~]# kubectl describe pod calico-node-cnrvm -n kube-system
...
Node-Selectors: calico=2.5.1
Tolerations: node.alpha.kubernetes.io/notReady:NoExecute
node.alpha.kubernetes.io/unreachable:NoExecute /proc/2005/net/dev: no such file or directory
I0119 13:53:02.039754 2029 helpers.go:101] Unable to get network stats from pid 16385: couldn't read network stats: failure opening /proc/16385/net/dev: open /proc/16385/net/dev: no such file or directory
...
I0119 13:53:10.273923 2029 reconciler.go:299] MountVolume operation started for volume "kubernetes.io/secret/aee483dc-f6c3-11e7-9d36-5254b24cbf5e-default-token-qu72f" (spec.Name: "default-token-qu72f") to pod "aee483dc-f6c3-11e7-9d36-5254b24cbf5e" (UID: "aee483dc-f6c3-11e7-9d36-5254b24cbf5e"). Volume is already mounted to pod, but remount was requested.
...
I0119 13:54:33.846820 2029 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem stats
I0119 13:54:33.846949 2029 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem stats解决问题
执行重启命令就可以了
docker restart containerID
END