kubernetes错误node故障总结
kubelet启动失败，Error: No space left on device
查看日志，关键语句如下
Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/blkio: no space left on device
或
Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/cpu,cpuacct: no space left on device
解决办法，参考https://blog.csdn.net/xiaofang2015/article/details/80649548
[root@node6 ~]# cat /proc/sys/fs/inotify/max_user_watches
8196
[root@node6 ~]# sysctl fs.inotify.max_user_watches=1048576
systemctl命令超时，Failed to start reboot.target: Connection timed out
现象如下：
解决办法：参考地址：https://serverfault.com/questions/712928/systemctl-commands-timeout-when-ran-as-root
[root@node7 ~]# systemctl --force --force reboot
Rebooting.
packet_write_wait: Connection to 10.180.3.107 port 22: Broken pipe
System OOM encountered
Warning SystemOOM 34m (x8 over 34m) kubelet, node5 System OOM encountered
Sep 30 18:36:22 node5 kubelet[134096]: E0930 18:36:22.037042 134096 kubelet_node_status.go:106] Unable to register node "node5" with API server: Post https://localhost:6443/api/v1/nodes: dial tcp 127.0.0.1:6443: getsockopt: connection refused
Sep 30 18:36:21 node5 kubelet[134096]: E0930 18:36:21.898997 134096 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://localhost:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dnode5&limit=500&resourceVersion=0: dial tcp 127.0.0.1:6443: getsockopt: connection refused
直接重启机器吧，原因不知道
ContainerGCFailed rpc error: code = DeadlineExceeded desc = context deadline exceeded
现象：
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Thu, 10 Oct 2019 20:38:34 +0800 Mon, 08 Oct 2018 23:08:22 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 10 Oct 2019 20:38:34 +0800 Fri, 23 Aug 2019 21:12:52 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 10 Oct 2019 20:38:34 +0800 Wed, 15 May 2019 16:12:45 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
Ready False Thu, 10 Oct 2019 20:38:34 +0800 Thu, 10 Oct 2019 20:27:48 +0800 KubeletNotReady PLEG is not healthy: pleg was last seen active 13m51.370099044s ago; threshold is 3m0s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeNotReady 10m (x2 over 137d) kubelet, node37 Node node37 status is now: NodeNotReady
Warning ContainerGCFailed 2m (x4 over 11m) kubelet, node37 rpc error: code = DeadlineExceeded desc = context deadline exceeded
解决办法：
systemctl daemon-reexec
systemctl restart docker(是的，需要重启docker)
Get https://127.0.0.1:6443/api/v1/namespaces/java-service/pods/example-554976b69b-4bnxh: dial tcp 127.0.0.1:6443: getsockopt: connection refused
现象：
node notready，查看describe无event时间信息。
在node上查看kubelet日志systemctl status kubelet.service -l，如下
解决办法：
在node上重启kubelet。有时候重启也不行，看下磁盘，cpu，内存，文件描述符这些资源的占用情况。
有一次就是因为磁盘根目录使用率到99%了，重启完kubelet后，刚恢复又notready了，但是describe里有没有显示出来磁盘空间不足，导致没有往磁盘空间这个方面考虑。
还有一次是因为内存不足了
container runtime is down
不知道
FailedCreatePodSandBox Failed create pod sandbox.
不知道
