kubelet日志报错node not found处理

之前正常运行的环境，突然反馈Kubernetes集群不可用，无法执行kubectl命令。

查看kubelet服务状态，看到日志中提示node "master1" not found。

[root@master1 var]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2022-05-05 10:32:20 CST; 13s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 15016 (kubelet)
    Tasks: 14
   Memory: 38.7M
   CGroup: /system.slice/kubelet.service
           └─15016 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod...

May 05 10:32:32 master1 kubelet[15016]: E0505 10:32:32.353984   15016 kubelet.go:2263] node "master1" not found
May 05 10:32:32 master1 kubelet[15016]: E0505 10:32:32.454152   15016 kubelet.go:2263] node "master1" not found
May 05 10:32:32 master1 kubelet[15016]: E0505 10:32:32.554312   15016 kubelet.go:2263] node "master1" not found
May 05 10:32:32 master1 kubelet[15016]: E0505 10:32:32.654451   15016 kubelet.go:2263] node "master1" not found
May 05 10:32:32 master1 kubelet[15016]: E0505 10:32:32.754553   15016 kubelet.go:2263] node "master1" not found

master1节点就是本机，网络都是正常的，尝试重启kubelet服务无果。

无意中在输入命令时，使用tab键补全命令，得到了错误的提示：

[root@master1 log]# cat yum-bash: cannot create temp file for here-document: No space left on device

发现应该是磁盘空间不足了，查看磁盘使用量发现，果然磁盘已经占用100%了。

[root@master1 var]# df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/centos_202-root   39G   39G   20K 100% /
devtmpfs                     3.9G     0  3.9G   0% /dev
tmpfs                        3.9G  4.0K  3.9G   1% /dev/shm
tmpfs                        3.9G  385M  3.5G  10% /run
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0                    11G   11G     0 100% /opt/centos

定位到docker容器目录下，有个容器占用了大量的磁盘，因为该容器长时间输出错误日志导致。

[root@master1 0d7b200dd9209005759e20e3286a21b1435ff05c2179847a5f294deeb9bba52f]# ls -hlst
total 11G
 11G -rw-r----- 1 root root  11G May  3 06:51 0d7b200dd9209005759e20e3286a21b1435ff05c2179847a5f294deeb9bba52f-json.log
8.0K -rw------- 1 root root 5.8K Apr 24 09:29 config.v2.json
4.0K -rw-r--r-- 1 root root 2.0K Apr 24 09:29 hostconfig.json
   0 drwx------ 2 root root    6 Apr 24 09:29 mounts
   0 drwx------ 2 root root    6 Apr 24 09:29 checkpoints

清空该容器控制台日志文件：

cat /dev/null > 0d7b200dd9209005759e20e3286a21b1435ff05c2179847a5f294deeb9bba52f-json.log 

重启docker和kubelet服务后，Kubernetes集群恢复。

systemctl restart docker
systemctl restart kubelet