Kubernetes的node,NotReady 如何查问题,针对问题解决

背景:平时运行的好好的k8s集群,早上过来一看发现好多红色指标,马上查 下集群状况,发现一个节点跪了。

Kubernetes的node,NotReady 如何查问题,针对问题解决_第1张图片
Kubernetes的node,NotReady 如何查问题,针对问题解决_第2张图片
Kubernetes的node,NotReady 如何查问题,针对问题解决_第3张图片

可以确定work01是真的跪了了,那么如何查找问题呢? 我们用它:journalctl工具是CentOS-7才有的工具

Systemd 统一管理所有 Unit 的启动日志。带来的好处就是 ,可以只用journalctl一个命令,查看所有日志(内核日志和 应用日志)。日志的配置文件/etc/systemd/journald.conf 

journalctl用法:
查看所有日志(默认情况下 ,只保存本次启动的日志): journalctl 
查看内核日志(不显示应用日志): journalctl -k 
查看系统本次启动的日志: journalctl -b

查看上一次启动的日志(需更改设置):
 在该[Journal]部分下,将该Storage=选项设置为“persistent”以启用持久记录:
        vim    /etc/systemd/journald.conf
    . . .
    [Journal]
    Storage=persistent

查看httpd服务的日志信息:

我们使用的是:

追踪日志
要主动追踪当前正在编写的日志,大家可以使用-f标记。同样功能类似为tail -f,只要不终止,会一直监控
journalctl -f

也许最有用的过滤方式是你感兴趣的单位。我们可以使用这个-u选项来过滤我们可以使用这个-u选项来过滤
journalctl -u

所以我们最终使用的命令是:
journalctl -f -u kubelet
[root@***-***-work01 ~]# journalctl -f -u kubelet
-- Logs begin at Wed 2018-09-05 11:16:12 CST. --
Sep 05 11:43:30 ***-***-work01 kubelet[5764]: E0905 11:43:30.325784    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.048633    5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.304730    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:34 ***-***-work01 kubelet[5764]: E0905 11:43:34.325829    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:36 ***-***-work01 kubelet[5764]: E0905 11:43:36.293212    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:38 ***-***-work01 kubelet[5764]: E0905 11:43:38.301224    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:40 ***-***-work01 kubelet[5764]: E0905 11:43:40.330581    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:42 ***-***-work01 kubelet[5764]: E0905 11:43:42.289427    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:44 ***-***-work01 kubelet[5764]: E0905 11:43:44.311478    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:46 ***-***-work01 kubelet[5764]: E0905 11:43:46.296680    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:48 ***-***-work01 kubelet[5764]: E0905 11:43:48.320159    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:50 ***-***-work01 kubelet[5764]: E0905 11:43:50.295335    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.049269    5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.332253    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:54 ***-***-work01 kubelet[5764]: E0905 11:43:54.319333    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:56 ***-***-work01 kubelet[5764]: E0905 11:43:56.298336    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.

NOTE:因为我的原因,很明显,work01节点是机器直接跪了,
Kubernetes的node,NotReady 如何查问题,针对问题解决_第4张图片

最后重新启动就OK了。

systemctl restart docker

你可能感兴趣的:(Kubernetes)