Kubernetes的node，NotReady 如何查问题，针对问题解决

背景：平时运行的好好的k8s集群，早上过来一看发现好多红色指标，马上查 下集群状况，发现一个节点跪了。

可以确定work01是真的跪了了，那么如何查找问题呢？ 我们用它：journalctl工具是CentOS-7才有的工具

Systemd 统一管理所有 Unit 的启动日志。带来的好处就是 ，可以只用journalctl一个命令，查看所有日志（内核日志和 应用日志）。日志的配置文件/etc/systemd/journald.conf 

journalctl用法：
查看所有日志（默认情况下 ，只保存本次启动的日志）： journalctl 
查看内核日志（不显示应用日志）： journalctl -k 
查看系统本次启动的日志： journalctl -b

查看上一次启动的日志（需更改设置）：
 在该[Journal]部分下，将该Storage=选项设置为“persistent”以启用持久记录：
        vim    /etc/systemd/journald.conf
    . . .
    [Journal]
    Storage=persistent

查看httpd服务的日志信息：

我们使用的是：

追踪日志
要主动追踪当前正在编写的日志，大家可以使用-f标记。同样功能类似为tail -f，只要不终止，会一直监控
journalctl -f

也许最有用的过滤方式是你感兴趣的单位。我们可以使用这个-u选项来过滤我们可以使用这个-u选项来过滤
journalctl -u

所以我们最终使用的命令是：
journalctl -f -u kubelet

[root@***-***-work01 ~]# journalctl -f -u kubelet
-- Logs begin at Wed 2018-09-05 11:16:12 CST. --
Sep 05 11:43:30 ***-***-work01 kubelet[5764]: E0905 11:43:30.325784    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.048633    5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:32 ***-***-work01 kubelet[5764]: E0905 11:43:32.304730    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:34 ***-***-work01 kubelet[5764]: E0905 11:43:34.325829    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:36 ***-***-work01 kubelet[5764]: E0905 11:43:36.293212    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:38 ***-***-work01 kubelet[5764]: E0905 11:43:38.301224    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:40 ***-***-work01 kubelet[5764]: E0905 11:43:40.330581    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:42 ***-***-work01 kubelet[5764]: E0905 11:43:42.289427    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:44 ***-***-work01 kubelet[5764]: E0905 11:43:44.311478    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:46 ***-***-work01 kubelet[5764]: E0905 11:43:46.296680    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:48 ***-***-work01 kubelet[5764]: E0905 11:43:48.320159    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:50 ***-***-work01 kubelet[5764]: E0905 11:43:50.295335    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.049269    5764 file.go:76] Unable to read manifest path "/etc/kubernetes/manifests": path does not exist, ignoring
Sep 05 11:43:52 ***-***-work01 kubelet[5764]: E0905 11:43:52.332253    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:54 ***-***-work01 kubelet[5764]: E0905 11:43:54.319333    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.
Sep 05 11:43:56 ***-***-work01 kubelet[5764]: E0905 11:43:56.298336    5764 kubelet_volumes.go:128] Orphaned pod "1adba3f5-a9d2-11e8-818b-005056ba13fc" found, but volume paths are still present on disk : There were a total of 15 errors similar to this. Turn up verbosity to see them.

NOTE:因为我的原因，很明显，work01节点是机器直接跪了，

最后重新启动就OK了。

systemctl restart docker