k8s问题记录与解决
一、问题：error: open /var/lib/kubelet/config.yaml: no such file or directory
解决：关键文件缺失，多发生于没有做 kubeadm init就运行了systemctl start kubelet。 要先成功运行kubeadm init
二、kubelet.service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.
解决：打开/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 中的配置：
[root@k8s-master ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
要打开注释此# ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
三、journalctl -f -u kubelet （-f是 --follow, -u是过滤出kubelet日志）
centos7 查看日志
四、kubeadm安装的k8s，重新安装k8s-mst
检查 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 中的配置
systemctl restart kubelet
kubeadm reset
kubeadm init --kubernetes-version=v1.13.4 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
其他节点执行
kubeadm reset
kubeadm join ...命令
五、kubeadm 生成的token过期后，集群增加节点
参考：https://www.jianshu.com/p/a5e379638577
六、[reset] unmounting mounted directories in "/var/lib/kubelet" 卡主
重启机器，重新执行命令
-------------------20190518 update-------------------
七、master所在虚机重启后，kube-master1 kubelet[34770]: E0419 13:52:09.511348 34770 kubelet.go:2266] node "kube-master1" not found，并发现获取到的ip地址为空，ifconfig命令去查看网卡配置情况，却发现根本没有配置eth0/ens33网卡
解决：依次systemctl stop kubelet、systemctl stop docker、systemctl restart network、systemctl restart docker、systemctl restart kubelet、ifconfig、kubectl get node
-------------------20190614 update-------------------
八、kubelet, k8s-node1 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1b7fb9d83e89dbe2815cc10fb1daf342162cb74da30568c0a59585e1dc9329a4" network for pod "wxapp-redis-1": NetworkPlugin cni failed to set up pod "wxapp-redis-1_pnup" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24
解决：到有问题的机器，执行如下命令：
[root@k8s-node1 redis]# cd /var/lib/cni/flannel/
[root@k8s-node1 flannel]# ll
total 32
-rw-------. 1 root root 206 Apr 19 11:30 0727fe1a742f28b9a5d5d3188496bdc0aec220599caf6ce8c28f1b9c8ef1b8d4
-rw-------. 1 root root 206 Apr 19 14:30 13776cebe870d3f58982a123e2e32a4a89780c421db1bc425f13f13756822f81
-rw-------. 1 root root 206 Apr 19 11:30 1f249dd31dae8177a4fa5d3009eec7a36a4dccd8a836975f1d798adf43afda51
-rw-------. 1 root root 206 Apr 19 14:34 317e1ce2f78fddd1b36879be5dff169d27fefd4ca191dcd9d85781ab65cc14d8
-rw-------. 1 root root 187 Jun 14 10:05 5b104d16ea2042bd67f8958b8042fff01de3ba6a69f1a62f4b3ded81955c24bb
-rw-------. 1 root root 206 Apr 23 15:30 5bfe39c5fb73ecb09b7343260b8fc2526bb99b7d00a216ab0c76d87d247f3bc0
-rw-------. 1 root root 187 Jun 14 10:05 85de9489e5a4b5091bc40b5dc216a9f15fdb9aa077a28df32cfef97f8abd0c81
-rw-------. 1 root root 206 Apr 19 11:30 db015f887bbedf2ad7731aaa4a321183594a1d2c9b95398bb25f61ad5b052092
[root@k8s-node1 flannel]# systemctl stop docker
[root@k8s-node1 flannel]# systemctl stop kubelet
[root@k8s-node1 flannel]# systemctl stop kube-proxy
Failed to stop kube-proxy.service: Unit kube-proxy.service not loaded.
[root@k8s-node1 flannel]# rm -rf /var/lib/cni/flannel/ && rm -rf /var/lib/cni/networks/cbr0/ && ip link delete cni0
[root@k8s-node1 flannel]# rm -rf /var/lib/cni/networks/cni0/*
[root@k8s-node1 flannel]# systemctl start docker
[root@k8s-node1 flannel]# systemctl start kubelet
-------------------20190622 update-------------------
九、转（kubernetes --> kube-dns 安装 https://blog.csdn.net/zhuchuangang/article/details/76093887 https://www.cnblogs.com/chimeiwangliang/p/8809280.html）
十、转（kubernetes中网络报错问题排查 http://www.mamicode.com/info-detail-2315259.html）
文献：
1.kubernetes---CentOS7安装kubernetes1.11.2图文完整版
2.安装k8s 1.9.0 实践：问题集锦