k8s集群coredns始终处于ContainerCreating状态解决思路【network: stat /var/lib/calico/nodename: no such file or dir】

情况说明

原始集群中使用flannel作为网络插件，后来想在集群中使用calico网络插件建立网络规则，但在安装的过程中因为一些失误操作（没有读懂官方的安装说明），导致集群宕机，coredns始终处于ContainerCreating状态。

报错信息：

首先需要说明一下，coredns的状态和flannel有着必然的联系，所以首先使用describe查看pods的错误信息。

[root@master ~]# kubectl get pods -n kube-system
NAME                             READY   STATUS              RESTARTS   AGE
coredns-5bfd685c78-mmjxc         0/1     ContainerCreating   0          9s
coredns-5bfd685c78-zmmpv         0/1     ContainerCreating   0          39s
etcd-master                      1/1     Running             5          47d
kube-apiserver-master            1/1     Running             5          47d
kube-controller-manager-master   1/1     Running             5          47d
kube-flannel-ds-8vzsv            1/1     Running             1          15m
kube-flannel-ds-zbqt9            1/1     Running             1          15m
kube-flannel-ds-zwxrh            1/1     Running             1          15m
kube-proxy-6r25s                 1/1     Running             5          47d
kube-proxy-m8gxx                 1/1     Running             5          47d
kube-proxy-s2jb8                 1/1     Running             5          47d
kube-scheduler-master            1/1     Running             5          47d

使用describe查看错误信息。

[root@master ~]# kubectl describe pods -n kube-system coredns-5bfd685c78-mmjxc
Name:                 coredns-5bfd685c78-mmjxc
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 node2/192.168.10.220
Start Time:           Tue, 14 Jul 2020 22:11:33 +0800
Labels:               k8s-app=kube-dns
                      pod-template-hash=5bfd685c78
Annotations:          <none>
Status:               Pending
IP:                   
Controlled By:        ReplicaSet/coredns-5bfd685c78
Containers:
  coredns:
    Container ID:  
    Image:         k8s.gcr.io/coredns:1.3.1
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-vcfp8 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-vcfp8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-vcfp8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Normal   Scheduled               25s                 default-scheduler  Successfully assigned kube-system/coredns-5bfd685c78-mmjxc to node2
  Warning  FailedCreatePodSandBox  24s                 kubelet, node2     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "071c8fc6acb87838fd4ee341479a0769a97401c481a93b5b54f8812ba6fa0ed4" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Warning  FailedCreatePodSandBox  23s                 kubelet, node2     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e35ce83b7c4f18d1aa8c2a57cea6ade03b33ad98041d1b5c0d218ddfc300f23e" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Warning  FailedCreatePodSandBox  22s                 kubelet, node2     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e043cc0de7e693fa36d8115ae07b44560ab1883363127fca15becc36b7976ecc" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Warning  FailedCreatePodSandBox  21s                 kubelet, node2     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d128ed26899ab9cf2bf9f62784a3bb723b63141cd3fff82f860325ad3433c322" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Warning  FailedCreatePodSandBox  20s                 kubelet, node2     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "045f258154329ba4e46192a01df441cf04c395a3599fd68ebd5a5bf7c648fcd1" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
  Warning  FailedCreatePodSandBox  19s                 kubelet, node2     Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "dc776f01abbbf1a219f8014c377f0359db90c4ec74cfe8baa993ff63a0b805f7" network for pod "coredns-5bfd685c78-mmjxc": NetworkPlugin cni failed to set up pod "coredns-5bfd685c78-mmjxc_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

可以看出node节点中的网络不通畅，

解决方法

卸载master、node节点上所有关于calico的安装信息。
l连接Node节点，使用以下命令删除node节点上关于calico的配置信息，并重启kubelet服务

rm -rf /etc/cni/net.d/*

rm -rf /var/lib/cni/calico

systemctl  restart kubelet