[kubernates]Unable to update cni config: no networks found in /etc/cni/net.d

先查看node情况

---

kubectl get nodes

[root@test-129-70 ~]# kubectl get nodes
NAME          STATUS     ROLES    AGE     VERSION
test-129-21   NotReady   <none>   8m28s   v1.16.1
test-129-70   Ready      master   63m     v1.16.1
test-129-71   Ready      <none>   10m     v1.16.1
test-129-97   Ready      <none>   8m27s   v1.16.1

test-129-21 节点的状态是 NotReady

查看该节点的kubelet日志

---

journalctl -f -u kubelet.service

[root@test-129-21 ~]# journalctl -f -u kubelet.service
Oct 10 16:47:34 test-129-21 kubelet[4988]: E1010 16:47:34.208780    4988 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Oct 10 16:47:34 test-129-21 kubelet[4988]: E1010 16:47:34.208821    4988 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "kube-proxy-pltg7_kube-system(15069b7b-efe7-4186-a30b-7884f30b24e4)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Oct 10 16:47:34 test-129-21 kubelet[4988]: E1010 16:47:34.208840    4988 kuberuntime_manager.go:710] createPodSandbox for pod "kube-proxy-pltg7_kube-system(15069b7b-efe7-4186-a30b-7884f30b24e4)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Oct 10 16:47:34 test-129-21 kubelet[4988]: E1010 16:47:34.208886    4988 pod_workers.go:191] Error syncing pod 15069b7b-efe7-4186-a30b-7884f30b24e4 ("kube-proxy-pltg7_kube-system(15069b7b-efe7-4186-a30b-7884f30b24e4)"), skipping: failed to "CreatePodSandbox" for "kube-proxy-pltg7_kube-system(15069b7b-efe7-4186-a30b-7884f30b24e4)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-proxy-pltg7_kube-system(15069b7b-efe7-4186-a30b-7884f30b24e4)\" failed: rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.1\": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 10 16:47:34 test-129-21 kubelet[4988]: W1010 16:47:34.516124    4988 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Oct 10 16:47:34 test-129-21 kubelet[4988]: E1010 16:47:34.933229    4988 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Oct 10 16:47:39 test-129-21 kubelet[4988]: W1010 16:47:39.516248    4988 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Oct 10 16:47:39 test-129-21 kubelet[4988]: E1010 16:47:39.933977    4988 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Oct 10 16:47:43 test-129-21 kubelet[4988]: E1010 16:47:43.126323    4988 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Oct 10 16:47:43 test-129-21 kubelet[4988]: E1010 16:47:43.126367    4988 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "kube-flannel-ds-amd64-vwp9r_kube-system(a757fcbe-a105-49e9-ae93-f2bc178bd990)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Oct 10 16:47:43 test-129-21 kubelet[4988]: E1010 16:47:43.126380    4988 kuberuntime_manager.go:710] createPodSandbox for pod "kube-flannel-ds-amd64-vwp9r_kube-system(a757fcbe-a105-49e9-ae93-f2bc178bd990)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Oct 10 16:47:43 test-129-21 kubelet[4988]: E1010 16:47:43.126431    4988 pod_workers.go:191] Error syncing pod a757fcbe-a105-49e9-ae93-f2bc178bd990 ("kube-flannel-ds-amd64-vwp9r_kube-system(a757fcbe-a105-49e9-ae93-f2bc178bd990)"), skipping: failed to "CreatePodSandbox" for "kube-flannel-ds-amd64-vwp9r_kube-system(a757fcbe-a105-49e9-ae93-f2bc178bd990)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-flannel-ds-amd64-vwp9r_kube-system(a757fcbe-a105-49e9-ae93-f2bc178bd990)\" failed: rpc error: code = Unknown desc = failed pulling image \"k8s.gcr.io/pause:3.1\": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 10 16:47:44 test-129-21 kubelet[4988]: W1010 16:47:44.516382    4988 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Oct 10 16:47:44 test-129-21 kubelet[4988]: E1010 16:47:44.934672    4988 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Oct 10 16:47:49 test-129-21 kubelet[4988]: W1010 16:47:49.516515    4988 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Oct 10 16:47:49 test-129-21 kubelet[4988]: E1010 16:47:49.935328    4988 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Oct 10 16:47:54 test-129-21 kubelet[4988]: W1010 16:47:54.516643    4988 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Oct 10 16:47:54 test-129-21 kubelet[4988]: E1010 16:47:54.936249    4988 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

查看下pod情况

---

kubectl get pods -o wide -n kube-system

[root@test-129-70 ~]# kubectl get pods -o wide -n kube-system
NAME                                  READY   STATUS              RESTARTS   AGE     IP              NODE          NOMINATED NODE   READINESS GATES
coredns-5644d7b6d9-72gz5              1/1     Running             0          63m     10.244.0.3      test-129-70   <none>           <none>
coredns-5644d7b6d9-bcg49              1/1     Running             0          63m     10.244.0.2      test-129-70   <none>           <none>
etcd-test-129-70                      1/1     Running             0          62m     172.16.129.70   test-129-70   <none>           <none>
kube-apiserver-test-129-70            1/1     Running             0          62m     172.16.129.70   test-129-70   <none>           <none>
kube-controller-manager-test-129-70   1/1     Running             0          62m     172.16.129.70   test-129-70   <none>           <none>
kube-flannel-ds-amd64-6mk5s           1/1     Terminating         0          10m     172.16.129.71   test-129-71   <none>           <none>
kube-flannel-ds-amd64-b8jhc           1/1     Terminating         0          12m     172.16.129.70   test-129-70   <none>           <none>
kube-flannel-ds-amd64-gxkxt           1/1     Running             0          18s     172.16.129.70   test-129-70   <none>           <none>
kube-flannel-ds-amd64-j9ns7           1/1     Terminating         0          8m18s   172.16.129.97   test-129-97   <none>           <none>
kube-flannel-ds-amd64-k5jwg           1/1     Running             0          18s     172.16.129.71   test-129-71   <none>           <none>
kube-flannel-ds-amd64-psx7m           1/1     Running             0          18s     172.16.129.97   test-129-97   <none>           <none>
kube-flannel-ds-amd64-vwp9r           0/1     Init:0/1            0          18s     172.16.129.21   test-129-21   <none>           <none>
kube-proxy-d58bv                      1/1     Running             0          63m     172.16.129.70   test-129-70   <none>           <none>
kube-proxy-nk64c                      1/1     Running             0          10m     172.16.129.71   test-129-71   <none>           <none>
kube-proxy-nxgp9                      1/1     Running             0          8m18s   172.16.129.97   test-129-97   <none>           <none>
kube-proxy-pltg7                      0/1     ContainerCreating   0          8m19s   172.16.129.21   test-129-21   <none>           <none>
kube-scheduler-test-129-70            1/1     Running             0          62m     172.16.129.70   test-129-70   <none>           <none>

看到 kube-flannel-ds-amd64-vwp9r 这个pod的状态是init跟journalctl日志说的问题一致，问题肯定在这里，看下他的日志。

查看日志

---

kubectl -n kube-system logs kube-flannel-ds-amd64-vwp9r

[root@test-129-70 ~]# kubectl -n kube-system logs kube-flannel-ds-amd64-vwp9r
Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-amd64-vwp9r" is waiting to start: PodInitializing

查看pod detail info

---

kubectl describe pod kube-flannel-ds-amd64-vwp9r -n kube-system

[root@test-129-70 ~]# kubectl describe pod kube-flannel-ds-amd64-vwp9r -n kube-system
Name:         kube-flannel-ds-amd64-vwp9r
Namespace:    kube-system
Priority:     0
Node:         test-129-21/172.16.129.21
Start Time:   Thu, 10 Oct 2019 16:41:29 +0800
Labels:       app=flannel
              controller-revision-hash=67f65bfbc7
              pod-template-generation=1
              tier=node
Annotations:  <none>
Status:       Pending
IP:           172.16.129.21
IPs:
  IP:           172.16.129.21
Controlled By:  DaemonSet/kube-flannel-ds-amd64
Init Containers:
  install-cni:
    Container ID:
    Image:         quay.io/coreos/flannel:v0.11.0-amd64
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
    Args:
      -f
      /etc/kube-flannel/cni-conf.json
      /etc/cni/net.d/10-flannel.conflist
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/cni/net.d from cni (rw)
      /etc/kube-flannel/ from flannel-cfg (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-6ftj7 (ro)
Containers:
  kube-flannel:
    Container ID:
    Image:         quay.io/coreos/flannel:v0.11.0-amd64
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
    Args:
      --ip-masq
      --kube-subnet-mgr
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:       kube-flannel-ds-amd64-vwp9r (v1:metadata.name)
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/flannel from run (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from flannel-token-6ftj7 (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  run:
    Type:          HostPath (bare host directory volume)
    Path:          /run/flannel
    HostPathType:
  cni:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-flannel-cfg
    Optional:  false
  flannel-token-6ftj7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  flannel-token-6ftj7
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     :NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason                  Age                  From                  Message
  ----     ------                  ----                 ----                  -------
  Normal   Scheduled               <unknown>            default-scheduler     Successfully assigned kube-system/kube-flannel-ds-amd64-vwp9r to test-129-21
  Warning  FailedCreatePodSandBox  10s (x6 over 4m16s)  kubelet, test-129-21  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

好了，是个愚蠢的错误，node节点不能获取到kube的镜像导致的，解决方法：1、从master节点把镜像传过去；2、init的时候设置仓库地址；3、科学上网一劳永逸。