[kubernates]Unable to update cni config: No networks found in /etc/cni/net.d

kubernates join failed,response:

Jun 10 11:22:57 00VMTL-FabricPeer-172-19-102-59 kubelet[19992]: W0610 11:22:57.514771   19992 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 10 11:22:57 00VMTL-FabricPeer-172-19-102-59 kubelet[19992]: E0610 11:22:57.778191   19992 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jun 10 11:23:02 00VMTL-FabricPeer-172-19-102-59 kubelet[19992]: W0610 11:23:02.515736   19992 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 10 11:23:02 00VMTL-FabricPeer-172-19-102-59 kubelet[19992]: E0610 11:23:02.780046   19992 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jun 10 11:23:07 00VMTL-FabricPeer-172-19-102-59 kubelet[19992]: W0610 11:23:07.516026   19992 cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
Jun 10 11:23:07 00VMTL-FabricPeer-172-19-102-59 kubelet[19992]: E0610 11:23:07.781684   19992 kubelet.go:2170] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

 

analysising：

kubectl get nodes

NAME                              STATUS     ROLES    AGE     VERSION
00vmdl-fabriccli-172-19-101-xx    NotReady      2m35s   v1.14.3
00vmtl-fabricpeer-172-19-102-xx   Ready         7h42m   v1.14.3
172-19-102-xx                     Ready         24h     v1.14.3
172-19-102-xx                     Ready      master   25h     v1.14.3

found one node NotReady

kubectl get pods -n kube-system -o wide |grep 00vmdl-fabriccli-172-19-101-xx

kube-flannel-ds-amd64-hk5sn             0/1     Init:0/1   0          8m49s   172.19.101.63   00vmdl-fabriccli-172-19-101-63               
kube-proxy-vd7lp                        1/1     Running    0          8m49s   172.19.101.63   00vmdl-fabriccli-172-19-101-63               

found the pod kube-flannel-ds-amd64-hk5sn has been init so long.we need analysis it further , look it's logs.

kubectl --namespace kube-system logs kube-flannel-ds-amd64-hk5sn

Error from server (BadRequest): container "kube-flannel" in pod "kube-flannel-ds-amd64-hk5sn" is waiting to start: PodInitializing

see the pod detail info:

kubectl describe pod kube-flannel-ds-amd64-hk5sn  --namespace=kube-system

Events:
  Type     Reason          Age                    From                                     Message
  ----     ------          ----                   ----                                     -------
  Normal   Scheduled       20m                    default-scheduler                        Successfully assigned kube-system/kube-flannel-ds-amd64-hk5sn to 00vmdl-fabriccli-172-19-101-63
  Warning  Failed          17m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Failed to pull image "quay.io/coreos/flannel:v0.11.0-amd64": rpc error: code = Unknown desc = net/http: TLS handshake timeout
  Warning  Failed          17m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Error: ErrImagePull
  Normal   BackOff         17m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Back-off pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed          17m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Error: ImagePullBackOff
  Normal   Pulling         17m (x2 over 20m)      kubelet, 00vmdl-fabriccli-172-19-101-63  Pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed          10m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Error: ErrImagePull
  Warning  Failed          10m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Failed to pull image "quay.io/coreos/flannel:v0.11.0-amd64": rpc error: code = Unknown desc = unexpected EOF
  Normal   SandboxChanged  10m                    kubelet, 00vmdl-fabriccli-172-19-101-63  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         10m (x2 over 10m)      kubelet, 00vmdl-fabriccli-172-19-101-63  Back-off pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed          10m (x2 over 10m)      kubelet, 00vmdl-fabriccli-172-19-101-63  Error: ImagePullBackOff
  Normal   Pulling         9m57s (x2 over 15m)    kubelet, 00vmdl-fabriccli-172-19-101-63  Pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed          2m47s                  kubelet, 00vmdl-fabriccli-172-19-101-63  Failed to pull image "quay.io/coreos/flannel:v0.11.0-amd64": rpc error: code = Unknown desc = net/http: TLS handshake timeout
  Warning  Failed          2m47s                  kubelet, 00vmdl-fabriccli-172-19-101-63  Error: ErrImagePull
  Normal   BackOff         2m47s                  kubelet, 00vmdl-fabriccli-172-19-101-63  Back-off pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed          2m47s                  kubelet, 00vmdl-fabriccli-172-19-101-63  Error: ImagePullBackOff
  Normal   Pulling         2m32s (x2 over 8m31s)  kubelet, 00vmdl-fabriccli-172-19-101-63  Pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Normal   Pulling         33s                    kubelet, 00vmdl-fabriccli-172-19-101-63  Pulling image "quay.io/coreos/flannel:v0.11.0-amd64"

anyway,you need set the following param to comfirm close swap limit , otherways may cause error:

使用kubelet的启动参数–fail-swap-on=false去掉必须关闭Swap的限制。 修改/etc/sysconfig/kubelet，加入：

KUBELET_EXTRA_ARGS="--fail-swap-on=false"