解决k8s拉取flannel失败的方法

1. 运行环境

k8s版本：1.26.0
flannel：0.20.2
flannel-cni-plugin：v1.1.0

2. 当前问题

flannel是k8s常用的网络插件，正常的部署步骤为：

1. 打开flannel项目：https://github.com/flannel-io/flannel
2. 按照指引执行：kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

但是在国内如果按照该步骤，则会显示

[root@k8s-node1 ~]# kubectl get pod -n kube-flannel
NAMESPACE      NAME                                READY   STATUS                  RESTARTS       AGE
kube-flannel   kube-flannel-ds-2wz5z               0/1     Init:ImagePullBackOff   0              13s
kube-flannel   kube-flannel-ds-n6wcb               0/1     Init:ImagePullBackOff   0              13s
kube-flannel   kube-flannel-ds-wk62c               0/1     Init:ImagePullBackOff   0              13s

此时查看任意pod报错原因，显示为：

[root@k8s-node1 ~]# kubectl describe pod kube-flannel-ds-2wz5z -n kube-flannel
Name:                 kube-flannel-ds-2wz5z
Namespace:            kube-flannel
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      flannel
Node:                 k8s-node3/10.15.0.23
Start Time:           Thu, 22 Jun 2023 13:57:58 +0800
Labels:               app=flannel
                      controller-revision-hash=6d89ffc7b6
                      pod-template-generation=1
                      tier=node
Annotations:          <none>
Status:               Pending
IP:                   10.15.0.23
IPs:
  IP:           10.15.0.23
Controlled By:  DaemonSet/kube-flannel-ds
Init Containers:
  install-cni-plugin:
    Container ID:  
    Image:         docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
    Args:
      -f
      /flannel
      /opt/cni/bin/flannel
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /opt/cni/bin from cni-plugin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj6jn (ro)
  install-cni:
    Container ID:  
    Image:         docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      cp
    Args:
      -f
      /etc/kube-flannel/cni-conf.json
      /etc/cni/net.d/10-flannel.conflist
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/cni/net.d from cni (rw)
      /etc/kube-flannel/ from flannel-cfg (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj6jn (ro)
Containers:
  kube-flannel:
    Container ID:  
    Image:         docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/bin/flanneld
    Args:
      --ip-masq
      --kube-subnet-mgr
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:           kube-flannel-ds-2wz5z (v1:metadata.name)
      POD_NAMESPACE:      kube-flannel (v1:metadata.namespace)
      EVENT_QUEUE_DEPTH:  5000
    Mounts:
      /etc/kube-flannel/ from flannel-cfg (rw)
      /run/flannel from run (rw)
      /run/xtables.lock from xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nj6jn (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  run:
    Type:          HostPath (bare host directory volume)
    Path:          /run/flannel
    HostPathType:  
  cni-plugin:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  flannel-cfg:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-flannel-cfg
    Optional:  false
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  kube-api-access-nj6jn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 :NoSchedule op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   Scheduled       49s                default-scheduler  Successfully assigned kube-flannel/kube-flannel-ds-2wz5z to k8s-node3
  Warning  FailedMount     48s                kubelet            MountVolume.SetUp failed for volume "kube-api-access-nj6jn" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulling         47s                kubelet            Pulling image "docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0"
  Warning  Failed          22s                kubelet            Failed to pull image "docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0": failed to resolve reference "docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-flannelcni-flannel-cni-plugin/manifests/v1.1.0": net/http: TLS handshake timeout
  Warning  Failed          22s                kubelet            Error: ErrImagePull
  Normal   SandboxChanged  15s (x7 over 21s)  kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff         15s (x7 over 21s)  kubelet            Back-off pulling image "docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0"
  Warning  Failed          15s (x7 over 21s)  kubelet            Error: ImagePullBackOff

原因也很明显，k8s在拉取docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0时失败了，默认拉取的镜像仓库在国外，被墙了

3. 解决方案

作为一个完全不懂k8s和docker的新手，我琢磨出以下曲线解决方案

3.1. 安装并配置docker国内源

这一步网上资料很多，不再赘述，只要保证执行docker info后Registry Mirrors中能看到国内镜像源，执行docker pull hello-world成功即可

[root@k8s-node1 ~]# docker info
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 3.10.0-1062.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.777GiB
 Name: k8s-node1
 ID: 30778c8d-86b5-4c16-84e9-990ea7cf4d22
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://dockerproxy.com/
  https://hub-mirror.c.163.com/
  https://mirror.baidubce.com/
  https://ccr.ccs.tencentyun.com/
 Live Restore Enabled: false
 [root@k8s-node1 containerd]# docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
Digest: sha256:c2e23624975516c7e27b1b25be3682a8c6c4c0cea011b791ce98aa423b5040a0
Status: Image is up to date for hello-world:latest
docker.io/library/hello-world:latest
[root@k8s-node1 containerd]# docker images
REPOSITORY                                                          TAG       IMAGE ID       CREATED         SIZE
hello-world                                                         latest    9c7a54a9a43c   6 weeks ago     13.3kB

3.2. 从docker国内源拉取flannel到宿主机中

在步骤2中，我们展示了命令kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml，实际上是kubectl在线拉取了https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml，我们可以wget或者直接网页打开该地址，获取该yaml文件内容，保存到本地，文件名仍然为kube-flannel.yml，此时执行grep image kube-flannel.yml可得到需要的镜像的地址，此时就可以用docker pull进行下载了

[root@k8s-node1 ~]# grep image kube-flannel.yml 
       #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
        image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
       #image: flannelcni/flannel:v0.20.2 for ppc64le and mips64le (dockerhub limitations may apply)
        image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
       #image: flannelcni/flannel:v0.20.2 for ppc64le and mips64le (dockerhub limitations may apply)
        image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
[root@k8s-node1 ~]# docker pull docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
v1.1.0: Pulling from rancher/mirrored-flannelcni-flannel-cni-plugin
Digest: sha256:28d3a6be9f450282bf42e4dad143d41da23e3d91f66f19c01ee7fd21fd17cb2b
Status: Image is up to date for rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
[root@k8s-node1 ~]# docker pull docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
v0.20.2: Pulling from rancher/mirrored-flannelcni-flannel
Digest: sha256:ec0f0b7430c8370c9f33fe76eb0392c1ad2ddf4ccaf2b9f43995cca6c94d3832
Status: Image is up to date for rancher/mirrored-flannelcni-flannel:v0.20.2
docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2

3.3. 将flannel上传到个人镜像仓库中

这里使用的阿里云个人镜像仓库，网上有很多教程如何注册，不再赘述
此处使用的广州节点，然后随便脸滚键盘写个命名空间ahiusydqiu，然后在里面建立仓库flannel和flannel-cni-plugin，再将步骤3.2下载的镜像上传到仓库中

[root@k8s-node1 containerd]# docker images
REPOSITORY                                                          TAG       IMAGE ID       CREATED         SIZE
hello-world                                                         latest    9c7a54a9a43c   6 weeks ago     13.3kB
rancher/mirrored-flannelcni-flannel                                 v0.20.2   b5c6c9203f83   6 months ago    59.6MB
rancher/mirrored-flannelcni-flannel-cni-plugin                      v1.1.0    fcecffc7ad4a   13 months ago   8.09MB
[root@k8s-node1 containerd]# docker tag rancher/mirrored-flannelcni-flannel:v0.20.2 registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel:v0.20.2
[root@k8s-node1 containerd]# docker push registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel:v0.20.2
[root@k8s-node1 containerd]# docker tag rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0 registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel-cni-plugin:v1.1.0
[root@k8s-node1 containerd]# docker push registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel-cni-plugin:v1.1.0

3.4. 更改kube-flannel.yml中的image

对比步骤3.2，更改后的kube-flannel.yml的image为：

[root@k8s-node1 ~]# grep image kube-flannel.yml 
       #image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
       #image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
        image: registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel-cni-plugin:v1.1.0
       #image: flannelcni/flannel:v0.20.2 for ppc64le and mips64le (dockerhub limitations may apply)
       #image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
        image: registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel:v0.20.2
       #image: flannelcni/flannel:v0.20.2 for ppc64le and mips64le (dockerhub limitations may apply)
       #image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.2
        image: registry.cn-guangzhou.aliyuncs.com/ahiusydqiu/flannel:v0.20.2

3.5. 终章：拉取flannel

[root@k8s-node1 ~]# kubectl apply -f kube-flannel.yml 
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@k8s-node1 ~]# kubectl get pod -n kube-flannel
NAMESPACE      NAME                                READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-h7pvv               1/1     Running   0          21m
kube-flannel   kube-flannel-ds-q5d5x               1/1     Running   0          21m
kube-flannel   kube-flannel-ds-tzfns               1/1     Running   0          21m

如果拉取不下来，使用kubectl describe pod kube-flannel-ds-mp2j6 -n kube-flannel中报错信息有denied等字样，可能是阿里云仓库中设为私有仓库，设为公有仓库即可