k8s 部署deploy创建pod异常

环境

• 2台node (10.4.7.21,10.4.7.22)
• kubernetes 1.15.2
• docker 20.10.12
• harbor.od.com 是自建的harbor仓库地址

deployment 文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: dev
spec:
  replicas: 3
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
      - image: harbor.od.com/public/nginx:1.17.1
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP

异常如下：

10.4.7.21 上异常

 kubelet, hdss7-21.host.com  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob864764295: no such file or directory

分析

检查 pause 镜像是否可下载

-> # docker pull harbor.od.com/public/pause:v1.0.0
v1.0.0: Pulling from public/pause
4f4fb700ef54: Pulling fs layer 
b9c8ec465f6b: Pulling fs layer 
open /data/docker/tmp/GetImageBlob927909633: no such file or directory

以上信息提示，目录不存在。

-> # ll /data/docker
总用量 28K
drwx--x--- 12 root root 8.0K 2月  12 11:30 containers
drwx--x--- 21 root root  16K 2月  12 11:30 overlay2

缺少tmp目录。在10.4.7.22上查看 docker目录

-> # ll /data/docker 
总用量 24K
drwx--x--x  4 root root  120 1月   9 11:18 buildkit
drwx--x--- 14 root root 8.0K 2月  12 16:27 containers
drwx------  3 root root   22 1月   9 11:18 image
drwxr-x---  3 root root   19 1月   9 11:18 network
drwx--x--- 47 root root  12K 2月  12 16:27 overlay2
drwx------  4 root root   32 1月   9 11:18 plugins
drwx------  2 root root    6 2月  11 23:35 runtimes
drwx------  2 root root    6 1月   9 11:18 swarm
drwx------  2 root root    6 2月  12 16:21 tmp
drwx------  2 root root    6 1月   9 11:18 trust
drwx-----x  4 root root  194 2月  11 23:35 volumes

明显少了不少文件夹。解决办法就是重新启动docker看能否重建这些目录 。

-> # systemctl restart docker 
root@hdss7-21 [16时56分56秒] [~] 
-> # ll /data/docker
总用量 28K
drwx--x--x  4 root root  120 2月  12 16:56 buildkit
drwx--x--- 23 root root 8.0K 2月  12 16:56 containers
drwx------  3 root root   22 2月  12 16:56 image
drwxr-x---  3 root root   19 2月  12 16:56 network
drwx--x--- 47 root root  16K 2月  12 16:56 overlay2
drwx------  4 root root   32 2月  12 16:56 plugins
drwx------  2 root root    6 2月  12 16:56 runtimes
drwx------  2 root root    6 2月  12 16:56 swarm
drwx------  2 root root    6 2月  12 16:56 tmp
drwx------  2 root root    6 2月  12 16:56 trust
drwx-----x  2 root root   50 2月  12 16:56 volumes

查看 pod events 详情:
kubectl describe pods nginx-5d58f7f679-qqx2j -n dev

Events:
  Type     Reason                  Age                  From                        Message
  ----     ------                  ----                 ----                        -------
  Normal   Scheduled               30m                  default-scheduler           Successfully assigned dev/nginx-5d58f7f679-qqx2j to hdss7-21.host.com
  Warning  FailedCreatePodSandBox  30m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob389074113: no such file or directory
  Warning  FailedCreatePodSandBox  30m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob037184812: no such file or directory
  Warning  FailedCreatePodSandBox  29m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob497930395: no such file or directory
  Warning  FailedCreatePodSandBox  29m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob959697214: no such file or directory
  Warning  FailedCreatePodSandBox  29m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob021467936: no such file or directory
  Warning  FailedCreatePodSandBox  29m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob882862930: no such file or directory
  Warning  FailedCreatePodSandBox  29m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob810866825: no such file or directory
  Warning  FailedCreatePodSandBox  28m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob758117460: no such file or directory
  Warning  FailedCreatePodSandBox  28m                  kubelet, hdss7-21.host.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob511806883: no such file or directory
  Warning  FailedCreatePodSandBox  5m8s (x38 over 28m)  kubelet, hdss7-21.host.com  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob120923664: no such file or directory
  Normal   Pulling                 15s                  kubelet, hdss7-21.host.com  Pulling image "harbor.od.com/public/nginx:1.17.1"

最后一行显示正常拉取镜像