环境
- 2台node (10.4.7.21,10.4.7.22)
- kubernetes 1.15.2
- docker 20.10.12
- harbor.od.com 是自建的harbor仓库地址
deployment 文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: dev
spec:
replicas: 3
selector:
matchLabels:
name: nginx
template:
metadata:
labels:
name: nginx
spec:
containers:
- image: harbor.od.com/public/nginx:1.17.1
name: nginx
ports:
- containerPort: 80
protocol: TCP
异常如下：
10.4.7.21 上异常
kubelet, hdss7-21.host.com (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob864764295: no such file or directory
分析
检查 pause 镜像是否可下载
-> # docker pull harbor.od.com/public/pause:v1.0.0
v1.0.0: Pulling from public/pause
4f4fb700ef54: Pulling fs layer
b9c8ec465f6b: Pulling fs layer
open /data/docker/tmp/GetImageBlob927909633: no such file or directory
以上信息提示，目录不存在。
-> # ll /data/docker
总用量 28K
drwx--x--- 12 root root 8.0K 2月 12 11:30 containers
drwx--x--- 21 root root 16K 2月 12 11:30 overlay2
缺少tmp目录。在10.4.7.22上查看 docker目录
-> # ll /data/docker
总用量 24K
drwx--x--x 4 root root 120 1月 9 11:18 buildkit
drwx--x--- 14 root root 8.0K 2月 12 16:27 containers
drwx------ 3 root root 22 1月 9 11:18 image
drwxr-x--- 3 root root 19 1月 9 11:18 network
drwx--x--- 47 root root 12K 2月 12 16:27 overlay2
drwx------ 4 root root 32 1月 9 11:18 plugins
drwx------ 2 root root 6 2月 11 23:35 runtimes
drwx------ 2 root root 6 1月 9 11:18 swarm
drwx------ 2 root root 6 2月 12 16:21 tmp
drwx------ 2 root root 6 1月 9 11:18 trust
drwx-----x 4 root root 194 2月 11 23:35 volumes
明显少了不少文件夹。解决办法就是重新启动docker看能否重建这些目录 。
-> # systemctl restart docker
root@hdss7-21 [16时56分56秒] [~]
-> # ll /data/docker
总用量 28K
drwx--x--x 4 root root 120 2月 12 16:56 buildkit
drwx--x--- 23 root root 8.0K 2月 12 16:56 containers
drwx------ 3 root root 22 2月 12 16:56 image
drwxr-x--- 3 root root 19 2月 12 16:56 network
drwx--x--- 47 root root 16K 2月 12 16:56 overlay2
drwx------ 4 root root 32 2月 12 16:56 plugins
drwx------ 2 root root 6 2月 12 16:56 runtimes
drwx------ 2 root root 6 2月 12 16:56 swarm
drwx------ 2 root root 6 2月 12 16:56 tmp
drwx------ 2 root root 6 2月 12 16:56 trust
drwx-----x 2 root root 50 2月 12 16:56 volumes
查看 pod events 详情:
kubectl describe pods nginx-5d58f7f679-qqx2j -n dev
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 30m default-scheduler Successfully assigned dev/nginx-5d58f7f679-qqx2j to hdss7-21.host.com
Warning FailedCreatePodSandBox 30m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob389074113: no such file or directory
Warning FailedCreatePodSandBox 30m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob037184812: no such file or directory
Warning FailedCreatePodSandBox 29m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob497930395: no such file or directory
Warning FailedCreatePodSandBox 29m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob959697214: no such file or directory
Warning FailedCreatePodSandBox 29m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob021467936: no such file or directory
Warning FailedCreatePodSandBox 29m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob882862930: no such file or directory
Warning FailedCreatePodSandBox 29m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob810866825: no such file or directory
Warning FailedCreatePodSandBox 28m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob758117460: no such file or directory
Warning FailedCreatePodSandBox 28m kubelet, hdss7-21.host.com Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob511806883: no such file or directory
Warning FailedCreatePodSandBox 5m8s (x38 over 28m) kubelet, hdss7-21.host.com (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "harbor.od.com/public/pause:v1.0.0": open /data/docker/tmp/GetImageBlob120923664: no such file or directory
Normal Pulling 15s kubelet, hdss7-21.host.com Pulling image "harbor.od.com/public/nginx:1.17.1"
最后一行显示正常拉取镜像