aws eks 集群container runtime升级容器管理工具的切换
参考资料
- https://cloud-atlas.readthedocs.io/zh_CN/latest/kubernetes/debug/crictl.html
- https://zhuanlan.zhihu.com/p/562014518
container runtime
Low-Level和High-Level容器运行时。runc、lxc、lmctfy、Docker（容器）、rkt、cri-o。每一个都是为不同的情况而构建的，并实现了不同的功能。有些，如 containerd 和 cri-o，实际上使用 runc 来运行容器，在High-Level实现镜像管理和 API。与 runc 的Low-Level实现相比，可以将这些功能（包括镜像传输、镜像管理、镜像解包和 API）视为High-Level功能。
- 只专注于正在运行的容器的runtime通常称为“Low-Level容器运行时”
- 支持更多高级功能（如镜像管理和gRPC / Web API）的运行时通常称为“High-Level容器运行时”
在创建一个docker容器的时候，Docker Daemon 并不能直接帮我们创建了，而是请求 containerd 来创建一个容器。当containerd 收到请求后，也不会直接去操作容器，而是创建一个叫做 containerd-shim 的进程。让这个进程去操作容器，我们指定容器进程是需要一个父进程来做状态收集、维持 stdin 等 fd 打开等工作的，假如这个父进程就是 containerd，那如果 containerd 挂掉的话，整个宿主机上所有的容器都得退出了，而引入 containerd-shim 这个垫片就可以来规避这个问题了，就是提供的live-restore的功能。这里需要注意systemd的 MountFlags=slave。然后创建容器需要做一些 namespaces 和 cgroups 的配置，以及挂载 root 文件系统等操作。runc 就可以按照这个 OCI 文档来创建一个符合规范的容器。
container相比调用链更短，组件少且稳定
docker 运行时的调用关系为： kubelet --> dockershim （在 kubelet 进程中） --> dockerd --> containerd
containerd 运行时的调用关系为： kubelet --> cri plugin（在 containerd 进程中） --> containerd
查看node上的进程
root 32041 0.0 0.2 712204 9712 ? Sl 10:50 0:00 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id c2d4f6f9df81a41d0b361700e76eb4e46ada1c7a12629b7b3895e8ab0215223d -address /run/containerd/containerd.sock
root 32042 0.0 0.2 712460 10040 ? Sl 10:50 0:00 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 352402b119bc5ad4dd3c87e0617dec3a41f72747b39720a1a0aadef1c44e3c41 -address /run/containerd/containerd.sock
查看kubelet
ExecStart=/usr/bin/kubelet --cloud-provider aws \
--image-credential-provider-config /etc/eks/ecr-credential-provider/ecr-credential-provider-config \
--image-credential-provider-bin-dir /etc/eks/ecr-credential-provider \
--config /etc/kubernetes/kubelet/kubelet-config.json \
--kubeconfig /var/lib/kubelet/kubeconfig \
--container-runtime remote \
--container-runtime-endpoint unix:///run/containerd/containerd.sock \
$KUBELET_ARGS $KUBELET_EXTRA_ARGS
查看containerd的配置文件
$ cat /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
[grpc]
address = "/run/containerd/containerd.sock"
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn/eks/pause:3.5"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
ctr
ctr是containerd的一个客户端工具。ctr客户端 主要区分了 3 个命名空间分别是k8s.io、moby和default
查看容器和任务
$ sudo ctr -n=k8s.io container ls
查看容器info
sudo ctr -n=k8s.io container info f77f39a204f120e179140394b230fce0605251a21b50cbc327bee21c621aed82
查看image
sudo ctr -n=k8s.io images ls
sudo ctr -n=k8s.io images list -q
管理镜像
sudo ctr -n=k8s.io image image pull busybox
sudo ctr -n=k8s.io images rm docker.io/library/busybox:latest
sudo ctr -n=k8s.io images pull docker.io/library/busybox:latest
crictl
crictl是遵循 CRI 接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像。
crictl 只有一个k8s.io命名空间，没有-n 参数。
参考资料
- 使用 crictl 对 Kubernetes 节点进行调试
- Container Runtime Interface (CRI) CLI
- https://cloud-atlas.readthedocs.io/zh_CN/latest/kubernetes/debug/crictl.html
- https://www.bookstack.cn/read/kubernetes-tasks-1.19-zh/0e2ab828b2c33e61.md
安装crictl
VERSION="v1.24.1"
curl -L https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-${VERSION}-linux-amd64.tar.gz --output crictl-${VERSION}-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz
配置/etc/crictl.yaml如下，或者在执行时通过-r指定endpoint：
runtime-endpoint:unix:///run/containerd/containerd.sock image-endpoint:unix:///run/containerd/containerd.sock timeout:10 debug:false
常用命令
$ crictl -v
$ crictl pull nginx:alpine
$ crictl rmi nginx:alpine
$ crictl images
设置代理
$ cat /etc/systemd/system/containerd.service.d/10-compat-symlink.conf
[Service]
ExecStartPre=/bin/ln -sf /run/containerd/containerd.sock /run/dockershim.sock
Set HTTP_PROXY and HTTPS_PROXY for containerd in /etc/systemd/system/containerd.service.d/http-proxy.conf:
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:65001"
Environment="HTTPS_PROXY=http://127.0.0.1:65001"
查看容器
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock ps
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock images
查看容器日志
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock logs d2910afb661ea
2022/12/30 10:50:57 Server is listening on :5678
2022/12/30 10:58:39 [ERR] Unknown signal terminated
# crictl logs bf37d99b64364
查看容器info
sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock inspect d2910afb661ea
查看stats
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock stats
CONTAINER CPU % MEM DISK INODES
0a1ba2b85a2f4 0.00 1.528MB 0B 13
2470b65dc9824 0.12 12.8MB 0B 15
81dc5e769badb 0.00 1.884MB 0B 13
89112d90b0d9b 0.09 12.82MB 0B 15
a71f81a4af8b0 0.00 1.54MB 0B 13
b3c937668a77c 0.01 47.42MB 0B 26
d2910afb661ea 0.00 1.565MB 0B 13
daf6338c1d55e 0.00 14.89MB 0B 14
f77f39a204f12 0.00 1.528MB 0B 13
stop和start容器
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock stop dead6a33fc0fd
查看pod
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock pods
在容器中执行exec
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock exec -it 5f0e264ddd5c1 bash
拉取镜像
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock pull busybox
Image is up to date for sha256:827365c7baf137228e94bcfc6c47938b4ffde26c68c32bf3d3a7762cd04056a5
查看容器运行时信息
$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock info
{
"config": {
"containerd": {
"snapshotter": "overlayfs",
"defaultRuntimeName": "runc",
"defaultRuntime": {
"runtimeType": "",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": null,
"ContainerAnnotations": null,
"runtimeRoot": "",
"options": null,
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
},
"untrustedWorkloadRuntime": {
"runtimeType": "",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": null,
"ContainerAnnotations": null,
"runtimeRoot": "",
"options": null,
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
},
"runtimes": {
"runc": {
"runtimeType": "io.containerd.runc.v2",
"runtimePath": "",
"runtimeEngine": "",
"PodAnnotations": null,
"ContainerAnnotations": null,
"runtimeRoot": "",
"options": {
"SystemdCgroup": true
},
"privileged_without_host_devices": false,
"baseRuntimeSpec": "",
"cniConfDir": "",
"cniMaxConfNum": 0
}
},
"noPivot": false,
"disableSnapshotAnnotations": true,
"discardUnpackedLayers": false,
"ignoreRdtNotEnabledErrors": false
},
"cni": {
"binDir": "/opt/cni/bin",
"confDir": "/etc/cni/net.d",
"maxConfNum": 1,
"confTemplate": "",
"ipPref": ""
},
"registry": {
"configPath": "/etc/containerd/certs.d:/etc/docker/certs.d",
"mirrors": null,
"configs": null,
"auths": null,
"headers": null
},
"imageDecryption": {
"keyModel": "node"
},
"disableTCPService": true,
"streamServerAddress": "127.0.0.1",
"streamServerPort": "0",
"streamIdleTimeout": "4h0m0s",
"enableSelinux": false,
"selinuxCategoryRange": 1024,
"sandboxImage": "918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn/eks/pause:3.5",
"statsCollectPeriod": 10,
"systemdCgroup": false,
"enableTLSStreaming": false,
"x509KeyPairStreaming": {
"tlsCertFile": "",
"tlsKeyFile": ""
},
"maxContainerLogSize": 16384,
"disableCgroup": false,
"disableApparmor": false,
"restrictOOMScoreAdj": false,
"maxConcurrentDownloads": 3,
"disableProcMount": false,
"unsetSeccompProfile": "",
"tolerateMissingHugetlbController": true,
"disableHugetlbController": true,
"device_ownership_from_security_context": false,
"ignoreImageDefinedVolumes": false,
"netnsMountsUnderStateDir": false,
"enableUnprivilegedPorts": false,
"enableUnprivilegedICMP": false,
"containerdRootDir": "/var/lib/containerd",
"containerdEndpoint": "/run/containerd/containerd.sock",
"rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
"stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
},
"golang": "go1.18.6",
"lastCNILoadStatus": "OK",
"lastCNILoadStatus.default": "OK"
}
nerdctl
参考资料
- https://link.zhihu.com/?target=https%3A//github.com/containerd/nerdctl/releases
- https://github.com/containerd/nerdctl
- https://zhuanlan.zhihu.com/p/562014518
nttlabs贡献了一个名为nerdctl的containerd客户端，可以兼容docker命令行工具
安装nerdctl
wget https://github.com/containerd/nerdctl/releases/download/v1.1.0/nerdctl-1.1.0-linux-amd64.tar.gz
sudo tar -xzvf nerdctl-1.1.0-linux-amd64.tar.gz -C /usr/local/bin/
sudo su -
nerdctl -n k8s.io container ls
通过docker运行nerdctl
docker build -t nerdctl .
docker run -it --rm --privileged nerdctl
nerdctl兼容了docker命令，可以无痛切换
https://github.com/containerd/nerdctl
https://blog.csdn.net/weixin_39246554/article/details/120958424
查看名称空间
sudo nerdctl ns ls
查看image
nerdctl -n k8s.io image ls
拉取镜像，dockerhub
nerdctl pull nginx:latest
sudo nerdctl -n default image ls | grep nginx
查看容器
sudo nerdctl -n k8s.io container ls -a
拉取ecr镜像，需要确保在root用户下执行
aws ecr get-login-password --region cn-north-1 | nerdctl login --username AWS --password-stdin xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn
WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.