aws eks 集群container runtime升级容器管理工具的切换

参考资料

• https://cloud-atlas.readthedocs.io/zh_CN/latest/kubernetes/debug/crictl.html
• https://zhuanlan.zhihu.com/p/562014518

container runtime

Low-Level和High-Level容器运行时。runc、lxc、lmctfy、Docker（容器）、rkt、cri-o。每一个都是为不同的情况而构建的，并实现了不同的功能。有些，如 containerd 和 cri-o，实际上使用 runc 来运行容器，在High-Level实现镜像管理和 API。与 runc 的Low-Level实现相比，可以将这些功能（包括镜像传输、镜像管理、镜像解包和 API）视为High-Level功能。

• 只专注于正在运行的容器的runtime通常称为“Low-Level容器运行时”
• 支持更多高级功能（如镜像管理和gRPC / Web API）的运行时通常称为“High-Level容器运行时”

在创建一个docker容器的时候，Docker Daemon 并不能直接帮我们创建了，而是请求 containerd 来创建一个容器。当containerd 收到请求后，也不会直接去操作容器，而是创建一个叫做 containerd-shim 的进程。让这个进程去操作容器，我们指定容器进程是需要一个父进程来做状态收集、维持 stdin 等 fd 打开等工作的，假如这个父进程就是 containerd，那如果 containerd 挂掉的话，整个宿主机上所有的容器都得退出了，而引入 containerd-shim 这个垫片就可以来规避这个问题了，就是提供的live-restore的功能。这里需要注意systemd的 MountFlags=slave。然后创建容器需要做一些 namespaces 和 cgroups 的配置，以及挂载 root 文件系统等操作。runc 就可以按照这个 OCI 文档来创建一个符合规范的容器。

container相比调用链更短，组件少且稳定

docker 运行时的调用关系为： kubelet --> dockershim （在 kubelet 进程中） --> dockerd --> containerd

containerd 运行时的调用关系为： kubelet --> cri plugin（在 containerd 进程中） --> containerd

查看node上的进程

root     32041  0.0  0.2 712204  9712 ?        Sl   10:50   0:00 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id c2d4f6f9df81a41d0b361700e76eb4e46ada1c7a12629b7b3895e8ab0215223d -address /run/containerd/containerd.sock
root     32042  0.0  0.2 712460 10040 ?        Sl   10:50   0:00 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 352402b119bc5ad4dd3c87e0617dec3a41f72747b39720a1a0aadef1c44e3c41 -address /run/containerd/containerd.sock

查看kubelet

ExecStart=/usr/bin/kubelet --cloud-provider aws \
    --image-credential-provider-config /etc/eks/ecr-credential-provider/ecr-credential-provider-config \
   --image-credential-provider-bin-dir /etc/eks/ecr-credential-provider \
    --config /etc/kubernetes/kubelet/kubelet-config.json \
    --kubeconfig /var/lib/kubelet/kubeconfig \
    --container-runtime remote \
    --container-runtime-endpoint unix:///run/containerd/containerd.sock \
    $KUBELET_ARGS $KUBELET_EXTRA_ARGS

查看containerd的配置文件

$ cat /etc/containerd/config.toml
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn/eks/pause:3.5"

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

ctr

ctr是containerd的一个客户端工具。ctr客户端 主要区分了 3 个命名空间分别是k8s.io、moby和default

查看容器和任务

$ sudo ctr -n=k8s.io container ls

查看容器info

sudo ctr -n=k8s.io container info f77f39a204f120e179140394b230fce0605251a21b50cbc327bee21c621aed82

查看image

sudo ctr -n=k8s.io images ls
sudo ctr -n=k8s.io images list -q

管理镜像

sudo ctr -n=k8s.io image image pull busybox
sudo ctr -n=k8s.io images rm docker.io/library/busybox:latest
sudo ctr -n=k8s.io images pull docker.io/library/busybox:latest

crictl

crictl是遵循 CRI 接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像。

crictl 只有一个k8s.io命名空间，没有-n 参数。

参考资料

• 使用 crictl 对 Kubernetes 节点进行调试
• Container Runtime Interface (CRI) CLI
• https://cloud-atlas.readthedocs.io/zh_CN/latest/kubernetes/debug/crictl.html
• https://www.bookstack.cn/read/kubernetes-tasks-1.19-zh/0e2ab828b2c33e61.md

安装crictl

VERSION="v1.24.1"
curl -L https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-${VERSION}-linux-amd64.tar.gz --output crictl-${VERSION}-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

配置/etc/crictl.yaml如下，或者在执行时通过-r指定endpoint：

runtime-endpoint:unix:///run/containerd/containerd.sock image-endpoint:unix:///run/containerd/containerd.sock timeout:10 debug:false

常用命令

$ crictl -v
$ crictl  pull nginx:alpine
$ crictl  rmi  nginx:alpine
$ crictl  images

设置代理

$ cat /etc/systemd/system/containerd.service.d/10-compat-symlink.conf
[Service]
ExecStartPre=/bin/ln -sf /run/containerd/containerd.sock /run/dockershim.sock

Set HTTP_PROXY and HTTPS_PROXY for containerd in /etc/systemd/system/containerd.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://127.0.0.1:65001"
Environment="HTTPS_PROXY=http://127.0.0.1:65001"

查看容器

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock ps

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock images

查看容器日志

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock logs d2910afb661ea
2022/12/30 10:50:57 Server is listening on :5678
2022/12/30 10:58:39 [ERR] Unknown signal terminated
# crictl logs bf37d99b64364

查看容器info

sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock inspect d2910afb661ea

查看stats

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock stats
CONTAINER           CPU %               MEM                 DISK                INODES
0a1ba2b85a2f4       0.00                1.528MB             0B                  13
2470b65dc9824       0.12                12.8MB              0B                  15
81dc5e769badb       0.00                1.884MB             0B                  13
89112d90b0d9b       0.09                12.82MB             0B                  15
a71f81a4af8b0       0.00                1.54MB              0B                  13
b3c937668a77c       0.01                47.42MB             0B                  26
d2910afb661ea       0.00                1.565MB             0B                  13
daf6338c1d55e       0.00                14.89MB             0B                  14
f77f39a204f12       0.00                1.528MB             0B                  13

stop和start容器

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock stop dead6a33fc0fd

查看pod

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock pods

在容器中执行exec

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock exec -it 5f0e264ddd5c1 bash

拉取镜像

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock pull busybox
Image is up to date for sha256:827365c7baf137228e94bcfc6c47938b4ffde26c68c32bf3d3a7762cd04056a5

查看容器运行时信息

$ sudo /usr/local/bin/crictl -r unix:///run/containerd/containerd.sock info
{
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimePath": "",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {
            "SystemdCgroup": true
          },
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false,
      "ignoreRdtNotEnabledErrors": false
    },
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": "",
      "ipPref": ""
    },
    "registry": {
      "configPath": "/etc/containerd/certs.d:/etc/docker/certs.d",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "918309763551.dkr.ecr.cn-north-1.amazonaws.com.cn/eks/pause:3.5",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "device_ownership_from_security_context": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "enableUnprivilegedPorts": false,
    "enableUnprivilegedICMP": false,
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.18.6",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}

nerdctl

参考资料

• https://link.zhihu.com/?target=https%3A//github.com/containerd/nerdctl/releases
• https://github.com/containerd/nerdctl
• https://zhuanlan.zhihu.com/p/562014518

nttlabs贡献了一个名为nerdctl的containerd客户端，可以兼容docker命令行工具

安装nerdctl

wget https://github.com/containerd/nerdctl/releases/download/v1.1.0/nerdctl-1.1.0-linux-amd64.tar.gz
sudo tar -xzvf nerdctl-1.1.0-linux-amd64.tar.gz -C /usr/local/bin/
sudo su -
nerdctl -n k8s.io container ls

通过docker运行nerdctl

docker build -t nerdctl .
docker run -it --rm --privileged nerdctl

nerdctl兼容了docker命令，可以无痛切换

https://github.com/containerd/nerdctl

https://blog.csdn.net/weixin_39246554/article/details/120958424

查看名称空间

sudo nerdctl ns ls

查看image

nerdctl -n k8s.io image ls

拉取镜像，dockerhub

nerdctl pull nginx:latest
sudo nerdctl -n default image ls | grep nginx

查看容器

sudo nerdctl -n k8s.io container ls -a

拉取ecr镜像，需要确保在root用户下执行

aws ecr get-login-password --region cn-north-1 | nerdctl login --username AWS --password-stdin xxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn
WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.