OpenShift 4 - 在集群节点用crictl对Pod/Image/Container进行操作
文章目录
- 查看crictl命令的配置文件
- 用crictl命令操作Pod/Image/Container
- 参考
CRI-O 容器引擎为运行开放容器倡议 (OCI) 兼容的运行时提供了一个稳定、更安全、更高性能的平台。OpenShift 4集群不再使用 Docker 作为容器运行环境，而是用了 CRI-O 的容器环境，这样CRI-O就取代了Docker服务成为实现OpenShift容器平台的容器运行时接口（CRI）的容器引擎。通过使用 CRI-O 容器引擎，可以访问符合OCI的运行时（如默认的 OCI 运行时 runc或 Kata Containers）来启动容器和 Pod。在集群节点上，OpenShift 使用了 crictl 工具来访问 CRI-O 环境，进而实现对容器和相关资源进行操作。
有关OpenShift集群节点的kubelet、CRI-O环境的介绍，请参见《 OpenShift 4 - 节点是如何通过CRI-O运行容器的》。
查看crictl命令的配置文件
- 查看OpenShift集群的节点，然后进入一个worker类型的节点，最后切换到root用户。
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-134-103.ap-southeast-1.compute.internal Ready master 143m v1.18.3+012b3ec 10.0.134.103 Red Hat Enterprise Linux CoreOS 45.82.202007240629-0 (Ootpa) 4.18.0-193.13.2.el8_2.x86_64 cri-o://1.18.3-5.rhaos4.5.git1c13d1d.el8
ip-10-0-157-96.ap-southeast-1.compute.internal Ready worker 129m v1.18.3+012b3ec 10.0.157.96 Red Hat Enterprise Linux CoreOS 45.82.202007240629-0 (Ootpa) 4.18.0-193.13.2.el8_2.x86_64 cri-o://1.18.3-5.rhaos4.5.git1c13d1d.el8
ip-10-0-178-197.ap-southeast-1.compute.internal Ready worker 129m v1.18.3+012b3ec 10.0.178.197 Red Hat Enterprise Linux CoreOS 45.82.202007240629-0 (Ootpa) 4.18.0-193.13.2.el8_2.x86_64 cri-o://1.18.3-5.rhaos4.5.git1c13d1d.el8
ip-10-0-178-236.ap-southeast-1.compute.internal Ready master 143m v1.18.3+012b3ec 10.0.178.236 Red Hat Enterprise Linux CoreOS 45.82.202007240629-0 (Ootpa) 4.18.0-193.13.2.el8_2.x86_64 cri-o://1.18.3-5.rhaos4.5.git1c13d1d.el8
ip-10-0-221-178.ap-southeast-1.compute.internal Ready master 143m v1.18.3+012b3ec 10.0.221.178 Red Hat Enterprise Linux CoreOS 45.82.202007240629-0 (Ootpa) 4.18.0-193.13.2.el8_2.x86_64 cri-o://1.18.3-5.rhaos4.5.git1c13d1d.el8
$ oc debug node/
Starting pod/ip-10-0-157-96ap-southeast-1computeinternal-debug ...
To use host binaries, run `chroot /host`
sh-4.2# chroot /host
- 查看crictl的配置，缺省crictl连接的是该节点本地的“unix:///var/run/crio/crio.sock”
sh-4.4# cat /etc/crictl.yaml
runtime-endpoint: unix:///var/run/crio/crio.sock
用crictl命令操作Pod/Image/Container
- 查询该Node上名称包含machine-config-daemon的Pod实例，查到的Pod实例完整名称是“machine-config-daemon-8bblm”，它运行在“openshift-machine-config-operator”命名空间中。
sh-4.4# crictl pods --name machine-config-daemon
POD ID CREATED STATE NAME NAMESPACE ATTEMPT
ba978ea7afe83 2 hours ago Ready machine-config-daemon-8bblm openshift-machine-config-operator 0
sh-4.4# POD_ID=ba978ea7afe83
- 根据名为machine-config-daemon-8bblm的pod的短POD_ID找到完整POD_ID（需要去掉引号）。
sh-4.4# FULL_POD_ID=$(crictl inspectp $POD_ID | jq .status.id | cut -d "\"" -f 2)
sh-4.4# echo $FULL_POD_ID
ba978ea7afe836c1fd0189e6b9198a58dd931a6610dc8a0bc5084b3c489f634b
- 根据POD_ID查看pod信息。记录Pod用到Image的DIGEST信息。
sh-4.4# crictl inspectp $POD_ID | grep image
"image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb9ab6f21487d70c0fda256729adc82845aa3b68f9b84be18892d3096999d055",
sh-4.4# IMAGE_DIGEST=sha256:eb9ab6f21487d70c0fda256729adc82845aa3b68f9b84be18892d3096999d055
- 使用Image的DIGEST查完整的IMAGE_ID。
sh-4.4# crictl images --no-trunc --digests | grep $IMAGE_DIGEST
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 sha256:eb9ab6f21487d70c0fda256729adc82845aa3b68f9b84be18892d3096999d055 e66662827187986d2c58eba25a6300d4c7926c2042b7e021901f47975f10b54c 255MB
sh-4.4# IMAGE_ID=e66662827187986d2c58eba25a6300d4c7926c2042b7e021901f47975f10b54c
- 查看Image的详细配置。
sh-4.4# crictl inspecti $IMAGE_ID
{
"status": {
"id": "e66662827187986d2c58eba25a6300d4c7926c2042b7e021901f47975f10b54c",
"repoTags": [],
"repoDigests": [
"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eb9ab6f21487d70c0fda256729adc82845aa3b68f9b84be18892d3096999d055"
],
"size": "254920605",
"uid": {
"value": "0"
},
"username": ""
},
。。。
- 查看所有状态的Container，其中包括Running和Exited状态。
sh-4.4# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
59026587584a2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a60f5a4429d512c7bc8e438ba949b54583d16147e23d30aef9aaa836efa76783 44 minutes ago Running container-00 0 8a70054afce03
91030ef25af00 registry.redhat.io/distributed-tracing/jaeger-rhel7-operator@sha256:69f4fe261dab28d899b2e58c86e6f35bdd9b44aa45229b7bae0e474454e77aa9 2 hours ago Running jaeger-operator 0 62bd3cc3e6e6e
4be20be818779 registry.redhat.io/amq7/amq-streams-rhel7-operator@sha256:8637549bae76119b18fd1361120d9f2c68086425cc8c4751c3b8204b3942a7d9 2 hours ago Running cluster-operator 0 fb8e935887a77
。。。
a80e7f8aaed35 2b51d9d692993b904dad1169eb34f94be377a9eb34bc51577cf061f78b0e9f58 3 hours ago Exited whereabouts-cni 0 3a26a488f2350
463eac215f338 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3afc303dc91949d801b0fd806f53889d3ecff2b43307d6c7ddf3783c810ca10f 3 hours ago Exited whereabouts-cni-bincopy 0 3a26a488f2350
456b4c895975a quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72f164b5b9363614e2106950a502bbbeddb91b4fb6b239561d1ee6fc0d4563de 3 hours ago Exited routeoverride-cni 0 3a26a488f2350
db59dcee290fb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aba21f0cf0f8035ab6b786366df0422085d044ca27bae59e94cb19d64a955e5d 3 hours ago Exited cni-plugins 0 3a26a488f2350
- 使用前面设置的FULL_POD_ID查找该Pod包括的Container。可以看到名为“machine-config-daemon-8bblm”的Pod中运行了2个Container，分别是名为machine-config-daemon和oauth-proxy的Container。
sh-4.4# crictl ps --pod $FULL_POD_ID
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
a925dcb856164 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:21c49efa4fd9a4c6747c32cc6b2b0f877694d3fa5b3d3f66230129e603b152f0 2 hours ago Running oauth-proxy 0 ba978ea7afe83
cbfbb31e114ca quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3a8e6a396a9f399ed2493fe5c65ec8e8aecd0f83d45f162de63aef9c2d88400 2 hours ago Running machine-config-daemon 0 ba978ea7afe83
- 根据上一步执行结果设置名为machine-config-daemon容器的CONTAINER_ID和FULL_CONTAINER_ID。
sh-4.4# CONTAINER_ID=cbfbb31e114ca
sh-4.4# FULL_CONTAINER_ID=$(crictl inspect $CONTAINER_ID | jq .status.id | cut -d "\"" -f 2)
sh-4.4# echo $FULL_CONTAINER_ID
cbfbb31e114ca4789906666f97f51a641da3ea568d026cb7d5216a6d379bc731
- 直接在machine-config-daemon容器中执行命令。
sh-4.4# crictl exec -it $CONTAINER_ID ls
bin boot dev etc home lib lib64 manifests media mnt opt proc root rootfs run sbin srv sys tmp usr var
- 查看pod中名为machine-config-daemon容器的日志。
sh-4.4# crictl logs $CONTAINER_ID
I0909 10:36:06.499290 2719 start.go:74] Version: v4.5.0-202007240519.p0-dirty (99eb744f5094224edb60d88ca85d607ab151ebdf)
I0909 10:36:06.501247 2719 start.go:84] Calling chroot("/rootfs")
I0909 10:36:06.501428 2719 rpm-ostree.go:368] Running captured: rpm-ostree status --json
I0909 10:36:06.712099 2719 daemon.go:211] Booted osImageURL: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77e9ace116cec652637a79449dea00c6b4af5463ca2474463549cf145bc44438 (45.82.202007240629-0)
I0909 10:36:06.714599 2719 metrics.go:106] Registering Prometheus metrics
I0909 10:36:06.714664 2719 metrics.go:111] Starting metrics listener on 127.0.0.1:8797
I0909 10:36:06.715541 2719 update.go:1404] Starting to manage node: ip-10-0-157-96.ap-southeast-1.compute.internal
I0909 10:36:06.719018 2719 rpm-ostree.go:368] Running captured: rpm-ostree status
I0909 10:36:06.753767 2719 daemon.go:818] State: idle
Deployments:
pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77e9ace116cec652637a79449dea00c6b4af5463ca2474463549cf145bc44438
CustomOrigin: Managed by machine-config-operator
Version: 45.82.202007240629-0 (2020-07-24T06:33:19Z)
ostree://67315b4b010341ffd396fe699287defe530830b17879d695fec0243b87e97c82
Version: 45.82.202007141718-0 (2020-07-14T17:21:59Z)
I0909 10:36:06.753780 2719 rpm-ostree.go:368] Running captured: journalctl --list-boots
I0909 10:36:06.757759 2719 daemon.go:825] journalctl --list-boots:
-1 4ea87e5f3d75460c94379c39c37dcb7b Wed 2020-09-09 10:30:54 UTC—Wed 2020-09-09 10:34:20 UTC
0 4afde637b05b409884433b6a359f10dc Wed 2020-09-09 10:34:35 UTC—Wed 2020-09-09 10:36:06 UTC
I0909 10:36:06.757771 2719 daemon.go:568] Starting MachineConfigDaemon
I0909 10:36:06.757840 2719 daemon.go:575] Enabling Kubelet Healthz Monitor
I0909 10:36:36.716933 2719 trace.go:116] Trace[1106410694]: "Reflector ListAndWatch" name:github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101 (started: 2020-09-09 10:36:06.716511929 +0000 UTC m=+0.241089202) (total time: 30.000399506s):
Trace[1106410694]: [30.000399506s] [30.000399506s] END
E0909 10:36:36.716955 2719 reflector.go:178] github.com/openshift/machine-config-operator/pkg/generated/informers/externalversions/factory.go:101: Failed to list *v1.MachineConfig: Get https://172.30.0.1:443/apis/machineconfiguration.openshift.io/v1/machineconfigs?limit=500&resourceVersion=0: dial tcp 172.30.0.1:443: i/o timeout
I0909 10:36:36.716931 2719 trace.go:116] Trace[646203300]: "Reflector ListAndWatch" name:k8s.io/client-go/informers/factory.go:135 (started: 2020-09-09 10:36:06.716166848 +0000 UTC m=+0.240744117) (total time: 30.00073537s):
Trace[646203300]: [30.00073537s] [30.00073537s] END
- 查看运行CONTAINER_ID占用的环境资源。
sh-4.4# crictl stats --id $CONTAINER_ID
CONTAINER CPU % MEM DISK INODES
cbfbb31e114ca 0.05 77.86MB 62B 4
- 根据FULL_CONTAINER_ID，在其对应的存储目录中查看machine-config-daemon容器的配置。
sh-4.4# more /run/containers/storage/overlay-containers/${FULL_CONTAINER_ID}/userdata/config.json
{
"ociVersion": "1.0.2",
"process": {
"user": {
"uid": 0,
"gid": 0
},
"args": [
"/usr/bin/machine-config-daemon",
"start"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=ip-10-0-157-96",
"NSS_SDB_USE_CACHE=no",
"NODE_NAME=ip-10-0-157-96.ap-southeast-1.compute.internal",
"KUBERNETES_PORT=tcp://172.30.0.1:443",
"KUBERNETES_PORT_443_TCP_PORT=443",
"MACHINE_CONFIG_DAEMON_PORT_9001_TCP_PROTO=tcp",
"MACHINE_CONFIG_DAEMON_PORT_9001_TCP_ADDR=172.30.171.237",
"MACHINE_CONFIG_DAEMON_PORT_9001_TCP_PORT=9001",
"KUBERNETES_SERVICE_HOST=172.30.0.1",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"MACHINE_CONFIG_DAEMON_SERVICE_PORT_METRICS=9001",
"MACHINE_CONFIG_DAEMON_PORT=tcp://172.30.171.237:9001",
"MACHINE_CONFIG_DAEMON_PORT_9001_TCP=tcp://172.30.171.237:9001",
"KUBERNETES_SERVICE_PORT=443",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
"KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
"MACHINE_CONFIG_DAEMON_SERVICE_HOST=172.30.171.237",
"MACHINE_CONFIG_DAEMON_SERVICE_PORT=9001",
"__doozer=merge",
"BUILD_RELEASE=202007240519.p0",
"BUILD_VERSION=v4.5.0",
"OS_GIT_MAJOR=4",
"OS_GIT_MINOR=5",
"OS_GIT_PATCH=0",
"OS_GIT_TREE_STATE=clean",
"OS_GIT_VERSION=4.5.0-202007240519.p0-99eb744",
"SOURCE_GIT_TREE_STATE=clean",
"OS_GIT_COMMIT=99eb744",
"SOURCE_DATE_EPOCH=1595455266",
"SOURCE_GIT_COMMIT=99eb744f5094224edb60d88ca85d607ab151ebdf",
"SOURCE_GIT_TAG=machine-config-daemon-4.5.0-202006231303.p0-12-g99eb744f",
"SOURCE_GIT_URL=https://github.com/openshift/machine-config-operator",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
],
。。。
参考
https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md