不像OpenShift 3和底层RHEL的那种松散关系(安装和维护都是相对独立的),OpenShift 4 和RHCOS(Red Hat CoreOS)操作系统是紧密依赖关系,它们是一起安装、一起运行。
为了获得Immutable Architecture(不可变架构)的优势,RedHat官方不建议直接进入RHCOS操作系统进行操作系统配置修改,而建议通过OpenShift 4的Operator机制修改和操作RHCOS的配置。无论是运行在节点操作系统上的kubelet以及CRI-O用到的Registry配置、还是Kernel、NetworkManager或系统时间同步,甚至连RHCOS升级都是通过Operator完成的。因此,从某个角度上可以把看作是OpenShift 4用Operator控制的一种Kubernetes资源。
OpenShift 4 是使用系统级的Machine Config Operator(MCO)相关对象实现集群对节点的RHCOS管理。MCO及其相关组件都运行在openshift-machine-config-operator项目中,可以执行以下命令查看该项目下的主要对象。
$ oc get all -n openshift-machine-config-operator
NAME READY STATUS RESTARTS AGE
pod/etcd-quorum-guard-7666889cd6-2hch9 1/1 Running 0 20h
pod/etcd-quorum-guard-7666889cd6-c6m8k 1/1 Running 0 20h
pod/etcd-quorum-guard-7666889cd6-ht7bb 1/1 Running 0 20h
pod/machine-config-controller-7574fd5777-zssfz 1/1 Running 0 20h
pod/machine-config-daemon-2bfzg 2/2 Running 0 21h
pod/machine-config-daemon-2g5kw 2/2 Running 0 21h
pod/machine-config-daemon-dw2r9 2/2 Running 0 21h
pod/machine-config-daemon-gg5vd 2/2 Running 0 21h
pod/machine-config-daemon-w6cg2 2/2 Running 0 21h
pod/machine-config-operator-6477b6c7c8-vcxrt 1/1 Running 0 20h
pod/machine-config-server-5bxmf 1/1 Running 0 20h
pod/machine-config-server-8h78p 1/1 Running 0 20h
pod/machine-config-server-lqxql 1/1 Running 0 20h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/machine-config-daemon ClusterIP 172.30.9.67 <none> 9001/TCP 9d
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/machine-config-daemon 5 5 5 5 5 kubernetes.io/os=linux 9d
daemonset.apps/machine-config-server 3 3 3 3 3 node-role.kubernetes.io/master= 9d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/etcd-quorum-guard 3/3 3 3 9d
deployment.apps/machine-config-controller 1/1 1 1 9d
deployment.apps/machine-config-operator 1/1 1 1 9d
NAME DESIRED CURRENT READY AGE
replicaset.apps/etcd-quorum-guard-5fc4989788 0 0 0 9d
replicaset.apps/etcd-quorum-guard-7666889cd6 3 3 3 20h
replicaset.apps/machine-config-controller-67b8b87bb8 0 0 0 9d
replicaset.apps/machine-config-controller-7574fd5777 1 1 1 21h
replicaset.apps/machine-config-operator-5b87d8f4c5 0 0 0 9d
replicaset.apps/machine-config-operator-6477b6c7c8 1 1 1 21h
Machine Config Operator是通过基于CRD的对象来定义以上这些和machineconfig相关的对象的。我们可以查看在OpenShift内部定义的与machineconfig相关的CRD对象:
$ oc get crd -n openshift-config | grep machineconfig
containerruntimeconfigs.machineconfiguration.openshift.io 2020-09-09T10:17:19Z
controllerconfigs.machineconfiguration.openshift.io 2020-09-09T10:24:12Z
kubeletconfigs.machineconfiguration.openshift.io 2020-09-09T10:17:20Z
machineconfigpools.machineconfiguration.openshift.io 2020-09-09T10:17:22Z
machineconfigs.machineconfiguration.openshift.io 2020-09-09T10:17:21Z
在OpenShift 4中,Node和MachineConfig是多对多关系,即一个Node可以使用多个MachineConfig,同时一个MachineConfig可以属于多个Node。一个Node使用哪个MachineConfig配置是MachineConfigPool决定的,即MachineConfigPool将MachineConfig和Node关联起来。
$ oc get machineconfig
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
00-master 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
00-worker 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
01-master-container-runtime 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
01-master-kubelet 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
01-worker-container-runtime 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
01-worker-kubelet 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
99-master-58a1f119-d339-4e40-b218-ca459783104e-registries 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
99-master-ssh 2.2.0 3d21h
99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
99-worker-ssh 2.2.0 3d21h
rendered-master-68ea973a611b4e8a11a6492328c661bd 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f 99eb744f5094224edb60d88ca85d607ab151ebdf 2.2.0 3d21h
$ oc get node
NAME STATUS ROLES AGE VERSION
ip-10-0-139-50.ap-southeast-1.compute.internal Ready master 3d21h v1.18.3+012b3ec
ip-10-0-155-228.ap-southeast-1.compute.internal Ready worker 3d21h v1.18.3+012b3ec
ip-10-0-172-203.ap-southeast-1.compute.internal Ready master 3d21h v1.18.3+012b3ec
ip-10-0-190-239.ap-southeast-1.compute.internal Ready worker 3d21h v1.18.3+012b3ec
ip-10-0-205-123.ap-southeast-1.compute.internal Ready master 3d21h v1.18.3+012b3ec
$ oc get machineconfigpool
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-68ea973a611b4e8a11a6492328c661bd True False False 3 3 3 0 3d21h
worker rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f True False False 2 2 2 0 3d21h
$ oc describe MachineConfigPool worker
...
Spec:
Configuration:
Name: rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Machine Config Selector:
Match Labels:
machineconfiguration.openshift.io/role: worker
Node Selector:
Match Labels:
node-role.kubernetes.io/worker:
...
$ RENDERED_WORKER_MACHINE_CONFIG=$(oc get machineconfigpool worker --template={
{
.spec.configuration.name}})
$ oc describe machineconfigs ${RENDERED_WORKER_MACHINE_CONFIG} | grep Path:
Path: /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
Path: /etc/tmpfiles.d/cleanup-cni.conf
Path: /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
Path: /etc/containers/storage.conf
Path: /etc/NetworkManager/dispatcher.d/90-long-hostname
Path: /etc/modules-load.d/iptables.conf
Path: /etc/kubernetes/kubelet-ca.crt
Path: /etc/systemd/system.conf.d/kubelet-cgroups.conf
Path: /etc/NetworkManager/conf.d/sdn.conf
Path: /var/lib/kubelet/config.json
Path: /etc/kubernetes/ca.crt
Path: /etc/sysctl.d/forward.conf
Path: /etc/sysctl.d/inotify.conf
Path: /usr/local/sbin/set-valid-hostname.sh
Path: /etc/kubernetes/kubelet-plugins/volume/exec/.dummy
Path: /etc/containers/registries.conf
Path: /etc/crio/crio.conf.d/00-default
Path: /etc/containers/policy.json
Path: /etc/kubernetes/cloud.conf
Path: /etc/kubernetes/kubelet.conf
$ oc debug node/
sh-4.4# cat /etc/containers/registries.conf
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 50-worker-container-registries
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:,%5Bregistries.search%5D%0Aregistries%20%3D%20%5B'registry.access.redhat.com'%2C%20'docker.io'%2C%20'quay.io'%5D%0A%0A%5Bregistries.insecure%5D%0Aregistries%20%3D%20%5B%5D%0A%0A%5Bregistries.block%5D%0Aregistries%20%3D%20%5B%5D%0A
filesystem: root
mode: 420
path: /etc/containers/registries.conf
EOF
$ oc create -f 50-worker-container-registries.yaml
$ oc get nodes -l node-role.kubernetes.io/worker -w
NAME STATUS ROLES AGE VERSION
ip-10-0-155-228.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready worker 8d v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal Ready worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal Ready worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready,SchedulingDisabled worker 8d v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal Ready worker 8d v1.18.3+6c42de8
sh-4.4# cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'quay.io']
[registries.insecure]
registries = []
[registries.block]
registries = []
sh-4.4# exit
exit
sh-4.2# exit
exit
$ oc get machineconfigpool
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-68ea973a611b4e8a11a6492328c661bd True False False 3 3 3 0 9d
worker rendered-worker-90e84f625715399f91f5ffff66019273 True False False 2 2 2 0 9d
。。。
Spec:
Configuration:
Name: rendered-worker-90e84f625715399f91f5ffff66019273
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-worker
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-worker-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 50-worker-container-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-worker-ssh
Machine Config Selector:
Match Labels:
machineconfiguration.openshift.io/role: worker
Node Selector:
Match Labels:
node-role.kubernetes.io/worker:
...
$ oc describe machineconfigs ${RENDERED_WORKER_MACHINE_CONFIG} | grep Path:
Path: /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
Path: /etc/tmpfiles.d/cleanup-cni.conf
Path: /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
Path: /etc/containers/storage.conf
Path: /etc/NetworkManager/dispatcher.d/90-long-hostname
Path: /etc/modules-load.d/iptables.conf
Path: /etc/kubernetes/kubelet-ca.crt
Path: /etc/systemd/system.conf.d/kubelet-cgroups.conf
Path: /etc/NetworkManager/conf.d/sdn.conf
Path: /var/lib/kubelet/config.json
Path: /etc/kubernetes/ca.crt
Path: /etc/sysctl.d/forward.conf
Path: /etc/sysctl.d/inotify.conf
Path: /usr/local/sbin/set-valid-hostname.sh
Path: /etc/kubernetes/kubelet-plugins/volume/exec/.dummy
Path: /etc/containers/registries.conf
Path: /etc/crio/crio.conf.d/00-default
Path: /etc/containers/policy.json
Path: /etc/kubernetes/cloud.conf
Path: /etc/kubernetes/kubelet.conf
Path: /etc/containers/registries.conf
https://github.com/openshift/machine-config-operator
https://github.com/coreos/ignition/blob/master/doc/configuration-v3_0.md
https://www.redhat.com/en/blog/openshift-container-platform-4-how-does-machine-config-pool-work
http://jstakun.blogspot.com/2019/06/managing-cluster-nodes-configuration-in.html