OpenShift 4 - 如何用Machine Config Operator修改集群节点CoreOS的配置

OpenShift 4 和 RHCOS

不像OpenShift 3和底层RHEL的那种松散关系（安装和维护都是相对独立的），OpenShift 4 和RHCOS（Red Hat CoreOS）操作系统是紧密依赖关系，它们是一起安装、一起运行。
为了获得Immutable Architecture（不可变架构）的优势，RedHat官方不建议直接进入RHCOS操作系统进行操作系统配置修改，而建议通过OpenShift 4的Operator机制修改和操作RHCOS的配置。无论是运行在节点操作系统上的kubelet以及CRI-O用到的Registry配置、还是Kernel、NetworkManager或系统时间同步，甚至连RHCOS升级都是通过Operator完成的。因此，从某个角度上可以把看作是OpenShift 4用Operator控制的一种Kubernetes资源。

MachineConfig相关概念

MachineConfigOperator 及相关对象

OpenShift 4 是使用系统级的Machine Config Operator（MCO）相关对象实现集群对节点的RHCOS管理。MCO及其相关组件都运行在openshift-machine-config-operator项目中，可以执行以下命令查看该项目下的主要对象。

$ oc get all -n openshift-machine-config-operator
NAME                                             READY   STATUS    RESTARTS   AGE
pod/etcd-quorum-guard-7666889cd6-2hch9           1/1     Running   0          20h
pod/etcd-quorum-guard-7666889cd6-c6m8k           1/1     Running   0          20h
pod/etcd-quorum-guard-7666889cd6-ht7bb           1/1     Running   0          20h
pod/machine-config-controller-7574fd5777-zssfz   1/1     Running   0          20h
pod/machine-config-daemon-2bfzg                  2/2     Running   0          21h
pod/machine-config-daemon-2g5kw                  2/2     Running   0          21h
pod/machine-config-daemon-dw2r9                  2/2     Running   0          21h
pod/machine-config-daemon-gg5vd                  2/2     Running   0          21h
pod/machine-config-daemon-w6cg2                  2/2     Running   0          21h
pod/machine-config-operator-6477b6c7c8-vcxrt     1/1     Running   0          20h
pod/machine-config-server-5bxmf                  1/1     Running   0          20h
pod/machine-config-server-8h78p                  1/1     Running   0          20h
pod/machine-config-server-lqxql                  1/1     Running   0          20h
 
NAME                            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/machine-config-daemon   ClusterIP   172.30.9.67   <none>        9001/TCP   9d
 
NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/machine-config-daemon   5         5         5       5            5           kubernetes.io/os=linux            9d
daemonset.apps/machine-config-server   3         3         3       3            3           node-role.kubernetes.io/master=   9d
 
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/etcd-quorum-guard           3/3     3            3           9d
deployment.apps/machine-config-controller   1/1     1            1           9d
deployment.apps/machine-config-operator     1/1     1            1           9d
 
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/etcd-quorum-guard-5fc4989788           0         0         0       9d
replicaset.apps/etcd-quorum-guard-7666889cd6           3         3         3       20h
replicaset.apps/machine-config-controller-67b8b87bb8   0         0         0       9d
replicaset.apps/machine-config-controller-7574fd5777   1         1         1       21h
replicaset.apps/machine-config-operator-5b87d8f4c5     0         0         0       9d
replicaset.apps/machine-config-operator-6477b6c7c8     1         1         1       21h

在MCO中包括以下三个子组件，MCS、MCC和MCD：

1. MCS（MachineConfigServer）：为新加入集群的新Machine提供Ignition配置。在openshift-machine-config-operator项目中它是以DaemonSet的形式运行在。可以看到MCS会在每个Master节点上都运行一份，新加入节点是通过haproxy访问MCS的。
   
2. MCC（MachineConfigController）：当使用MachineConfig定义了配置，MCC负责协调所有的Machines进行升级。它在openshift-machine-config-operator项目是以Deployment的方式存在的。
   1）Template Controller：负责从内部模板中为machine的预定义角色（缺省是master, worker）生成MachineConfigs。
   2）Render Controller：负责发现MachineConfigPoold的MachineConfigs，并生成静态MachineConfig定义文件（例如rendered-master-XXXX和rendered-worker-XXXX）。
   3）Update Controller：负责与每台machine上运行的MCD程协调，将machine升级到MachineConfig定义的。
   4）KubeletConfigController ：负责将获得的KubeletConfig CRD配置对象实施到节点的kubelet运行环境中。
3. MCD（MachineConfigDaemon）：负责在升级过程中对Machine实施新的MachineConfig。它在openshift-machine-config-operator项目中MCD是以DaemonSet运行在集群中的所有节点上。另外MCD还负责通过OSTree对RHCOS进行滚动升级。

Machine Config Operator是通过基于CRD的对象来定义以上这些和machineconfig相关的对象的。我们可以查看在OpenShift内部定义的与machineconfig相关的CRD对象：

$ oc get crd -n openshift-config | grep machineconfig
containerruntimeconfigs.machineconfiguration.openshift.io   2020-09-09T10:17:19Z
controllerconfigs.machineconfiguration.openshift.io         2020-09-09T10:24:12Z
kubeletconfigs.machineconfiguration.openshift.io            2020-09-09T10:17:20Z
machineconfigpools.machineconfiguration.openshift.io        2020-09-09T10:17:22Z
machineconfigs.machineconfiguration.openshift.io            2020-09-09T10:17:21Z

MachineConfigPool

在OpenShift 4中，Node和MachineConfig是多对多关系，即一个Node可以使用多个MachineConfig，同时一个MachineConfig可以属于多个Node。一个Node使用哪个MachineConfig配置是MachineConfigPool决定的，即MachineConfigPool将MachineConfig和Node关联起来。

1. 查看machineconfig

$ oc get machineconfig
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                                   99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
00-worker                                                   99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
01-master-container-runtime                                 99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
01-master-kubelet                                           99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
01-worker-container-runtime                                 99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
01-worker-kubelet                                           99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
99-master-58a1f119-d339-4e40-b218-ca459783104e-registries   99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
99-master-ssh                                                                                          2.2.0             3d21h
99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries   99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
99-worker-ssh                                                                                          2.2.0             3d21h
rendered-master-68ea973a611b4e8a11a6492328c661bd            99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h
rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f            99eb744f5094224edb60d88ca85d607ab151ebdf   2.2.0             3d21h

2. 查看node

$ oc get node
NAME                                              STATUS   ROLES    AGE     VERSION
ip-10-0-139-50.ap-southeast-1.compute.internal    Ready    master   3d21h   v1.18.3+012b3ec
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready    worker   3d21h   v1.18.3+012b3ec
ip-10-0-172-203.ap-southeast-1.compute.internal   Ready    master   3d21h   v1.18.3+012b3ec
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready    worker   3d21h   v1.18.3+012b3ec
ip-10-0-205-123.ap-southeast-1.compute.internal   Ready    master   3d21h   v1.18.3+012b3ec

3. 查看machineconfigpool。

$ oc get machineconfigpool
NAME     CONFIG                                              UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-68ea973a611b4e8a11a6492328c661bd   True      False      False      3              3                   3                     0                      3d21h
worker   rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f   True      False      False      2              2                   2                     0                      3d21h
$ oc describe MachineConfigPool worker
...
Spec:
  Configuration:
    Name:  rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-worker
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-ssh
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  worker
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/worker:
...

4. 对于名为worker的MachineConfigPool，查看被RenderController加工后所包含到的相关配置文件。

$ RENDERED_WORKER_MACHINE_CONFIG=$(oc get machineconfigpool worker --template={
      {
      .spec.configuration.name}})
$ oc describe machineconfigs ${RENDERED_WORKER_MACHINE_CONFIG} | grep Path:
        Path:        /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
        Path:        /etc/tmpfiles.d/cleanup-cni.conf
        Path:        /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
        Path:        /etc/containers/storage.conf
        Path:        /etc/NetworkManager/dispatcher.d/90-long-hostname
        Path:        /etc/modules-load.d/iptables.conf
        Path:        /etc/kubernetes/kubelet-ca.crt
        Path:        /etc/systemd/system.conf.d/kubelet-cgroups.conf
        Path:        /etc/NetworkManager/conf.d/sdn.conf
        Path:        /var/lib/kubelet/config.json
        Path:        /etc/kubernetes/ca.crt
        Path:        /etc/sysctl.d/forward.conf
        Path:        /etc/sysctl.d/inotify.conf
        Path:        /usr/local/sbin/set-valid-hostname.sh
        Path:        /etc/kubernetes/kubelet-plugins/volume/exec/.dummy
        Path:        /etc/containers/registries.conf
        Path:        /etc/crio/crio.conf.d/00-default
        Path:        /etc/containers/policy.json
        Path:        /etc/kubernetes/cloud.conf
        Path:        /etc/kubernetes/kubelet.conf

通过MachineConfig修改Worker节点CRI-O使用的registries.conf配置

1. 在Terminal -1执行命令进入一个Worker Node，然后查看“/etc/containers/registries.conf”文件内容，确认文件里的内容。

$ oc debug node/
sh-4.4# cat /etc/containers/registries.conf
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']

2. 在Terminal-2创建内容如下的“50-worker-container-registries.yaml文件”。

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-worker-container-registries
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:,%5Bregistries.search%5D%0Aregistries%20%3D%20%5B'registry.access.redhat.com'%2C%20'docker.io'%2C%20'quay.io'%5D%0A%0A%5Bregistries.insecure%5D%0Aregistries%20%3D%20%5B%5D%0A%0A%5Bregistries.block%5D%0Aregistries%20%3D%20%5B%5D%0A
        filesystem: root
        mode: 420
        path: /etc/containers/registries.conf
EOF

3. 在Terminal-2中执行命令，创建MachineConfig。

$ oc create -f 50-worker-container-registries.yaml

4. 在Terminal-2中查看worker节点的状态，通过状态（SchedulingDisabled）确认2个worker节点逐一进行了配置更新。

$ oc get nodes -l node-role.kubernetes.io/worker -w
NAME                                              STATUS                     ROLES    AGE   VERSION
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.18.3+6c42de8
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.18.3+6c42de8

5. 在Terminal-1中查看Worker节点的“/etc/containers/registries.conf”f配置已经被修改。

sh-4.4#  cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'quay.io']
 
[registries.insecure]
registries = []
 
[registries.block]
registries = []
sh-4.4# exit
exit
sh-4.2# exit
exit

6. 再次查看machineconfigpool，确认worker使用的rendered后的配置已经变了。

$ oc get machineconfigpool
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-68ea973a611b4e8a11a6492328c661bd   True      False      False      3              3                   3                     0                      9d
worker   rendered-worker-90e84f625715399f91f5ffff66019273   True      False      False      2              2                   2                     0                      9d

7. 再次查看名为worker的machineconfigpool的配置信息，其中包括前面新加的名为“50-worker-container-registries”的MachineConfig。

。。。
Spec:
  Configuration:
    Name:  rendered-worker-90e84f625715399f91f5ffff66019273
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-worker
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         50-worker-container-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-ssh
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  worker
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/worker:
...

8. 再次查看名为worker的machineconfigs中对应的配置文件，确认最后一行已经有对“registries.conf”文件的配置了。

$ oc describe machineconfigs ${RENDERED_WORKER_MACHINE_CONFIG} | grep Path:
        Path:        /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
        Path:        /etc/tmpfiles.d/cleanup-cni.conf
        Path:        /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
        Path:        /etc/containers/storage.conf
        Path:        /etc/NetworkManager/dispatcher.d/90-long-hostname
        Path:        /etc/modules-load.d/iptables.conf
        Path:        /etc/kubernetes/kubelet-ca.crt
        Path:        /etc/systemd/system.conf.d/kubelet-cgroups.conf
        Path:        /etc/NetworkManager/conf.d/sdn.conf
        Path:        /var/lib/kubelet/config.json
        Path:        /etc/kubernetes/ca.crt
        Path:        /etc/sysctl.d/forward.conf
        Path:        /etc/sysctl.d/inotify.conf
        Path:        /usr/local/sbin/set-valid-hostname.sh
        Path:        /etc/kubernetes/kubelet-plugins/volume/exec/.dummy
        Path:        /etc/containers/registries.conf
        Path:        /etc/crio/crio.conf.d/00-default
        Path:        /etc/containers/policy.json
        Path:        /etc/kubernetes/cloud.conf
        Path:        /etc/kubernetes/kubelet.conf
        Path:        /etc/containers/registries.conf

参考

https://github.com/openshift/machine-config-operator
https://github.com/coreos/ignition/blob/master/doc/configuration-v3_0.md
https://www.redhat.com/en/blog/openshift-container-platform-4-how-does-machine-config-pool-work
http://jstakun.blogspot.com/2019/06/managing-cluster-nodes-configuration-in.html