OpenShift 4 Hands-on Lab (12) 通过 KubeFed 配置 OpenShift 联邦集群

Kubernetes Federation

Kubernetes联邦集群是由https://github.com/kubernetes-sigs主导开发，目前已经进入Kubefed v2的0.2.0-alpha.1版本。通过OpenShift内置的Kubefed Operator，我们可以在OpenShift之上运行联邦集群。在Kubefed中通过Controller Manager维护联邦集群中每个Kubernetes集群成员的资源状态。

Kubenetes Federation可以多个Kubernetes集群之间实现以下高可用功能：

• 简化管理多个集群的Kubernetes 组件(如Deployment, Service 等)。
• 在多个集群之间分散工作负载(容器)，以提升应用(服务)的可靠性。
• 跨集群的资源编排，依据编排策略在多个集群进行应用(服务)部署。
• 在不同集群中，能更快速更容易地迁移应用(服务)。
• 跨集群的服务发现，服务可以提供给当地存取，以降低延迟。
• 实践多云(Multi-cloud)或混合云(Hybird Cloud)的部署。

准备环境

OpenShift集群环境

我们需要2个基于4.x的OpenShift集群环境：cluster1和cluster2，其中在cluster1部署KubeFed（即Federation Manager）。

配置客户端环境

我们用Linux作为客户端运行环境，需要在客户端上配置2个集群的context，并且安装kubefedctl客户端。

1. 在客户端登录OpenShift集群cluster1和cluster2。

$ oc login <OCP_CLUSER1>
$ oc config rename-context $(oc config current-context) cluster1
$ oc login <OCP_CLUSER2>
$ oc config rename-context $(oc config current-context) cluster2

2. 先查看将客户端的Config Context，然后把客户端登录的两个Context Name分别设为cluster1和cluster1。（可参照《OpenShift 4 之 GitOps（4）用ArgoCD向Multi-Cluster发布应用》）。
3. 设置完后先将缺省Config设置到cluster1，然后再查看Config Context为以下cluster1和cluster2。

$ oc config use-context cluster1
$ oc config get-contexts
CURRENT   NAME       CLUSTER                                                              AUTHINFO                                                                         NAM            ESPACE
*         cluster1   api-cluster-beijing-c3c2-beijing-c3c2-example-opentlc-com:6443       opentlc-mgr/api-cluster-beijing-c3c2-beijing-c3c2-example-opentlc-com:6443       def            ault
          cluster2   api-cluster-beijing-3beb-beijing-3beb-sandbox1785-opentlc-com:6443   opentlc-mgr/api-cluster-beijing-3beb-beijing-3beb-sandbox1785-opentlc-com:6443   def            ault

4. 安装kubefedctl客户端

$ curl -LO https://github.com/kubernetes-sigs/kubefed/releases/download/v0.1.0-rc6/kubefedctl-0.1.0-rc6-linux-amd64.tgz
$ tar -xvf kubefedctl-0.1.0-rc6-linux-amd64.tgz
$ sudo mv kubefedctl /usr/local/bin/

安装配置KubeFed Operator

1. 在cluster1上创建运行KubeFed Operator的Namespace。说明：KubeFed缺省使用名为“kube-federation-system”的项目作为其运行环境。因此在缺省的时候，本文以下部分中的“–kubefed-namespace kube-federation-system”其实都可以省掉。如果KubeFed运行在其它项目名中，则以下相关命令都需要提供“–kubefed-namespace”参数。

$ oc --context=cluster1 create ns kube-federation-system

2. 用集群管理员进入OpenShift Console的Administrator视图中的OperatorHub（确认此时是在kube-federation-system项目中）。然后找到“Kubefed Operator”点击进入。
3. 点击Install，然后选择部署在kube-federation-system项目中，最后点击Subscribe。
4. 在Installed Operators中等到状态变为“Up to date”后进入该Kubefed Operator。
   
5. 查看当前运行的Pod。

$ oc --context=cluster1 -n kube-federation-system get pods
NAME                                READY   STATUS    RESTARTS   AGE
kubefed-operator-76b674b9fd-dsjkl   1/1     Running   0          88s

6. 在Kubefed Operator的Overview页面中找到KubeFedWebHook，点击Create Instance，然后接受缺省配置创建KubeFedWebHook。最后查看Pod的状态应该如下。

$ oc --context=cluster1 -n kube-federation-system get pods
NAME                                		READY   STATUS    RESTARTS   AGE
kubefed-operator-76b674b9fd-dsjkl   		1/1     Running   0          88s
kubefed-admission-webhook-89fbcbb74-dvdq6   1/1     Running   1          10h

7. 在Kubefed Operator的Overview页面中找到KubeFed，点击Create Instance，然后接受缺省配置创建KubeFed。最后查看Pod的状态应该如下。

$ oc --context=cluster1 -n kube-federation-system get pods
NAME                                          READY   STATUS    RESTARTS   AGE
kubefed-admission-webhook-89fbcbb74-dvdq6     1/1     Running   1          10h
kubefed-controller-manager-6d5f46d745-nd8fw   1/1     Running   1          10h
kubefed-controller-manager-6d5f46d745-p6f6b   1/1     Running   1          10h
kubefed-operator-76b674b9fd-dsjkl   		  1/1     Running   0          88s

8. 执行命令（也可在下图中的Kubefed Opeator中的All Instance中查看），查看联邦集群中的对象实例。

$ oc --context=cluster1 -n kube-federation-system get kubefedconfig 
NAME      AGE
kubefed   31h
$ oc --context=cluster1 -n kube-federation-system get kubefedwebhook
NAME                      AGE
kubefedwebhook-resource   31h
$ oc --context=cluster1 -n kube-federation-system get kubefed
NAME               AGE
kubefed-resource   31h

将OpenShift集群加入到联邦集群

1. 执行以下命令，将cluster1和cluster2加入到联邦集群(用cluster1作为主管)中。

$ kubefedctl join cluster1 --cluster-context cluster1 --host-cluster-context cluster1 --kubefed-namespace=kube-federation-system --v=2
I0313 02:09:12.434998    1289 join.go:159] Args and flags: name cluster1, host: cluster1, host-system-namespace: kube-federation-system, kubeconfig: , cluster-context: cluster1, secret-name: , dry-run: false
I0313 02:09:12.535522    1289 join.go:240] Performing preflight checks.
I0313 02:09:12.539877    1289 join.go:246] Creating kube-federation-system namespace in joining cluster
I0313 02:09:12.544239    1289 join.go:382] Already existing kube-federation-system namespace
I0313 02:09:12.544257    1289 join.go:254] Created kube-federation-system namespace in joining cluster
I0313 02:09:12.544267    1289 join.go:403] Creating service account in joining cluster: cluster1
I0313 02:09:12.553436    1289 join.go:413] Created service account: cluster1-cluster1 in joining cluster: cluster1
I0313 02:09:12.553451    1289 join.go:441] Creating cluster role and binding for service account: cluster1-cluster1 in joining cluster: cluster1
I0313 02:09:12.577425    1289 join.go:450] Created cluster role and binding for service account: cluster1-cluster1 in joining cluster: cluster1
I0313 02:09:12.577455    1289 join.go:809] Creating cluster credentials secret in host cluster
I0313 02:09:13.596025    1289 join.go:835] Using secret named: cluster1-cluster1-token-8dxrm
I0313 02:09:13.608147    1289 join.go:878] Created secret in host cluster named: cluster1-9d4j7
I0313 02:09:13.638964    1289 join.go:282] Created federated cluster resource
 
$ kubefedctl join cluster2 --cluster-context cluster2 --host-cluster-context cluster1 --kubefed-namespace=kube-federation-system --v=2
I0315 08:54:52.816368     799 join.go:159] Args and flags: name cluster2, host: cluster1, host-system-namespace: kube-federation-system, kubeconfig: , cluster-context: cluster2, secret-name: , dry-run: false
I0315 08:54:53.178787     799 join.go:238] Performing preflight checks.
I0315 08:54:53.411417     799 join.go:244] Creating kube-federation-system namespace in joining cluster
I0315 08:54:53.665019     799 join.go:252] Created kube-federation-system namespace in joining cluster
I0315 08:54:53.665060     799 join.go:398] Creating service account in joining cluster: cluster2
I0315 08:54:53.854303     799 join.go:408] Created service account: cluster2-cluster1 in joining cluster: cluster2
I0315 08:54:53.854334     799 join.go:436] Creating cluster role and binding for service account: cluster2-cluster1 in joining cluster: cluster2
I0315 08:54:54.413824     799 join.go:445] Created cluster role and binding for service account: cluster2-cluster1 in joining cluster: cluster2
I0315 08:54:54.413852     799 join.go:804] Creating cluster credentials secret in host cluster
I0315 08:54:54.743515     799 join.go:830] Using secret named: cluster2-cluster1-token-jhprj
I0315 08:54:54.754980     799 join.go:873] Created secret in host cluster named: cluster2-jpsq9
I0315 08:54:54.782983     799 join.go:280] Created federated cluster resource

2. (可选步骤)-执行命令可将cluster2从联邦集群中去掉。

$ kubefedctl unjoin cluster2 --cluster-context cluster2 --host-cluster-context cluster1 --kubefed-namespace=kube-federation-system --v=2

3. 查看联邦集群中的OpenShift集群都是READY=True状态。

$ oc --context=cluster1 -n kube-federation-system get kubefedclusters 
NAME       READY   AGE
cluster1   True    1m
cluster2   True    1m

4. 执行命令设置允许运行联邦对象类型。任何Kubernetes API支持的类型都可以成为对应的联邦类型（包括CRD类型），例如以下的FederatedDeployment、FederatedService。注意：这里只是允许部署这些联邦对象类型，而不是在联邦集群中对这些联邦对象执行复制操作。

for type in namespaces secrets serviceaccounts services configmaps deployments.apps scc
do
  kubefedctl enable $type --kubefed-namespace kube-federation-system
done

执行后会有以下输出，说明这些类型都已经注册到联邦集群的FederatedTypeConfig中了。

customresourcedefinition.apiextensions.k8s.io/federatednamespaces.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/namespaces updated in namespace kube-federation-system
customresourcedefinition.apiextensions.k8s.io/federatedsecrets.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/secrets updated in namespace kube-federation-system
customresourcedefinition.apiextensions.k8s.io/federatedserviceaccounts.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/serviceaccounts created in namespace kube-federation-system
customresourcedefinition.apiextensions.k8s.io/federatedservices.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/services created in namespace kube-federation-system
customresourcedefinition.apiextensions.k8s.io/federatedconfigmaps.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/configmaps created in namespace kube-federation-system
customresourcedefinition.apiextensions.k8s.io/federateddeployments.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/deployments.apps created in namespace kube-federation-system
customresourcedefinition.apiextensions.k8s.io/federatedsecuritycontextconstraints.types.kubefed.io created
federatedtypeconfig.core.kubefed.io/securitycontextconstraints.security.openshift.io created in namespace kube-federation-system

5. 查看联邦集群中FederatedTypeConfigAPI包括的API类型，结果对应上一步设置的联邦集群允许的API资源对象类型。

$ oc --context=cluster1 -n kube-federation-system get FederatedTypeConfig
NAME                                               AGE
configmaps                                         3m47s
deployments.apps                                   3m46s
namespaces                                         6m28s
secrets                                            5m56s
securitycontextconstraints.security.openshift.io   4m3s
serviceaccounts                                    3m48s
services                                           3m48s

6. （可选）：可以用以下命令diable一种Kubernetes API的类型。例如下面命令会从FederatedTypeConfig中将Deployment类型删除（并同时删除构成其联邦部署能力的相关CRD对象），使其不再具有在联邦集群中的传播复制能力。

$ kubefedctl disable deployments.apps --delete-crd

7. （可选）*：以下命令可以查看集群支持的所有Kubernetes API类型。

$ oc api-resources

向联邦集群部署应用资源

概念说明

kubefedctl的federate命令是用来根据Kubernetes API Tpye对象生成对应的Federated化对象。其中Federated化对象中template部分就是对应一般的Kubernetes API Tpye对象。

kubefedctl federate <target kubernetes API type> <target resource name> [flags]

以下每个“kubefedctl federate”命令都将一个特定的Kubernetes API类型Federated化，即生成对应的Federated对象。

kubefedctl federate namespace test-namespace --kubefed-namespace kube-federation-system
kubefedctl federate secrets test-secret -n test-namespace --kubefed-namespace kube-federation-system
kubefedctl federate serviceaccounts test-serviceaccount -n test-namespace --kubefed-namespace kube-federation-system
kubefedctl federate services test-service -n test-namespace --kubefed-namespace kube-federation-system
kubefedctl federate configmaps test-configmap -n test-namespace --kubefed-namespace kube-federation-system
kubefedctl federate deployments.apps test-deployment -n test-namespace --kubefed-namespace kube-federation-system
kubefedctl federate federatedsecuritycontextconstraints test-anyuid -n test-namespace --kubefed-namespace kube-federation-system

可以用“kubefedctl federate”命令的“–output string”参数将Federated化对象转为YAML格式输出。

操作

参照以下操作指导可实现在集群联邦中部署应用。

1. 先确认cluster1和cluster2上都没有test-namespace项目。

$ oc --context=cluster1 get project test-namespace
Error from server (NotFound): namespaces "test-namespace" not found
$ oc --context=cluster2 get project test-namespace
Error from server (NotFound): namespaces "test-namespace" not found

2. 在cluster1上创建test-namespace项目，然后确认cluster2还没有test-namespace项目。这是由于test-namespace项目还未加入到KubeFed中。

$ oc --context=cluster1 new-project test-namespace
$ oc --context=cluster1 get project test-namespace   
NAME             DISPLAY NAME   STATUS
test-namespace                  Active
$ oc --context=cluster2 get project test-namespace   
Error from server (NotFound): namespaces "test-namespace" not found

3. 执行命令，把名为“test-namespace”的namespace对象Federated化，既生成了一个名为“test-namespace”的federatednamespace对象。然后查看这个对象。
   说明：只有Federated化资源对象（例如下面的test-namespace项目）才能被Kubefed的Controller Manager复制到联邦集群的Member Cluster上。

$ kubefedctl federate namespace test-namespace --kubefed-namespace kube-federation-system
W0315 13:00:01.270320    1080 federate.go:410] Annotations defined for Namespace "test-namespace" will not appear in the template of the federated resource: map[openshift.io/sa.scc.mcs:s0:c23,c7 openshift.io/sa.scc.supplemental-groups:1000520000/10000 openshift.io/sa.scc.uid-range:1000520000/10000]
I0315 13:00:01.270486    1080 federate.go:474] Resource to federate is a namespace. Given namespace will itself be the container for the federated namespace
I0315 13:00:01.394914    1080 federate.go:503] Successfully created FederatedNamespace "test-namespace/test-namespace" from Namespace	

说明：如果一个namespace中已经有资源了，可以使用“–contents”参数将项目中的已有的资源类型自动Federated化。

kubefedctl federate namespace my-namespace --contents --kubefed-namespace kube-federation-system

4. 分别执行以下命令，查看上一步从名为test-namespace的namespace生成的同名联邦对象federatednamespace实例和器详细信息。可以从详细信息中看到federatednamespace的传播范围其是cluster1和cluster。

$ oc --context=cluster1 -n test-namespace get federatednamespace
NAME             AGE
test-namespace   3m
 
$ oc --context=cluster1 -n test-namespace get federatednamespace test-namespace -o yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  creationTimestamp: "2020-03-20T16:43:32Z"
  finalizers:
  - kubefed.io/sync-controller
  generation: 1
  name: test-namespace
  namespace: test-namespace
  resourceVersion: "1126829"
  selfLink: /apis/types.kubefed.io/v1beta1/namespaces/test-namespace/federatednamespaces/test-namespace
  uid: ef74af24-6ac9-11ea-92df-1222d546a5b5
spec:
  placement:
    clusterSelector:
      matchLabels: {}
  template:
    spec: {}
status:
  clusters:
  - name: cluster2
  - name: cluster1
  conditions:
  - lastTransitionTime: "2020-03-21T09:15:03Z"
    lastUpdateTime: "2020-03-21T09:15:03Z"
    status: "True"
    type: Propagation

5. 确认此时cluster2上也有了test-namespace项目。

$ oc --context=cluster2 get project test-namespace   
NAME              DISPLAY NAME   STATUS
test-namespace                   Active

6. 将项目名改为test-namespace1，然后重复以上（1）-（5）步骤，确认test-namespace1项目可以自动从cluster1复制到cluster2。
7. 下载测试应用项目。注意：以下地址的上游项目中sample-app目录里所有文件内容的apiversion命名空间已经不适合OpenShift 4。

$ git clone https://github.com/liuxiaoyu-git/federation-dev
$ cd federation-dev/archive

8. 由于我们后面部署的测试应用容器需要更高运行权限，因此需要执行以下命令为test-namespace项目提权。

$ oc --context=cluster1 -n test-namespace adm policy add-scc-to-user anyuid -z default
securitycontextconstraints.security.openshift.io/anyuid added to: ["system:serviceaccount:test-namespace:default"]
$ oc --context=cluster2 -n test-namespace adm policy add-scc-to-user anyuid -z default
securitycontextconstraints.security.openshift.io/anyuid added to: ["system:serviceaccount:test-namespace:default"]

9. 确认“federation-dev/archive/sample-app”中的所有文件内容中的“namespace”设置的是我们本次需要同步的项目“test-namespace”。
10. 部署联邦应用资源，包括federatedconfigmap、federateddeployment、federatedsecret、federatedservice、federatedserviceaccount。然后依次查看这些联邦对象。

$ oc --context=cluster1 -n test-namespace apply -R -f sample-app/
federatedconfigmap.types.kubefed.k8s.io/test-configmap created
federateddeployment.types.kubefed.k8s.io/test-deployment created
federatedsecret.types.kubefed.k8s.io/test-secret created
federatedservice.types.kubefed.k8s.io/test-service created
federatedserviceaccount.types.kubefed.k8s.io/test-serviceaccount created
 
$ oc --context=cluster1 -n test-namespace get federatedconfigmap
NAME             AGE
test-configmap   3m
$ oc --context=cluster1 -n test-namespace get federateddeployment
NAME              AGE
test-deployment   3m
 
$ oc --context=cluster1 -n test-namespace get federatedsecret
NAME          AGE
test-secret   3m
$ o --context=cluster1 -n test-namespacec get federatedserviceaccount
NAME                  AGE
test-serviceaccount   3m
$ oc --context=cluster1 -n test-namespace get federatedservice 
NAME           AGE
test-service   3m

11. 查看以上每个YAML文件，以federateddeployment.yaml文件为例，确认对象类型是“FederatedDeployment”；部署的“placement”包括cluster1和cluster2；cluster2的配置通过“overrides将运行Pod副本“/spec/replicas”修改为“5”份。

$ more sample-app/federateddeployment.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: test-deployment
  namespace: test-namespace
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx
            name: nginx
  placement:
    clusters:
    - name: cluster1
    - name: cluster2
  overrides:
 12. clusterName: cluster2
    clusterOverrides:
    - path: "/spec/replicas"
      value: 5

13. 查看cluster1和cluster2上的应用运行Pod副本数量。

$ oc --context=cluster1 -n test-namespace get deployment test-deployment
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
test-deployment   3         3         3            3           11m
$ oc --context=cluster2 -n test-namespace get deployment test-deployment 
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
test-deployment   5         5         5            5           11m

14. 由于OpenShift的Route是和每个集群的域名相关，因此我们先采用命令的方式为cluster1和cluster2的Service生成各自的Route。

$ oc --context=cluster1 -n test-namespace expose svc test-service
$ oc --context=cluster2 -n test-namespace expose svc test-service

15. 重复以上（8）-（13）步，将sample-app应用部署到在（7）创建test-namespace1项目中（需要将以上相关步骤中的所有命令和部署YAML文件中的“test-namespace”替换为“test-namespace1”）。

操控联邦集群上的应用部署

控制应用在联邦集群中的部署范围

1. 执行命令，将cluster2从联邦集群中去掉。

$ oc --context=cluster1 -n test-namespace patch federateddeployment test-deployment --type=merge -p '{"spec":{"placement":{"clusters": [{"name":"cluster1"}]}}}'
$ oc get kubefedclusters -n kube-federation-system --context=cluster1
NAME       READY   AGE
cluster1   True    29m

2. 执行命令，将cluster2加入到联邦集群中。

$ oc --context=cluster1 -n test-namespace patch federateddeployment test-deployment --type=merge -p '{"spec":{"placement":{"clusters": [{"name":"cluster1"}, {"name":"cluster2"}]}}}'
$ oc get kubefedclusters -n kube-federation-system --context=cluster1
NAME       READY   AGE
cluster1   True    20m
cluster2   True    1m

控制联邦集群中部署的Pod数量

场景1-联邦部署优先级高于集群部署

1. 手动直接将cluster2上应用的Deployment运行的Pod部分改为3，然后马上查看Deployment的Pod变化情况。确认“DESIRED”的Pod会先变为3个，然后又会变为5，最后“AVAILABLE”也会变为5。这说明是在联邦集群中即便可以通过手动修改特性OpenShift集群中的应用配置，但是联邦集群机制还会根据联邦集群上的配置将其修正过来，即FederatedDeployment的优先级高于Deployment。

$ oc --context=cluster2 -n test-namespace patch deployment test-deployment --type=merge -p '{"spec":{"replicas":"3"}}'
$ oc get deployment test-deployment --context=cluster2 -n test-namespace -w
NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
test-deployment   3         3         3            3           27m
test-deployment   5         3         3            3           27m
test-deployment   5         5         5            3           27m
test-deployment   5         5         5            5           28m

场景2-ReplicaSchedulingPreference优先级高于联邦部署

在联邦集群中，我们可以用ReplicaSchedulingPreference对象来制定在不同集群上分配部署副本数量的策略。当使用了ReplicaSchedulingPreference，系统将忽略FederatedDeloyment对象中在spec.replicas部分指定的部署副本数量。

1. 创建包含以下内容的rsp.yaml文件。它定义了一个ReplicaSchedulingPreference，让应用运行的副本为6个，且cluster1和cluster2上运行的Pod数量比例是1:2。

apiVersion: scheduling.kubefed.io/v1alpha1
kind: ReplicaSchedulingPreference
metadata:
  name: test-deployment
  namespace: test-namespace
spec:
  targetKind: FederatedDeployment
  totalReplicas: 6
  clusters:
    cluster1:
      weight: 1
    cluster2:
      weight: 2

2. 执行命令，在test-namespace项目中创建ReplicaSchedulingPreference对象。

$ oc --context=cluster apply -f rsp.yaml

3. 稍后执行命令，查看cluster1和cluster2的Deployment副本数量。可以确认联邦集群中部署这个应用的Pod数量已经成为6个，且在cluster1和cluster2上运行的Pod数量比例是1:2。

$ oc --context=cluster1 -n test-namespace get deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
test-deployment   2/2     2            2           13h
$ oc --context=cluster2 -n test-namespace get deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
test-deployment   4/4     4            4           13h

4. 查看名为test-deployment的federateddeployment。可以看到FederatedDeployment的缺省部署数量（即每个集群的replicas为3个）被Override了（即cluster1的replicas为2、cluster2的replicas为4）。另外从最下面的Events可以看出KubeFed的federateddeployment-controller先后更新了cluster1和cluster2中的Deployment。

$ oc --context=cluster1 -n test-namespace describe federateddeployment test-deployment
Name:         test-deployment
Namespace:    test-namespace
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"types.kubefed.io/v1beta1","kind":"FederatedDeployment","metadata":{"annotations":{},"name":"test-deployment","namespace":"t..."
API Version:  types.kubefed.io/v1beta1
Kind:         FederatedDeployment
Metadata:
  Creation Timestamp:  2020-03-21T02:28:30Z
  Finalizers:
    kubefed.io/sync-controller
  Generation:        3
  Resource Version:  1285158
  Self Link:         /apis/types.kubefed.io/v1beta1/namespaces/test-namespace/federateddeployments/test-deployment
  UID:               a7a800d7-6b1b-11ea-abac-1222d546a5b5
Spec:
  Overrides:
    Cluster Name:  cluster1
    Cluster Overrides:
      Path:        /spec/replicas
      Value:       2
    Cluster Name:  cluster2
    Cluster Overrides:
      Path:   /spec/replicas
      Value:  4
      Path:   /spec/template/spec/containers/0/image
      Value:  nginx:1.17.0-alpine
      Op:     add
      Path:   /metadata/annotations
      Value:
        Foo:  bar
      Op:     remove
      Path:   /metadata/annotations/foo
  Placement:
    Clusters:
      Name:  cluster2
      Name:  cluster1
  Template:
    Metadata:
      Labels:
        App:  nginx
    Spec:
      Replicas:  3
      Selector:
        Match Labels:
          App:  nginx
      Template:
        Metadata:
          Labels:
            App:  nginx
        Spec:
          Containers:
            Image:  nginx
            Name:   nginx
Status:
  Clusters:
    Name:  cluster2
    Name:  cluster1
  Conditions:
    Last Transition Time:  2020-03-21T15:38:30Z
    Last Update Time:      2020-03-21T15:38:30Z
    Status:                True
    Type:                  Propagation
Events:
  Type    Reason           Age                      From                            Message
  ----    ------           ----                     ----                            -------
  Normal  UpdateInCluster  32m (x2 over 46m)        federateddeployment-controller  Updating Deployment "test-namespace/test-deployment" in cluster "cluster1"
  Normal  UpdateInCluster  7m35s (x172 over 3h47m)  federateddeployment-controller  Updating Deployment "test-namespace/test-deployment" in cluster "cluster2"

场景3-关闭联邦集群中的一个集群

在完成以上场景2后，如果关闭联邦集群中的一个集群（例如cluster2），KubeFed会将6个Pod全部运行在cluster1上。

kubefedctl 命令

kubefedctl enable和kubefedctl disable子命令

这两个命令可以分别打开和关闭在联邦集群中传播Kubernetes API类型。

kubefedctl federate子命令

Kubernetes API Type转成Federated Type

我们可以用“kubefedctl federate”命令将一个一般的Kubernetes API Type转成Federated Type，例如将configmap变成federatedconfigmap。在实现过程中有以下两种方法：

1. 根据YAML定义文件

1. 查看一个定义ConfigMap对象的YAML文件configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-file
data:
  index.html: "Hello from Kubernetes Cluster Federation!"

2. 用“kubefedctl federate”命令转换成FederatedConfigMap，输出便是FederatedConfigMap的YAML。可以根据输出结果创建FederatedConfigMap。

$ kubefedctl federate -f configmap.yaml
---
apiVersion: types.kubefed.io/v1beta1
kind: FederatedConfigMap
metadata:
  name: web-file
spec:
  placement:
    clusterSelector:
      matchLabels: {}
  template:
    data:
      index.html: Hello from Kubernetes Cluster Federation!

2. 根据已有对象，生成Federated类型的对象

1. 执行命令，根据当前已有名为web-file的ConfigMap对象，生成对应的同名FederatedConfigMap对象。

$ kubefedctl federate cm web-file
W0322 01:57:10.549787    3583 federate.go:410] Annotations defined for ConfigMap "test-namespace/web-file" will not appear in the template of the federated resource: map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"v1","data":{"index.html":"Hello from Kubernetes Cluster Federation!"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"web-file","namespace":"test-namespace"}}
]
I0322 01:57:10.560774    3583 federate.go:503] Successfully created FederatedConfigMap "test-namespace/web-file" from ConfigMap

2. 确认已经生成FederatedConfigMap对象。

$ oc get fcm web-file
NAME       AGE
web-file   2m32s

删除Federated类型的对象

如果删除Federated类型的对象，那么会在联邦集群范围内的所有相关集群中删除其对应的Kubernetes API对象。例如当执行命令删除了fsvc（即FederatedService），OpenShift会删除cluster1和cluster2中和FederatedService对应的Service对象。

$ oc delete fsvc --all
federatedservice.types.kubefed.io "test-service" deleted

参考文档

4. https://github.com/kubernetes-sigs/kubefed/blob/master/docs/userguide.md
5. https://github.com/openshift/federation-dev/tree/master/archive
6. https://github.com/openshift/federation-dev/blob/master/archive/docs/ocp4-namespace-scoped.md
7. https://github.com/openshift/federation-dev/blob/master/archive/docs/ocp4-cluster-scoped.md
8. https://github.com/redhat-gpte-devopsautomation/rhte2019_multi_cloud
9. https://www.linkedin.com/pulse/openshift-4x-foundations-fundamentals-kubefed-dewan-i-ahmed
10. https://juejin.im/post/5dafe985f265da5b560e0d2d
11. https://juejin.im/post/5db6a43be51d452a01253454
12. https://www.kubernetes.org.cn/5702.html