一 部署高可用kube-scheduler
1.1 高可用kube-scheduler介绍
本实验部署一个三实例 kube-scheduler 的集群,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节点不可用时,阻塞的节点将再次进行选举产生新的 leader 节点,从而保证服务的可用性。
为保证通信安全,本文档先生成 x509 证书和私钥,kube-controller-manager 在如下两种情况下使用该证书:
- 与 kube-apiserver 的安全端口通信;
- 在安全端口(https,10251) 输出 prometheus 格式的 metrics。
1.2 创建kube-scheduler证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat > kube-scheduler-csr.json <3 { 4 "CN": "system:kube-scheduler", 5 "hosts": [ 6 "127.0.0.1", 7 "172.24.8.71", 8 "172.24.8.72", 9 "172.24.8.73" 10 ], 11 "key": { 12 "algo": "rsa", 13 "size": 2048 14 }, 15 "names": [ 16 { 17 "C": "CN", 18 "ST": "Shanghai", 19 "L": "Shanghai", 20 "O": "system:kube-scheduler", 21 "OU": "System" 22 } 23 ] 24 } 25 EOF 26 #创建kube-scheduler的CA证书请求文件
解释:
hosts 列表包含所有 kube-scheduler 节点 IP;
CN 和 O 均为 system:kube-scheduler,kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限。
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cfssl gencert -ca=/opt/k8s/work/ca.pem \ 3 -ca-key=/opt/k8s/work/ca-key.pem -config=/opt/k8s/work/ca-config.json \ 4 -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler #生成CA密钥(ca-key.pem)和证书(ca.pem)
1.3 分发证书和私钥
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp kube-scheduler*.pem root@${master_ip}:/etc/kubernetes/cert/ 7 done
1.4 创建和分发kubeconfig
kube-scheduler 使用 kubeconfig 文件访问 apiserver,该文件提供了 apiserver 地址、嵌入的 CA 证书和 kube-scheduler 证书:
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# kubectl config set-cluster kubernetes \ 4 --certificate-authority=/opt/k8s/work/ca.pem \ 5 --embed-certs=true \ 6 --server=${KUBE_APISERVER} \ 7 --kubeconfig=kube-scheduler.kubeconfig 8 9 [root@k8smaster01 work]# kubectl config set-credentials system:kube-scheduler \ 10 --client-certificate=kube-scheduler.pem \ 11 --client-key=kube-scheduler-key.pem \ 12 --embed-certs=true \ 13 --kubeconfig=kube-scheduler.kubeconfig 14 15 [root@k8smaster01 work]# kubectl config set-context system:kube-scheduler \ 16 --cluster=kubernetes \ 17 --user=system:kube-scheduler \ 18 --kubeconfig=kube-scheduler.kubeconfig 19 20 [root@k8smaster01 work]# kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig 21 22 [root@k8smaster01 ~]# cd /opt/k8s/work 23 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 24 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 25 do 26 echo ">>> ${master_ip}" 27 scp kube-scheduler.kubeconfig root@${master_ip}:/etc/kubernetes/ 28 done
1.5 创建kube-scheduler 配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# cat >kube-scheduler.yaml.template <3 apiVersion: kubescheduler.config.k8s.io/v1alpha1 4 kind: KubeSchedulerConfiguration 5 bindTimeoutSeconds: 600 6 clientConnection: 7 burst: 200 8 kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig" 9 qps: 100 10 enableContentionProfiling: false 11 enableProfiling: true 12 hardPodAffinitySymmetricWeight: 1 13 healthzBindAddress: ##MASTER_IP##:10251 14 leaderElection: 15 leaderElect: true 16 metricsBindAddress: ##MASTER_IP##:10251 17 EOF
解释:
--kubeconfig:指定 kubeconfig 文件路径,kube-scheduler 使用它连接和验证 kube-apiserver;
--leader-elect=true:集群运行模式,启用选举功能;被选为 leader 的节点负责处理工作,其它节点为阻塞状态。
1 [root@k8smaster ~]# cd /opt/k8s/work 2 [root@k8smaster work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster work]# for (( i=0; i < 3; i++ )) 4 do 5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-scheduler.yaml.template > kube-scheduler-${MASTER_IPS[i]}.yaml 6 done #替换模板文件中的变量 7 [root@k8smaster01 work]# ls kube-scheduler*.yaml 8 #MASTER_NAMES 和 MASTER_IPS 为相同长度的 bash 数组,分别为节点名称和对应的 IP
1.6 分发配置文件
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 scp kube-scheduler-${master_ip}.yaml root@${master_ip}:/etc/kubernetes/kube-scheduler.yaml 7 done
1.7 创建kube-scheduler的systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# cat > kube-scheduler.service.template <4 [Unit] 5 Description=Kubernetes Scheduler 6 Documentation=https://github.com/GoogleCloudPlatform/kubernetes 7 8 [Service] 9 WorkingDirectory=${K8S_DIR}/kube-scheduler 10 ExecStart=/opt/k8s/bin/kube-scheduler \\ 11 --config=/etc/kubernetes/kube-scheduler.yaml \\ 12 --bind-address=##MASTER_IP## \\ 13 --secure-port=10259 \\ 14 --port=0 \\ 15 --tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\ 16 --tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\ 17 --authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\ 18 --client-ca-file=/etc/kubernetes/cert/ca.pem \\ 19 --requestheader-allowed-names="" \\ 20 --requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\ 21 --requestheader-extra-headers-prefix="X-Remote-Extra-" \\ 22 --requestheader-group-headers=X-Remote-Group \\ 23 --requestheader-username-headers=X-Remote-User \\ 24 --authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\ 25 --logtostderr=true \\ 26 --v=2 27 Restart=always 28 RestartSec=5 29 StartLimitInterval=0 30 31 [Install] 32 WantedBy=multi-user.target 33 EOF
1.8 分发systemd
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for (( i=0; i < 3; i++ )) 4 do 5 sed -e "s/##MASTER_NAME##/${MASTER_NAMES[i]}/" -e "s/##MASTER_IP##/${MASTER_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-${MASTER_IPS[i]}.service 6 done #修正相应IP 7 [root@k8smaster01 work]# ls kube-scheduler*.service 8 [root@k8smaster01 ~]# cd /opt/k8s/work 9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 11 do 12 echo ">>> ${master_ip}" 13 scp kube-scheduler-${master_ip}.service root@${master_ip}:/etc/systemd/system/kube-scheduler.service 14 done #分发system
二 启动并验证
2.1 启动kube-scheduler 服务
1 [root@k8smaster01 ~]# cd /opt/k8s/work 2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 4 do 5 echo ">>> ${master_ip}" 6 ssh root@${master_ip} "mkdir -p ${K8S_DIR}/kube-scheduler" 7 ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler" 8 done #启动服务前必须先创建工作目录
2.2 检查kube-scheduler 服务
1 [root@k8smaster01 ~]# source /opt/k8s/bin/environment.sh 2 [root@k8smaster01 ~]# for master_ip in ${MASTER_IPS[@]} 3 do 4 echo ">>> ${master_ip}" 5 ssh root@${master_ip} "systemctl status kube-scheduler|grep Active" 6 done
2.3 查看输出的 metrics
kube-scheduler 监听 10251 和 10251 端口:
- 10251:接收 http 请求,非安全端口,不需要认证授权;
- 10259:接收 https 请求,安全端口,需要认证授权。
- 两个接口都对外提供 /metrics 和 /healthz 的访问。
1 [root@k8smaster01 ~]# sudo netstat -lnpt |grep kube-sch
1 [root@k8smaster01 ~]# curl -s http://127.0.0.1:10251/metrics |head 2 [root@k8smaster01 ~]# curl -s --cacert /opt/k8s/work/ca.pem --cert /opt/k8s/work/admin.pem --key /opt/k8s/work/admin-key.pem https://172.24.8.71:10259/metrics |head
注意:以上命令在 kube-controller-manager 节点上执行。
2.4 查看当前leader
1 [root@k8smaster01 ~]# kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
当前leader为k8smaster01。