这个 PI 有一个性能测试(Performance test
, 简称 PT
)的任务,尘封了 1 年的用于 PT
的 k8s 集群将要大派用场。
但正当负责 PT
的同事磨刀霍霍,准备使用 kubectl
大展身手的时候,出现了以下错误:
root@xxxxx[03:47][/home/root]$ kubectl get po
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-02-09T03:52:01+01:00 is after 2022-02-03T02:58:17Z
上面的错误提示,证书过期了。于是,编写了本文简单记录一下这次更新证书操作的过程。
k8s
集群使用的 证书
和 私钥(key)
存放在 master
节点的 /etc/kubernetes/pki/
下:
root@xxxxxx[04:01][/home/root]$ ls -alt /etc/kubernetes/pki
total 60
drwxr-xr-x 4 root root 125 Feb 3 2021 ..
drwxr-xr-x 3 root root 4096 Feb 3 2021 .
-rw------- 1 root root 1679 Feb 3 2021 sa.key
-rw------- 1 root root 451 Feb 3 2021 sa.pub
-rw-r--r-- 1 root root 1135 Feb 3 2021 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Feb 3 2021 apiserver-etcd-client.key
drwxr-xr-x 2 root root 162 Feb 3 2021 etcd
-rw-r--r-- 1 root root 1103 Feb 3 2021 front-proxy-client.crt
-rw------- 1 root root 1679 Feb 3 2021 front-proxy-client.key
-rw-r--r-- 1 root root 1078 Feb 3 2021 front-proxy-ca.crt
-rw------- 1 root root 1675 Feb 3 2021 front-proxy-ca.key
-rw-r--r-- 1 root root 1143 Feb 3 2021 apiserver-kubelet-client.crt
-rw------- 1 root root 1679 Feb 3 2021 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1273 Feb 3 2021 apiserver.crt
-rw------- 1 root root 1675 Feb 3 2021 apiserver.key
-rw-r--r-- 1 root root 1066 Feb 3 2021 ca.crt
-rw------- 1 root root 1679 Feb 3 2021 ca.key
上面所列文件中,包含 ca
的属根证书及其对应的 私钥
,是在签发证书时候用,并不在本次更新的证书范围内。例如,apiserver-kubelet-client.crt
的 ca证书 就是 ca.crt
。用 openssl 查看 apiserver-kubelet-client.crt 的 issuer:
root@xxxxx[08:00][/home/root]$ openssl x509 -noout -issuer -subject -in /etc/kubernetes/pki/apiserver-kubelet-client.crt
issuer= /CN=kubernetes
subject= /O=system:masters/CN=kube-apiserver-kubelet-client
从上面,可知,颁发 apiserver-kubelet-client.crt
的 issuer
是 /CN=kubernetes
。再用 openssl
查看 ca.crt
的 subject
和 有效时间
:
root@xxxxx[08:01][/home/root]$ openssl x509 -noout -issuer -subject -enddate -in /etc/kubernetes/pki/ca.crt
issuer= /CN=kubernetes
subject= /CN=kubernetes
notAfter=Feb 1 02:58:17 2031 GMT
ca.crt
的 subject
和 issuer
为 /CN=kubernetes
,与 apiserver-kubelet-client.crt
的 issuer
一致,因此,ca.crt,为根证书,用于签发 apiserver-kubelet-client.crt
。
使用 openssl
检查 apiserver.crt
证书的有效期:
root@xxxxx[04:10][/home/root]$ openssl x509 -enddate -noout -in /etc/kubernetes/pki/apiserver.crt
notAfter=Feb 3 02:58:17 2022 GMT
证书的有效期至 2022 年 2 月 3 日,而当天时间已是 2022 年 2 月 9 日了,已经过期了。
如果 k8s
集群的 kubeadm
版本为 1.8
或更高的版本,则可以使用 kubeadm
进行证书的更新。按下面的步骤进行操作。
在上一节,我们手动检查 /etc/kubernetes/pki
下的证书情况,现在,我们可以通过 kubeadm
方便地检查集群中相关证书的信息:
root@xxxxx[09:25][/home/root]$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Feb 03, 2022 02:58 UTC no
apiserver Feb 03, 2022 02:58 UTC ca no
apiserver-etcd-client Feb 03, 2022 02:58 UTC etcd-ca no
apiserver-kubelet-client Feb 03, 2022 02:58 UTC ca no
controller-manager.conf Feb 03, 2022 02:58 UTC no
etcd-healthcheck-client Feb 03, 2022 02:58 UTC etcd-ca no
etcd-peer Feb 03, 2022 02:58 UTC etcd-ca no
etcd-server Feb 03, 2022 02:58 UTC etcd-ca no
front-proxy-client Feb 03, 2022 02:58 UTC front-proxy-ca no
scheduler.conf Feb 03, 2022 02:58 UTC no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Feb 01, 2031 02:58 UTC 8y no
etcd-ca Feb 01, 2031 02:58 UTC 8y no
front-proxy-ca Feb 01, 2031 02:58 UTC 8y no
我们可以通过 kubeadm certs check-expiration
命令查看集群相关证书情况,其中 CERTIFICATE
部分为证书部分,CERTIFICATE AUTHORITY
为 ca 部分。
可以使用下面命令一键更新所有证书和configuration文件:
kubeadm certs renew all
为了演示,下面将逐个更新证书。
root@xxxxx[09:29][/home/root]$ sudo kubeadm certs renew apiserver
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate for serving the Kubernetes API renewed
root@xxxxx[09:31][/home/root]$ sudo kubeadm certs renew apiserver-etcd-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate the apiserver uses to access etcd renewed
root@xxxxx[09:36][/home/root]$ sudo kubeadm certs renew apiserver-kubelet-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate for the API server to connect to kubelet renewed
root@xxxxx[09:38][/home/root]$ sudo kubeadm certs renew etcd-healthcheck-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate for liveness probes to healthcheck etcd renewed
root@xxxx[09:41][/home/root]$ sudo kubeadm certs renew etcd-peer
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate for etcd nodes to communicate with each other renewed
root@xxxxx[09:42][/home/root]$ sudo kubeadm certs renew etcd-server
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate for serving etcd renewed
root@xxxxx[09:43][/home/root]$ sudo kubeadm certs renew front-proxy-client
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate for the front proxy client renewed
在 /etc/kubernetes
下,有几个 conf
文件:
root@xxx[09:54][/home/root]$ sudo ls -alt /etc/kubernetes | grep -E .*conf$
-rw------- 1 root root 1942 Feb 3 2021 kubelet.conf
-rw------- 1 root root 5550 Feb 3 2021 scheduler.conf
-rw------- 1 root root 5602 Feb 3 2021 controller-manager.conf
-rw------- 1 root root 5570 Feb 3 2021 admin.conf
其实这些 conf
文件就是 k8s 集群各核心组件所用的 kubeconfig
文件:
admin
用户使用的 kubeconfig
。
更新 admin.conf
root@xxxxx[09:57][/home/root]$ sudo kubeadm certs renew admin.conf
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
controller manager
使用的 kubeconfig
。
更新 controller-manager.conf
root@xxxxx[10:21][/home/root]$ sudo kubeadm certs renew controller-manager.conf
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
certificate embedded in the kubeconfig file for the controller manager to use renewed
scheduler manager
使用的 kubeconfig
。
更新 scheduler.conf
root@xxxxx[10:22][/home/root]$ sudo kubeadm certs renew scheduler.conf
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
至此,为 k8s 集群更新证书的操作已经完成了。
再检查集群相关证书情况
root@xxxxx[10:37][/home/root]$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Feb 09, 2023 09:21 UTC 364d no
apiserver Feb 09, 2023 08:29 UTC 364d ca no
apiserver-etcd-client Feb 09, 2023 08:34 UTC 364d etcd-ca no
apiserver-kubelet-client Feb 09, 2023 08:38 UTC 364d ca no
controller-manager.conf Feb 09, 2023 09:22 UTC 364d no
etcd-healthcheck-client Feb 09, 2023 08:41 UTC 364d etcd-ca no
etcd-peer Feb 09, 2023 08:42 UTC 364d etcd-ca no
etcd-server Feb 09, 2023 08:43 UTC 364d etcd-ca no
front-proxy-client Feb 09, 2023 08:43 UTC 364d front-proxy-ca no
scheduler.conf Feb 09, 2023 09:24 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Feb 01, 2031 02:58 UTC 8y no
etcd-ca Feb 01, 2031 02:58 UTC 8y no
front-proxy-ca Feb 01, 2031 02:58 UTC 8y no
相关证书的有效期已经被设置成 2023 年 2 月 9 了。
再执行 kubectl get nodes:
root@xxxxx[10:46][/tmp]$ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
NAME STATUS ROLES AGE VERSION
node-1 Ready 371d v1.20.2
master Ready control-plane,master 371d v1.20.2
node-2 Ready 349d v1.20.4
node-3 Ready 371d v1.20.4