k8s证书过期---手动更新

背景：

执行命令发现报错：
Unable to connect to the server: x509: certificate has expired or is not yet valid

这就是k8s的证书过期了
k8s解决证书过期官方文档：https://kubernetes.io/zh/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
查看是k8s master 节点证书过期了，登录master服务器，进入 /etc/kubernetes/ 查看：

[root@k8s-145246 ~]# cd /etc/kubernetes
[root@k8s-145246 kubernetes]# ll
total 40
-rw------- 1 root root 5455 Mar 16  2021 admin.conf
-rw------- 1 root root 5491 Mar 16  2021 controller-manager.conf
-rw------- 1 root root 1879 Mar 16  2021 kubelet.conf
drwxr-xr-x 2 root root 4096 Jul 27  2021 manifests
drwxr-xr-x 5 root root 4096 Apr  2  2021 pki
-rw------- 1 root root 5435 Mar 16  2021 scheduler.conf
drwxr-xr-x 3 root root 4096 Mar 18  2021 volumes
[root@k8s-145246 kubernetes]# cd pki/
[root@k8s-145246 pki]# ll
total 76
-rw-r--r-- 1 root root 1224 Mar 16  2021 apiserver.crt
-rw-r--r-- 1 root root 1090 Mar 16  2021 apiserver-etcd-client.crt
-rw------- 1 root root 1679 Mar 16  2021 apiserver-etcd-client.key
-rw------- 1 root root 1679 Mar 16  2021 apiserver.key
-rw-r--r-- 1 root root 1099 Mar 16  2021 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Mar 16  2021 apiserver-kubelet-client.key
-rw-r--r-- 1 root root  162 Apr  1  2021 basic_auth_file
-rw-r--r-- 1 root root   32 Apr  1  2021 basic_auth_file_ops
-rw-r--r-- 1 root root 1025 Mar 16  2021 ca.crt
-rw------- 1 root root 1675 Mar 16  2021 ca.key
drwxr-xr-x 2 root root 4096 Mar 16  2021 etcd
-rw-r--r-- 1 root root 1038 Mar 16  2021 front-proxy-ca.crt
-rw------- 1 root root 1679 Mar 16  2021 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 Mar 16  2021 front-proxy-client.crt
-rw------- 1 root root 1679 Mar 16  2021 front-proxy-client.key
-rw------- 1 root root 1679 Mar 16  2021 sa.key
-rw------- 1 root root  451 Mar 16  2021 sa.pub
drwxr-xr-x 2 root root 4096 Apr  2  2021 ssl
drwxr-xr-x 2 root root 4096 Apr  2  2021 ssl_self

查看是否过期

[root@k8s-145246 pki]# openssl x509 -in apiserver.crt -noout -text |grep ' Not '
            Not Before: Mar 16 05:58:49 2021 GMT
            Not After : Mar 16 05:58:49 2022 GMT

检查k8s环境证书是否过期

[root@k8s-145246 pki]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

W0316 14:21:38.307724   49056 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Mar 16, 2022 05:58 UTC                                  no      
apiserver                  Mar 16, 2022 05:58 UTC          ca                      no      
apiserver-etcd-client      Mar 16, 2022 05:58 UTC          etcd-ca                 no      
apiserver-kubelet-client   Mar 16, 2022 05:58 UTC          ca                      no      
controller-manager.conf    Mar 16, 2022 05:58 UTC                                  no      
etcd-healthcheck-client    Mar 16, 2022 05:58 UTC          etcd-ca                 no      
etcd-peer                  Mar 16, 2022 05:58 UTC          etcd-ca                 no      
etcd-server                Mar 16, 2022 05:58 UTC          etcd-ca                 no      
front-proxy-client         Mar 16, 2022 05:58 UTC          front-proxy-ca          no      
scheduler.conf             Mar 16, 2022 05:58 UTC                                  no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 14, 2031 05:58 UTC   8y              no      
etcd-ca                 Mar 14, 2031 05:58 UTC   8y              no      
front-proxy-ca          Mar 14, 2031 05:58 UTC   8y              no      

经查看k8s master 组件 证书都过期了，有效期是一年

解决问题：

以下操作均在master节点

1、备份一下 /etc /kubernetes /pki 目录下的所有文件。

[root@k8s-145246 pki]# cd ..
[root@k8s-145246 kubernetes]# ll
total 40
-rw------- 1 root root 5455 Mar 16  2021 admin.conf
-rw------- 1 root root 5491 Mar 16  2021 controller-manager.conf
-rw------- 1 root root 1879 Mar 16  2021 kubelet.conf
drwxr-xr-x 2 root root 4096 Jul 27  2021 manifests
drwxr-xr-x 5 root root 4096 Apr  2  2021 pki
-rw------- 1 root root 5435 Mar 16  2021 scheduler.conf
drwxr-xr-x 3 root root 4096 Mar 18  2021 volumes
[root@k8s-145246 kubernetes]# cp -r pki pki.bak20220316

2、手动更新所有证书，执行命令

[root@k8s-145246 kubernetes]# cd pki
[root@k8s-145246 pki]# kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

W0316 14:22:47.549844    1406 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

3、查看证书有效期是否更新

[root@k8s-145246 pki]# openssl x509 -in apiserver.crt -noout -text |grep ' Not '
            Not Before: Mar 16 05:58:49 2021 GMT
            Not After : Mar 16 06:22:48 2023 GMT

[root@k8s-145246 pki]# kubectl get nodes
NAME         STATUS   ROLES         AGE    VERSION
k8s-145103   Ready    edge          365d   v1.18.16
k8s-145104   Ready            365d   v1.18.16
k8s-145246   Ready    edge,master   365d   v1.18.16

此时命令可用了
但是还没有结束，需要进行下一步，不然k8s组件会报错：

 authentication.go:65] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

导致不能重新创建pod
4、在master节点上将/etc/kubernetes目录下的所有配置文件备份

[root@k8s-145246 etc]# cp -r /etc/kubernetes /etc/kubernetes.bak

5、更新用户配置：执行下面多个命令

kubeadm alpha kubeconfig user --client-name=admin
kubeadm alpha kubeconfig user --org system:masters --client-name kubernetes-admin  > /etc/kubernetes/admin.conf
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > /etc/kubernetes/kubelet.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf

6、用更新后的admin.conf替换/root/.kube/config文件

cp -i /etc/kubernetes/admin.conf /root/.kube/config

将/etc/kubernetes/admin.conf 分发到其他node节点后，node节点就可以使用kubectl命令了
7、重启所有master节点上的apiserver和scheduler两个系统组件
正常tar包部署的k8s可以使用下面的命令重启：

systemctl restart kube-apiserver
systemctl restart kube-scheduler

但是kubeadm部署的方式需要重启相关pod后再重启对应的docker

[root@k8s-145246 pki]# kubectl get po -n kube-system|grep k8s
etcd-k8s-145246                           1/1     Running             17         17h
kube-apiserver-k8s-145246                 1/1     Running             20         18h
kube-controller-manager-k8s-145246        1/1     Running             19         17h
kube-scheduler-k8s-145246                 1/1     Running             18         18h

docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart

此时证书更新全部完成，pod恢复正常

附上一个大神写的自动更新10年证书的文章参考：
https://www.qikqiak.com/post/update-k8s-10y-expire-certs/