持续集成系列----k8s证书过期问题处理（一年后踩坑）

背景

突然收到Prometheus的报警，提示k8s证书离过期只差七天，wtf？！黑人问号脸，查了下资料发现kubeadm安装的集群，默认的自签名证书有效期只有一年。。。要是没有告警，证书到期，集群停摆，业务崩盘，是不是直接可以走人了

解决方案

查看证书到期情况

find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'

备份原证书，排除ca与sa证书

find /etc/kubernetes/pki/ -regex '.*.[crt|key]'|grep -v sa|grep -v ca|xargs -i mv {}{,bak}

导出kubeadm配置

kubeadm config view > cluster.yaml

重新生成master证书

kubeadm alpha phase certs etcd-healthcheck-client --config cluster.yaml
kubeadm alpha phase certs etcd-peer --config cluster.yaml
kubeadm alpha phase certs etcd-server --config cluster.yaml
kubeadm alpha phase certs front-proxy-client--config cluster.yaml
kubeadm alpha phase certs apiserver-etcd-client --config cluster.yaml
kubeadm alpha phase certs apiserver-kubelet-client --config cluster.yaml
kubeadm alpha phase certs apiserver --config cluster.yaml

拷贝到其它台master

for i in k8s1 k8s2;do ssh $i hostname;find /etc/kubernetes/pki/ -regex '.*.[crt|key]'|grep -v sa|grep -v ca|xargs -i scp {} ${i}:{};done

备份集群配置文件并重新生成

find /etc/kubernetes/ -name '*.conf'|xargs -i mv {}{,bak}
kubeadm alpha phase kubeconfig all --config cluster.yaml
\cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
KUBECONFIG=$HOME/.kube/config

拷贝到其它master

for i in k8s1 k8s2;do ssh $i hostname;find /etc/kubernetes/ -name '*.conf'|xargs -i scp {} ${i}:{};done

覆盖旧kubectl配置

\cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
KUBECONFIG=$HOME/.kube/config

重启服务

docker ps |grep kube-controller-manager|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-scheduler|grep -v pause|awk '{print $1}'|xargs -i docker restart {}
docker ps |grep kube-apiserver|grep -v pause|awk '{print $1}'|xargs -i docker restart {}

参考文档
http://blog.chinaunix.net/uid-28813320-id-5823201.html
https://www.cnblogs.com/aguncn/p/10488130.html