kubernetes证书时间修改

此文档针对k8s版本1.14.1（kubeadm安装k8s三主多从）

kubeadm 默认证书为一年，一年过期后，会导致api service不可用，使用过程中会出现：x509: certificate has expired or is not yet valid.

方案一 通过修改kubeadm 调整证书过期时间

1、创建工作目录

[root@master ~]# mkdir /data 

2、下载go环境(搜索go 中文网既可下载)

[root@master ~]# cd /data 
[root@master data]#  wget  https://studygolang.com/dl/golang/go1.13.5.linux-amd64.tar.gz  
[root@master data]# tar -xf go1.13.5.linux-amd64.tar.gz -C /usr/local/

3、设置环境变量

[root@master data]# vim /etc/profile
export PATH=$PATH:/usr/local/go/bin     //配置文件中加入此行

[root@master data]# source /etc/profile /生效

4、下载kubernetes源码(提前安装git)

[root@master data]#   git clone https://github.com/kubernetes/kubernetes.git

5、切换当前集群的版本

[root@master data]#  git checkout -b remotes/origin/release-1.14.1 v1.14.1
注意：如果切换报错，请执行：[root@master data]# git reset --hard  //放弃本地修改，直接覆盖

6、修改kubeadm源码包更新策略

6.1、进入到下载的源码包目录下

1.14版本之前修改此目录内容

[root@master kubernetes]# vim staging/src/k8s.io/client-go/util/cert/cert.go  

6.2、1.14版本至今修改此目录下内容

[root@master kubernetes]# vim cmd/kubeadm/app/util/pkiutil/pki_helpers.go 

6.3、添加内容如下（注意前面空格处是tab键）

6.4、开始编译kubeadm

[root@master kubernetes]# make WHAT=cmd/kubeadm GOFLAGS=-v   (如果编译失败 请检查配置文件)
[root@master kubernetes]# cp _output/bin/kubeadm  /root      (将编译出的文件拷贝到root目录下)
[root@master kubernetes]# cp /usr/bin/kubeadm /usr/bin/kubeadm.back（备份之前的文件以免失败）
[root@master kubernetes]# cp /root/kubeadm  /usr/bin/  （将编译出来的kubeadm文件拷贝到/usr/bin下）
[root@master kubernetes]# chmod a+x /usr/bin/kubeadm  (添加执行权限)
[root@master kubernetes]# cp -a /etc/kubernetes/pki   /etc/kubernetes/pki.back  （将pki目录备份）

6.5、生成新的kube-config.yaml配置文件

[root@master ~]# kubeadm config view > kube-config.yaml

6.6、生成新的证书，kube-config.yaml指定你刚生成的文件

[root@master ~]# kubeadm alpha certs renew all --config=kube-config.yaml

6.7、备份原有的conf文件

[root@master ~]# mkdir /etc/kubernetes/old-conf
[root@master ~]# mv /etc/kubernetes/*conf /etc/kubernetes/old-conf

6.8、根据新证书重新生成新的配置文件

[root@master ~]# kubeadm init phase kubeconfig all --config=kube-config.yaml

6.9、替换老的config文件

[root@master ~]# cp -f /etc/kubernetes/admin.conf ~/.kube/config

6.10、将master1上编译出的kubeadm拷贝到master2和master3(先在2和3备份一下)

[root@master kubernetes]# scp /root/kubeadm master2:/usr/bin
[root@master kubernetes]# scp /root/kubeadm master3:/usr/bin
[root@master kubernetes]# chmod a+x /usr/bin/kubeadm

6.11、在master2和master3分别执行以下操作

[root@master ~]# kubeadm alpha certs renew all --config=kube-config.yaml
[root@master ~]# kubeadm init phase kubeconfig all --config=kube-config.yaml
[root@master ~]# cp -f /etc/kubernetes/admin.conf ~/.kube/config

7、在三台master上重启kube-apiserver，kube-controller，kube-scheduler，etcd这四个容器，并重启kubelet，使其生效

[root@master ~]# docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart
或者过滤出每个容器的id，然后在重启
[root@master ~]# docker ps |grep kube
[root@master ~]# docker restart (容器id)

重启kubelet（node也重启下）
[root@master pki]# systemctl restart kubelet

8、验证

[root@master pki]# cd /etc/kubernetes/pki
[root@master pki]# for i in $(ls *.crt); do echo "===== $i ====="; openssl x509 -in $i -text -noout | grep -A 3 'Validity' ; done
===== apiserver.crt =====
        Validity
            Not Before: Sep 21 07:39:37 2020 GMT
            Not After : Sep 19 07:54:38 2030 GMT
        Subject: CN=kube-apiserver
===== apiserver-etcd-client.crt =====
        Validity
            Not Before: Sep 21 07:39:39 2020 GMT
            Not After : Sep 19 07:54:38 2030 GMT
        Subject: O=system:masters, CN=kube-apiserver-etcd-client
===== apiserver-kubelet-client.crt =====
        Validity
            Not Before: Sep 21 07:39:37 2020 GMT
            Not After : Sep 19 07:54:38 2030 GMT
        Subject: O=system:masters, CN=kube-apiserver-kubelet-client
===== ca.crt =====
        Validity
            Not Before: Sep 21 07:39:37 2020 GMT
            Not After : Sep 19 07:39:37 2030 GMT
        Subject: CN=kubernetes
===== front-proxy-ca.crt =====
        Validity
            Not Before: Sep 21 07:39:38 2020 GMT
            Not After : Sep 19 07:39:38 2030 GMT
        Subject: CN=front-proxy-ca
===== front-proxy-client.crt =====
        Validity
            Not Before: Sep 21 07:39:38 2020 GMT
            Not After : Sep 19 07:54:37 2030 GMT
        Subject: CN=front-proxy-client

9、批量验证证书

[root@master pki]# cd /etc/kubernetes/pki
[root@master pki]# for crt in $(find /etc/kubernetes/pki/ -name "*.crt"); do openssl x509 -in $crt -noout -dates; done

方案二 启用自动轮换kubelet 证书

kubelet证书分为server和client两种， k8s 1.9默认启用了client证书的自动轮换，但server证书自动轮换需要用户开启

1、增加 kubelet 参数

vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Environment="KUBELET_EXTRA_ARGS=--feature-gates=RotateKubeletServerCertificate=true"

2、增加 controller-manager 参数

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

- command:
    - kube-controller-manager
    - --experimental-cluster-signing-duration=87600h0m0s
    - --feature-gates=RotateKubeletServerCertificate=true
    - ....

3、创建 rbac 对象 创建rbac对象，允许节点轮换kubelet server证书

cat > ca-update.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/selfnodeserver
  verbs:
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-autoapprove-certificate-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:nodes
EOF

kubectl create –f ca-update.yaml

方案三

如果证书已经过期，如何进行重新签发证书 针对kubeadm 1.13.x 及以上处理 准备kubeadm.conf 配置文件一份

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.1 #-->这里改成你集群对应的版本
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers 
#这里使用国内的镜像仓库，否则在重新签发的时候会报错：could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt"

1、重新签发命令

kubeadm alpha certs renew all --config=/root/kubeadm.conf

运行如上命令会重新生成以下证书
#-- /etc/kubernetes/pki/apiserver.key
#-- /etc/kubernetes/pki/apiserver.crt

#-- /etc/kubernetes/pki/apiserver-etcd-client.key
#-- /etc/kubernetes/pki/apiserver-etcd-client.crt

#-- /etc/kubernetes/pki/apiserver-kubelet-client.key
#-- /etc/kubernetes/pki/apiserver-kubelet-client.crt

#-- /etc/kubernetes/pki/front-proxy-client.key
#-- /etc/kubernetes/pki/front-proxy-client.crt

#-- /etc/kubernetes/pki/etcd/healthcheck-client.key
#-- /etc/kubernetes/pki/etcd/healthcheck-client.crt

#-- /etc/kubernetes/pki/etcd/peer.key
#-- /etc/kubernetes/pki/etcd/peer.crt

#-- /etc/kubernetes/pki/etcd/server.key
#-- /etc/kubernetes/pki/etcd/server.crt

2、更新/etc/kubernetes/*.conf文件

3、备份删除旧的/etc/kubernetes/*.conf文件

mkdir /etc/kubernetes/old-conf  
mv /etc/kubernetes/*.conf /etc/kubernetes/old-conf

4、生成新的conf文件

kubeadm init phase kubeconfig all --config=/root/kubeadm.conf

#运行如上命令会重新生成以下conf文件
#-- /etc/kubernetes/admin.conf
#-- /etc/kubernetes/controller-manager.conf
#-- /etc/kubernetes/kubelet.conf
#-- /etc/kubernetes/scheduler.conf

5、完成后重启kube-apiserver,kube-controller,kube-scheduler,etcd这4个容器,最后覆盖config文件

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
docker ps | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | awk '{print $1}' | awk '{print "docker","restart",$1}' | bash

6、针对kubeadm 1.13.0（不包含1.13.0） 以下处理 移动证书和配置【注意！必须移动，不然会使用现有的证书，不会重新生成】

cd /etc/kubernetes
mkdir ./pki_bak
mkdir ./pki_bak/etcd
mkdir ./conf_bak
mv pki/apiserver* ./pki_bak/
mv pki/front-proxy-client.* ./pki_bak/
mv pki/etcd/healthcheck-client.* ./pki_bak/etcd/
mv pki/etcd/peer.* ./pki_bak/etcd/
mv pki/etcd/server.* ./pki_bak/etcd/
mv ./admin.conf ./conf_bak/
mv ./kubelet.conf ./conf_bak/
mv ./controller-manager.conf ./conf_bak/
mv ./scheduler.conf ./conf_bak/

7、创建证书

kubeadm alpha phase certs all --apiserver-advertise-address=${MASTER_API_SERVER_IP} --apiserver-cert-extra-sans=主机内网ip,主机公网ip

运行如上命令会重新生成以下证书
#-- /etc/kubernetes/pki/apiserver.key
#-- /etc/kubernetes/pki/apiserver.crt

#-- /etc/kubernetes/pki/apiserver-etcd-client.key
#-- /etc/kubernetes/pki/apiserver-etcd-client.crt

#-- /etc/kubernetes/pki/apiserver-kubelet-client.key
#-- /etc/kubernetes/pki/apiserver-kubelet-client.crt

#-- /etc/kubernetes/pki/front-proxy-client.key
#-- /etc/kubernetes/pki/front-proxy-client.crt

#-- /etc/kubernetes/pki/etcd/healthcheck-client.key
#-- /etc/kubernetes/pki/etcd/healthcheck-client.crt

#-- /etc/kubernetes/pki/etcd/peer.key
#-- /etc/kubernetes/pki/etcd/peer.crt

#-- /etc/kubernetes/pki/etcd/server.key
#-- /etc/kubernetes/pki/etcd/server.crt

不移动证书会有如下提示
#[certificates] Using the existing apiserver certificate and key.
#[certificates] Using the existing apiserver-kubelet-client certificate and key.
#[certificates] Using the existing front-proxy-client certificate and key.
#[certificates] Using the existing etcd/server certificate and key.
#[certificates] Using the existing etcd/peer certificate and key.
#[certificates] Using the existing etcd/healthcheck-client certificate and key.
#[certificates] Using the existing apiserver-etcd-client certificate and key.
#[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
#[certificates] Using the existing sa key.

8、生成新配置文件

kubeadm alpha phase kubeconfig all --apiserver-advertise-address=${MASTER_API_SERVER_IP}

9、将新生成的admin配置文件覆盖掉原本的admin文件

mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
sudo chmod 644 $HOME/.kube/config

10、完成后重启kube-apiserver,kube-controller,kube-scheduler,etcd这4个容器