kubernetes刷新证书

Note: 本文基于使用kubeadm搭建的kubernetes集群进行讲解


第一步:获取证书签发信息

  • 方式一:通过原有证书进行获取相关信息

    • 获取apiserver签发信息

      $ openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt
      ......
      X509v3 extensions:
          X509v3 Key Usage: critical
              Digital Signature, Key Encipherment
          X509v3 Extended Key Usage:
              TLS Web Server Authentication
          X509v3 Subject Alternative Name:
              DNS:localhost, DNS:node1, DNS:node2, DNS:node3, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc,   DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:10.96.0.1, IP Address:192.168.12.10, IP   Address:192.168.12.11, IP Address:192.168.12.12, IP Address:192.168.12.13
      ......
      
    • 从上面输出信息可知签发的DNS和IP详情

      DNS.1 = localhost
      # 各 master 节点 nodeName
      DNS.2 = node1
      DNS.3 = node2
      DNS.4 = node3
      # kubenertes svc 地址
      DNS.5 = kubernetes
      DNS.6 = kubernetes.default
      DNS.7 = kubernetes.default.svc
      # 带 dns domain 的 kubenertes svc 地址
      DNS.8 = kubernetes.default.svc.cluster.local
      IP.6 = 127.0.0.1
      # kubenertes svc 的 clusterip
      IP.5 = 10.96.0.1
      # api-server VIP 地址
      IP.4 = 192.168.12.10
      # 各 master 节点 IP
      IP.1 = 192.168.12.11
      IP.2 = 192.168.12.12
      IP.3 = 192.168.12.13
      
  • 方式二:自行统计各项信息

    • 在创建证书之前需要获取到以下信息,在签发证书的时候会用到它:

      • 各master节点IP
      • 各master节点nodeName
      • 如果有设置apiserver负载均衡则需要VIP,否则请忽略
      • kubernetes集群dns domain
      • kubenertes.default.svc的clusterip
    • 本文以下面集群信息为例:

      • 节点信息:

        nodeName ip relo
        slb 192.168.12.10 slb
        node1 192.168.12.11 master
        node2 192.168.12.12 master
        node3 192.168.12.13 master
        node4 192.168.12.14 worker
        node5 192.168.12.15 worker
      • 获取kubernetes dns domain:

        # CoreDNS 查看方法
        $ kubectl get cm -n kube-system coredns -o yaml | grep kubernetes
        kubernetes cluster.local in-addr.arpa ip6.arpa {
        # 由以上输出可知dns domain为 cluster.local
        
        # kube-dns 查看方法
        $ kubectl get deployment -n kube-system kube-dns -o yaml | grep domain
        - --domain=cluster.local.
        # 由以上输出可知dns domain为 cluster.local
        
      • kubernetes apiserver clusterip

        $ kubectl get svc kubernetes -n default
        NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
        kubernetes   ClusterIP   10.96.0.1              443/TCP   280d
        # 由结果可知apiserver clusterip
        

同理获取etcd集群证书签发的DNS和IP详情,默认etcd证书路径为/etc/kubernetes/pki/etcd

第二步:创建证书

Note: 我们只需要在一个节点上进行证书生成,生成的证书分发到其他节点即可


  • 创建CA服务端证书签名请求配置文件openssl.conf,内容如下,注意替换alt_names_clusteralt_names_etcd域中的值

    [ req ]
    default_bits = 2048
    default_md = sha256
    distinguished_name = req_distinguished_name
    
    [req_distinguished_name]
    
    [ v3_ca ]
    basicConstraints = critical, CA:TRUE
    keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign
    
    [ v3_req_server ]
    basicConstraints = CA:FALSE
    keyUsage = critical, digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth
    
    [ v3_req_client ]
    basicConstraints = CA:FALSE
    keyUsage = critical, digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth
    
    [ v3_req_apiserver ]
    basicConstraints = CA:FALSE
    keyUsage = critical, digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names_cluster
    
    [ v3_req_etcd ]
    basicConstraints = CA:FALSE
    keyUsage = critical, digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth, clientAuth
    subjectAltName = @alt_names_etcd
    
    [ alt_names_cluster ]
    DNS.1 = localhost
    DNS.2 = node1
    DNS.3 = node2
    DNS.4 = node3
    DNS.5 = kubernetes
    DNS.6 = kubernetes.default
    DNS.7 = kubernetes.default.svc
    DNS.8 = kubernetes.default.svc.cluster.local
    IP.1 = 127.0.0.1
    IP.2 = 10.96.0.1
    IP.3 = 192.168.12.10
    IP.4 = 192.168.12.11
    IP.5 = 192.168.12.12
    IP.6 = 192.168.12.13
    
    [ alt_names_etcd ]
    DNS.1 = localhost
    DNS.2 = node1
    DNS.3 = node2
    DNS.4 = node3
    IP.1 = 127.0.0.1
    IP.2 = 0:0:0:0:0:0:0:1
    IP.3 = 192.168.12.11
    IP.4 = 192.168.12.12
    IP.5 = 192.168.12.13
    
  • 创建集群 key 与 CA

    • 将要创建的 CA

      路径 Common Name 描述
      ca.crt,key kubernetes Kubernetes general CA
      etcd/ca.crt,key kubernetes For all etcd-related functions
      front-proxy-ca.crt,key kubernetes For the front-end proxy
    • 要注意 CA 中 CN(Common Name) 与 O(Organization) 等内容是会影响Kubernetes组件认证的

      • CA (Certificate Authority) 是自签名的根证书,用来签名后续创建的其它证书
      • CN (Common Name), apiserver 会从证书中提取该字段作为请求的用户名 (User Name)
      • O (Organization), apiserver 会从证书中提取该字段作为请求用户所属的组 (Group)

    一般CA根证书有效期为10年,若举例过期时间还长,可跳过本节操作,命令中的3650为3650天,即证书有效期。

    • kubernetes-ca

      openssl genrsa -out ca.key 2048
      openssl req -x509 -new -nodes -key ca.key \
        -subj "/CN=kubernetes" -config openssl.conf \
        -extensions v3_ca -out ca.crt -days 3560
      
    • etcd-ca

      openssl genrsa -out etcd/ca.key 2048
      openssl req -x509 -new -nodes -key etcd/ca.key \
        -subj "/CN=kubernetes" -config openssl.conf \
        -extensions v3_ca -out etcd/ca.crt -days 3560
      
    • front-proxy-ca

      openssl genrsa -out front-proxy-ca.key 2048
      openssl req -x509 -new -nodes -key front-proxy-ca.key \
        -subj "/CN=kubernetes" -config openssl.conf \
        -extensions v3_ca -out front-proxy-ca.crt -days 3560
      
  • 创建 Certificates

    • 将要创建的 Certificates

      Name Key Certificates Common Name Organization
      etcd/server etcd/server.key etcd/server.crt master
      etcd/peer etcd/peer.key etcd/peer.crt master
      etcd/healthcheck-client etcd/healthcheck-client.key etcd/healthcheck-client.crt kube-etcd-healthcheck-client system:masters
      apiserver-etcd-client apiserver-etcd-client.key apiserver-etcd-client.crt kube-apiserver-etcd-client system:masters
      apiserver apiserver.key apiserver.crt kube-apiserver
      apiserver-kubelet-client apiserver-kubelet-client.key apiserver-kubelet-client.crt kube-apiserver-kubelet-client system:masters
      front-proxy-client front-proxy-client.key front-proxy-client.crt front-proxy-client
      kube-scheduler kube-scheduler.key kube-scheduler.crt system:kube-scheduler
      sa(kube-controller-manager) sa.key(sa.pub) kube-controller-manager.crt system:kube-controller-manager
      admin(kubectl) admin.key admin.crt kubernetes-admin system:masters
      kubelet kubelet.key kubelet.crt system:node:master system:nodes
    • etcd/server

      openssl genrsa -out etcd/server.key 2048
      openssl req -new -key etcd/server.key \
        -subj "/CN=master" -out etcd/server.csr
      openssl x509 -in etcd/server.csr -req -CA etcd/ca.crt \
        -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
        -extfile openssl.conf -out etcd/server.crt -days 3560
      
    • etcd/peer

      openssl genrsa -out etcd/peer.key 2048
      openssl req -new -key etcd/peer.key \
        -subj "/CN=master" -out etcd/peer.csr
      openssl x509 -in etcd/peer.csr -req -CA etcd/ca.crt \
        -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
        -extfile openssl.conf -out etcd/peer.crt -days 3560
      
    • etcd/healthcheck-client

      openssl genrsa -out etcd/healthcheck-client.key 2048
      openssl req -new -key etcd/healthcheck-client.key \
        -subj "/CN=kube-etcd-healthcheck-client/O=system:masters" \
        -out etcd/healthcheck-client.csr
      openssl x509 -in etcd/healthcheck-client.csr -req -CA etcd/ca.crt \
        -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
        -extfile openssl.conf -out etcd/healthcheck-client.crt -days 3560
      
    • apiserver-etcd-client

      openssl genrsa -out apiserver-etcd-client.key 2048
      openssl req -new -key apiserver-etcd-client.key \
        -subj "/CN=kube-apiserver-etcd-client/O=system:masters" 
        -out apiserver-etcd-client.csr
      openssl x509 -in apiserver-etcd-client.csr -req -CA etcd/ca.crt \
        -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \
        -extfile openssl.conf -out apiserver-etcd-client.crt -days 3560
      
    • apiserver

      openssl genrsa -out apiserver.key 2048
      openssl req -new -key apiserver.key \
        -subj "/CN=kube-apiserver" -config openssl.conf \
        -out apiserver.csr
      openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key \
        -CAcreateserial -extensions v3_req_apiserver \
        -extfile openssl.conf -out apiserver.crt -days 3560
      
    • apiserver-kubelet-client

      openssl genrsa -out apiserver-kubelet-client.key 2048
      openssl req -new -key apiserver-kubelet-client.key \
        -subj "/CN=kube-apiserver-kubelet-client/O=system:masters" \
        -out apiserver-kubelet-client.csr
      openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key \
        -CAcreateserial -extensions v3_req_client \
        -extfile openssl.conf -out apiserver-kubelet-client.crt -days 3560
      
    • front-proxy-client

      openssl genrsa -out front-proxy-client.key 2048
      openssl req -new -key front-proxy-client.key \
        -subj "/CN=front-proxy-client" \
        -out front-proxy-client.csr
      openssl x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key \
        -CAcreateserial -extensions v3_req_client \
        -extfile openssl.conf -out front-proxy-client.crt -days 3560
      
    • kube-scheduler

      openssl genrsa -out kube-scheduler.key 2048
      openssl req -new -key kube-scheduler.key \
        -subj "/CN=system:kube-scheduler" \
        -out kube-scheduler.csr
      openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key \
        -CAcreateserial -extensions v3_req_client \
        -extfile openssl.conf -out kube-scheduler.crt -days 3560
      
    • sa(kube-controller-manager)

      openssl genrsa -out sa.key 2048
      openssl rsa -in sa.key -pubout -out sa.pub
      openssl req -new -key sa.key \
        -subj "/CN=system:kube-controller-manager" \
        -out kube-controller-manager.csr
      openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key \
        -CAcreateserial -extensions v3_req_client \
        -extfile openssl.conf -out kube-controller-manager.crt -days 3560
      
    • admin(kubectl)

      openssl genrsa -out admin.key 2048
      openssl req -new -key admin.key \
        -subj "/CN=kubernetes-admin/O=system:masters" \
        -out admin.csr
      openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key \
        -CAcreateserial -extensions v3_req_client \
        -extfile openssl.conf -out admin.crt -days 3560 
      
    • kubelet

      openssl genrsa -out kubelet.key 2048
      openssl req -new -key kubelet.key \
        -subj "/CN=system:node:master/O=system:nodes" \
        -out kubelet.csr
      openssl x509 -req -in kubelet.csr -CA ca.crt -CAkey ca.key \
        -CAcreateserial -extensions v3_req_client \
        -extfile openssl.conf -out kubelet.crt -days 3560 
      

第三步:生成kubernetes各组件配置文件并应用

  • 所要生成的配置文件列表

    配置文件名称 组件证书文件名称 组件秘钥文件名称 根证书文件名称
    admin.conf(kubectl) admin.crt admin.key ca.crt
    kubelet.conf kubelet.crt kubelet.key ca.crt
    scheduler.conf kube-scheduler.crt kube-scheduler.key ca.crt
    controller-manager.conf kube-controller-manager.crt sa.key ca.crt
    • 操作前请先备份原有配置文件
    • 除了kubelet.conf文件需注意配置为对应节点的nodeName,其余配置文件可通用
    • 以下操作请先在一台 master 节点上操作确认没有问题后再进行配置其他节点
    • –certificate-authority:指定根证书
    • –client-certificate、–client-key:指定组件证书及秘钥
    • –embed-certs=true:将组件证书内容嵌入到生成的配置文件中(不加时,写入的是证书文件路径)
  • admin.conf(kubectl)

    KUBE_APISERVER="https://192.168.12.10:6443"
    CLUSTER_NAME="kubernetes"
    KUBE_USER="kubernetes-admin"
    KUBE_CERT="admin"
    KUBE_CONFIG="admin.conf"
    
    # 设置集群参数
    kubectl config set-cluster ${CLUSTER_NAME} \
      --certificate-authority=/etc/kubernetes/pki/ca.crt \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置客户端认证参数
    kubectl config set-credentials ${KUBE_USER} \
      --client-certificate=/etc/kubernetes/pki/${KUBE_CERT}.crt \
      --client-key=/etc/kubernetes/pki/${KUBE_CERT}.key \
      --embed-certs=true \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置上下文参数
    kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
      --cluster=${CLUSTER_NAME} \
      --user=${KUBE_USER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置当前使用的上下文
    kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 查看生成的配置文件
    kubectl config view --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
  • kubelet.conf(注意配置对应的nodeName)

    KUBE_APISERVER="https://192.168.12.10:6443"
    CLUSTER_NAME="kubernetes"
    # 此处为 master 节点 nodeName,每个 master 生成对应的 kubelet.conf
    KUBE_USER="system:node1:master"
    KUBE_CERT="kubelet"
    KUBE_CONFIG="kubelet.conf"
    
    # 设置集群参数
    kubectl config set-cluster ${CLUSTER_NAME} \
      --certificate-authority=/etc/kubernetes/pki/ca.crt \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置客户端认证参数
    kubectl config set-credentials ${KUBE_USER} \
      --client-certificate=/etc/kubernetes/pki/${KUBE_CERT}.crt \
      --client-key=/etc/kubernetes/pki/${KUBE_CERT}.key \
      --embed-certs=true \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置上下文参数
    kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
      --cluster=${CLUSTER_NAME} \
      --user=${KUBE_USER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置当前使用的上下文
    kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 查看生成的配置文件
    kubectl config view --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
  • scheduler.conf

    KUBE_APISERVER="https://192.168.12.10:6443"
    CLUSTER_NAME="kubernetes"
    KUBE_USER="system:kube-scheduler"
    KUBE_CERT="kube-scheduler"
    KUBE_CONFIG="scheduler.conf"
    
    # 设置集群参数
    kubectl config set-cluster ${CLUSTER_NAME} \
      --certificate-authority=/etc/kubernetes/pki/ca.crt \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置客户端认证参数
    kubectl config set-credentials ${KUBE_USER} \
      --client-certificate=/etc/kubernetes/pki/${KUBE_CERT}.crt \
      --client-key=/etc/kubernetes/pki/${KUBE_CERT}.key \
      --embed-certs=true \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置上下文参数
    kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
      --cluster=${CLUSTER_NAME} \
      --user=${KUBE_USER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置当前使用的上下文
    kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 查看生成的配置文件
    kubectl config view --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
  • controller-manager.conf

    KUBE_APISERVER="https://192.168.12.10:6443"
    CLUSTER_NAME="kubernetes"
    KUBE_USER="system:kube-controller-manager"
    KUBE_CERT="kube-controller-manager"
    KUBE_CONFIG="controller-manager.conf"
    
    # 设置集群参数
    kubectl config set-cluster ${CLUSTER_NAME} \
      --certificate-authority=/etc/kubernetes/pki/ca.crt \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置客户端认证参数
    kubectl config set-credentials ${KUBE_USER} \
      --client-certificate=/etc/kubernetes/pki/${KUBE_CERT}.crt \
      --client-key=/etc/kubernetes/pki/sa.key \
      --embed-certs=true \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置上下文参数
    kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \
      --cluster=${CLUSTER_NAME} \
      --user=${KUBE_USER} \
      --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 设置当前使用的上下文
    kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
    # 查看生成的配置文件
    kubectl config view --kubeconfig=/etc/kubernetes/${KUBE_CONFIG}
    
  • 应用配置

    • 重启 Docker 和 Kubelet
      systemctl restart docker
      systemctl restart kubelet
      
    • 查看三个kubernetes组件(kubelet,controller-manager,scheduler)的日志,确认是否还有证书过期报错信息。
  • Worker节点证书更新操作

    • 停止docker和kubelet

      systemctl stop docker
      systemctl stop kubelet
      
    • 删除kubelet.conf文件,文件一般在/etc/kubernetes目录下。

    • 编辑bootstrap-kubelet.conf文件(文件一般在/etc/kubernetes目录下),修改certificate-authority-data内容,与master节点中的admin.conf文件的该区域内容相同。

    • 备份后删除该目录下的文件rm -rf /var/lib/kubelet/pki

    • 启动docker和kubelet

      systemctl start docker
      systemctl start kubelet
      

重启服务

  • 将全部kube-proxy重启
  • 将全部网络插件重启(比如:flannel)

你可能感兴趣的:(kubernetes刷新证书)