使用Kubeadm + HAProxy + Keepalived部署高可用Kubernetes集群

#使用Kubeadm + HAProxy + Keepalived部署高可用Kubernetes集群

这两天kubernetes爆出第一个特权升级高危漏洞，波及非常广泛，且没有有效的补丁可以修改此漏洞，只能将kubernetes升级。说巧不巧，与此同时，kubernetes v1.13.0版本发布，kubeadm也升级为GA版本，简单阅读了一下更新日志，先部署一套练练手～

1、环境说明

本次高可用集群基本参照官网步骤进行部署，官网给出了两种拓扑结构：堆叠control plane node和external etcd node，本文基于第一种拓扑结构进行部署，使用Keepalived + HAProxy搭建高可用Load balancer，完整的拓扑图如下：

单个mastre节点将部署keepalived、haproxy、etcd、apiserver、controller-manager、schedule六种服务，load balancer集群和etcd集群仅用来为kubernetes集群集群服务，不对外营业。如果必要，可以将load balancer或者etcd单独部署，为kubernetes集群提供服务的同时，也可以为其他有需要的系统提供服务，比如下面这样的拓扑结构：

这种拓扑结构也对应external etcd node～

本文仅部署master节点，使用kubeadm部署worker节点非常简单，不在赘述，环境清单：

[root@master-0 kubernetes]# kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
master-0   Ready    master   4h2m    v1.13.0   172.16.7.11   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master-1   Ready    master   3h53m   v1.13.0   172.16.7.12   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master-2   Ready    master   3h52m   v1.13.0   172.16.7.13   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0

镜像清单：

[root@master-0 kubernetes]# docker images
REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                v1.13.0             8fa56d18961f        34 hours ago        80.2MB
k8s.gcr.io/kube-scheduler            v1.13.0             9508b7d8008d        34 hours ago        79.6MB
k8s.gcr.io/kube-apiserver            v1.13.0             f1ff9b7e3d6e        34 hours ago        181MB
k8s.gcr.io/kube-controller-manager   v1.13.0             d82530ead066        34 hours ago        146MB
quay.io/calico/node                  v3.3.1              427a0694c75c        3 weeks ago         75.3MB
quay.io/calico/cni                   v3.3.1              fa6f35a1c14d        3 weeks ago         75.4MB
k8s.gcr.io/coredns                   1.2.6               f59dcacceff4        4 weeks ago         40MB
k8s.gcr.io/etcd                      3.2.24              3cab8e1b9802        2 months ago        220MB
k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        11 months ago       742kB

主要软件清单：

• keepalived-1.3.5-6.el7.x86_64.rpm
• haproxy-1.5.18-8.el7.x86_64.rpm
• docker-ce-18.09.0-3.el7.x86_64.rpm
• kubeadm-1.13.0-0.x86_64.rpm
• kubectl-1.13.0-0.x86_64.rpm
• kubelet-1.13.0-0.x86_64.rpm

网盘下载地址：https://pan.baidu.com/s/1PwS5kiUd9G-oY2i9TgJVyA（包括rpm依赖）

2、部署步骤

2.1、部署keepalived

此处的keeplived的主要作用是为haproxy提供vip（172.16.7.10），在三个haproxy实例之间提供主备，降低当其中一个haproxy失效的时对服务的影响。

• 系统配置

[root@master-0 ~]# cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1
EOF
[root@master-0 ~]# sysctl -p
net.ipv4.ip_forward = 1

In order for the Keepalived service to forward network packets properly to the real servers, each router node must have IP forwarding turned on in the kernel.

• 安装keepalived

[root@master-0 ~]# yum install -y keepalived

• 配置keepalived

[root@master-2 ~]# cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 250
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 35f18af7190d51c9f7f78f37300a0cbd
    }
    virtual_ipaddress {
        172.16.7.10
    }
    track_script {
        check_haproxy
    }
}
EOF

• killall -0 根据进程名称检测进程是否存活
• master-0节点为***MASTER***，其余节点为***BACKUP***
• priority各个几点到优先级相差50，范围：0～250（非强制要求）

• 启动并检测服务

[root@master-0 ~]# systemctl enable keepalived.service 
[root@master-0 ~]# systemctl start keepalived.service
[root@master-0 ~]# systemctl status keepalived.service 
[root@master-0 ~]# ip address show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:50:56:25:99:6e brd ff:ff:ff:ff:ff:ff
    inet 172.16.7.11/24 brd 172.16.7.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 172.16.7.10/32 scope global ens33
       valid_lft forever preferred_lft forever

2.2、部署haproxy

此处的haproxy为apiserver提供反向代理，haproxy将所有请求轮询转发到每个master节点上。相对于仅仅使用keepalived主备模式仅单个master节点承载流量，这种方式更加合理、健壮。

• 系统配置

[root@master-0 ~]# cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_nonlocal_bind = 1
EOF
[root@master-0 ~]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1

Load balancing in HAProxy and Keepalived at the same time also requires the ability to bind to an IP address that are nonlocal, meaning that it is not assigned to a device on the local system. This allows a running load balancer instance to bind to an IP that is not local for failover.

• 安装haproxy

[root@master-0 ~]# yum install -y haproxy

• 配置haproxy

[root@master-0 ~]# cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server  master-0 172.16.7.11:6443 check
    server  master-1 172.16.7.12:6443 check
    server  master-2 172.16.7.13:6443 check

#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
    bind                 *:1080
    stats auth           admin:awesomePassword
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats
EOF

所以master节点上的配置完全相同～

• 启动并检测服务

[root@master-0 ~]# systemctl enable haproxy.service 
[root@master-0 ~]# systemctl start haproxy.service 
[root@master-0 ~]# systemctl status haproxy.service 
[root@master-0 ~]# ss -lnt | grep -E "16443|1080"
LISTEN     0      128          *:1080                     *:*                  
LISTEN     0      128          *:16443                    *:*

2.3、安装kubeadm、kubectl、kubelet、docker

• 系统配置

[root@master-0 ~]# systemctl disable firewalld.service 
[root@master-0 ~]# systemctl stop firewalld.service 
[root@master-0 ~]# systemctl status firewalld.service 

[root@master-0 ~]# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[root@master-0 ~]# setenforce 0

[root@master-0 ~]# sed -i 's/\(.*swap.*\)/# \1/g' /etc/fstab
[root@master-0 ~]# swapoff -a

[root@master-0 ~]# cat <  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@master-0 ~]# sysctl --system

• 安装软件

### 设置docker-ce的yum源 ###
[root@master-0 ~]# cat < /etc/yum.repos.d/docker-ce.repo
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg
EOF
### 安装docker-ce ###
[root@master-0 ~]# yum install -y docker-ce --disableexcludes=docker-ce-stable

### 设置kubernetes的yum源 ###
[root@master-0 ~]# cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
### 安装 ###
[root@master-0 ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

• 设置开机启动

[root@master-0 ~]# systemctl enable docker.service 
[root@master-0 ~]# systemctl start docker.service 
[root@master-0 ~]# systemctl status docker.service 

[root@master-0 ~]# systemctl enable kubelet.service 

2.4、部署master-0

• 编辑hosts文件

[root@master-0 ~]# cat >> /etc/hosts << EOF
172.16.7.10 cluster.kube.com

172.16.7.11 master-0
172.16.7.12 master-1
172.16.7.13 master-2
EOF

• 编辑kubeadm配置文件

[root@master-0 ~]# cat > kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
apiServer:
  certSANs:
    - "cluster.kube.com"
controlPlaneEndpoint: "cluster.kube.com:16443"
networking:
  podSubnet: "192.168.0.0/16"
EOF

CNI使用Calico，设置podSubnet: “192.168.0.0/16”

• 初始化第一个master节点

[root@master-0 ~]# kubeadm init --config kubeadm-config.yaml 
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-0 localhost] and IPs [172.16.7.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-0 localhost] and IPs [172.16.7.11 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local cluster.kube.com cluster.kube.com] and IPs [10.96.0.1 172.16.7.11]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.503612 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-0" as an annotation
[mark-control-plane] Marking the node master-0 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master-0 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 29kttk.v6a5ts4021pkc4zr
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join cluster.kube.com:16443 --token 29kttk.v6a5ts4021pkc4zr --discovery-token-ca-cert-hash sha256:d0bdfd403cd9822177f66c6f3fa28735a67c0747f52511f253ba82843b70f99a
[root@master-0 ~]# mkdir -p $HOME/.kube
[root@master-0 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master-0 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

• 安装网络插件

[root@master-0 ~]# kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
[root@master-0 ~]# kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
configmap/calico-config created
service/calico-typha created
deployment.apps/calico-typha created
poddisruptionbudget.policy/calico-typha created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created

• 复制相关文件到其他master节点上

[root@master-0 ~]# ssh root@master-1 mkdir -p /etc/kubernetes/pki/etcd
[root@master-0 ~]# scp /etc/kubernetes/admin.conf root@master-1:/etc/kubernetes
[root@master-0 ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master-1:/etc/kubernetes/pki
[root@master-0 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@master-1:/etc/kubernetes/pki/etcd

2.5、部署master-other

[root@master-1 ~]# kubeadm join cluster.kube.com:16443 --token 29kttk.v6a5ts4021pkc4zr --discovery-token-ca-cert-hash sha256:d0bdfd403cd9822177f66c6f3fa28735a67c0747f52511f253ba82843b70f99a --experimental-control-plane
[preflight] Running pre-flight checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[discovery] Trying to connect to API Server "cluster.kube.com:16443"
[discovery] Created cluster-info discovery client, requesting info from "https://cluster.kube.com:16443"
[discovery] Requesting info from "https://cluster.kube.com:16443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "cluster.kube.com:16443"
[discovery] Successfully established connection with API Server "cluster.kube.com:16443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[join] Running pre-flight checks before initializing the new control plane instance
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local cluster.kube.com cluster.kube.com] and IPs [10.96.0.1 172.16.7.12]
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-1 localhost] and IPs [172.16.7.12 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-1 localhost] and IPs [172.16.7.12 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Using existing up-to-date kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Checking Etcd cluster health
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-1" as an annotation
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

3、检查测试

• 查看kubernetes集群状态

[root@master-0 etcd]# kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
master-0   Ready    master   11m     v1.13.0   172.16.7.11   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master-1   Ready    master   2m16s   v1.13.0   172.16.7.12   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
master-2   Ready    master   45s     v1.13.0   172.16.7.13   <none>        CentOS Linux 7 (Core)   3.10.0-862.el7.x86_64   docker://18.9.0
[root@master-0 etcd]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok                   
scheduler            Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
[root@master-0 etcd]# kubectl get csr
NAME                                                   AGE     REQUESTOR                 CONDITION
csr-6jgcr                                              11m     system:node:master-0      Approved,Issued
node-csr-8wOc8SoFxlkm8Ep5QEr9erUexnYJ-lrilft4QUJ7RfI   2m26s   system:bootstrap:29kttk   Approved,Issued
node-csr-l-ijTesiUBRFmaAXIHYj3J4vYEA7qbJLolIsKv5FZuM   55s     system:bootstrap:29kttk   Approved,Issued

• 查看etcd集群状态

[root@master-0 ~]# kubectl exec -ti -n kube-system etcd-master-0 sh
/ # export ETCDCTL_API=3
/ # etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/ku
bernetes/pki/etcd/healthcheck-client.key member list
25e9e768b2abcbb6, started, master-2, https://172.16.7.13:2380, https://172.16.7.13:2379
28e1924984642da1, started, master-1, https://172.16.7.12:2380, https://172.16.7.12:2379
e19f37b34b9497ee, started, master-0, https://172.16.7.11:2380, https://172.16.7.11:2379