etcd故障节点

[root@k8s-master1 ~]# kubectl get cs
NAME                 STATUS      MESSAGE                                  ERROR
controller-manager   Healthy     ok                                       
scheduler            Healthy     ok                                       
etcd-1               Healthy     {"health":"true"}                        
etcd-0               Healthy     {"health":"true"}                        
etcd-2               Unhealthy   HTTP probe failed with statuscode: 503

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.233.91:2379,https://192.168.233.93:2379,https://192.168.233.94:2379" endpoint health --write-out=table

1.将有故障的etcd节点remove出集群:
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.233.91:2379,https://192.168.233.93:2379,https://192.168.233.94:2379" --write-out=table member list

cf4f326398a30bd2

86ec40d44e54cf0a

/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.233.91:2379,https://192.168.233.93:2379,https://192.168.233.94:2379" member remove 故障节点的id   

2、来到故障节点

rm -rf /var/lib/etcd/default.etcd/member/

修改etcd配置文件,将下面new修改为:
vim /opt/etcd/cfg/etcd 

修改前:

ETCD_INITIAL_CLUSTER_STATE="new"
修改后:

ETCD_INITIAL_CLUSTER_STATE="existing"

3、重新加入etcd集群：
/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.233.91:2379,https://192.168.233.93:2379,https://192.168.233.94:2379" member add etcd-2 --peer-urls=https://192.168.233.94:2380

4、重启etcd故障节点