前言
Kubernetes 系统使用 Etcd 存储集群配置和一些状态信息。如果单一部署，etcd如果挂了会导致集群异常甚至集群崩溃无法恢复。所以单点部署是很风险的。所以我们要部署Etcd集群
项目地址：https://github.com/etcd-io/etcd
Etcd集群详解
环境说明：
etcd1 192.168.214.200
etcd2 192.168.214.201
etcd3 192.168.214.202
三台节点创建相关目录
[root@etcd1 ~]# mkdir -p /data/etcd
[root@etcd1 ~]# mkdir -p /opt/kubernetes/{bin,conf,ssl}
将我们之前生成的证书分发到节点的/opt/kubernetes/ssl/目录下，详见上篇文档《证书详解》
etcd集群用到的证书为ca.pem、kubernetes-key.pem、kubernetes.pem
[root@master1 ssl]# scp *.pem etcd1:/opt/kubernetes/ssl [root@master1 ssl]# scp *.pem etcd2:/opt/kubernetes/ssl [root@master1 ssl]# scp *.pem etcd3:/opt/kubernetes/ssl
三台etcd节点下载并解压二进制文件
[root@etcd1 ~]# wget https://github.com/coreos/etcd/releases/download/v3.2.11/etcd-v3.2.11-linux-amd64.tar.gz [root@etcd1 ~]# tar xzvf etcd-v3.2.11-linux-amd64.tar.gz [root@etcd1 ~]# cp etcd-v3.2.11-linux-amd64/etcd* /opt/kubernetes/bin/
创建etcd.service文件
在/usr/lib/systemd/system/下创建etcd.service
[root@etcd1 system]# vim etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/data/etcd/ EnvironmentFile=-/opt/kubernetes/conf/etcd.conf ExecStart=/opt/kubernetes/bin/etcd \ --name=etcd1 \ --cert-file=/opt/kubernetes/ssl/kubernetes.pem \ --key-file=/opt/kubernetes/ssl/kubernetes-key.pem \ --peer-cert-file=/opt/kubernetes/ssl/kubernetes.pem \ --peer-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \ --trusted-ca-file=/opt/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/opt/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls=https://192.168.214.200:2380 \ --listen-peer-urls=https://192.168.214.200:2380 \ --listen-client-urls=https://192.168.214.200:2379,http://127.0.0.1:2379 \ --advertise-client-urls=https://192.168.214.200:2379 \ --initial-cluster-token=etcd-cluster-0 \ --initial-cluster=etcd1=https://192.168.214.200:2380,etcd2=https://192.168.214.201:2380,etcd3=https://192.168.214.202:2380 \ --initial-cluster-state=new \ --data-dir=/data/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
说明：
自行根据节点IP设置其他节点
指定 etcd 的工作目录和数据目录为 /data/etcd，需在启动服务前创建这个目录；
--initial-cluster-state 值为 new 时，--name 的参数值必须位于 --initial-cluster 列表中；
为了保证通信安全，需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；
增加环境变量
[root@etcd1 ~]# vim .bash_profile export PATH=/opt/kubernetes/bin:$PATH [root@etcd1 ~]# source .bash_profile
启动etcd服务
[root@etcd1 ~]# systemctl daemon-reload [root@etcd1 ~]# systemctl enable etcd [root@etcd1 ~]# systemctl start etcd
集群验证
（1）验证集群状态
[root@etcd1 ssl]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem cluster-health member 31a87df7577ee4e3 is healthy: got healthy result from https://192.168.214.201:2379 member d009f1b31e51b9c3 is healthy: got healthy result from https://192.168.214.202:2379 member fb94879e6d597fdf is healthy: got healthy result from https://192.168.214.200:2379 cluster is healthy
（2）通过集群节点etcd1设置一个值，到etcd2和etcd3获取，如果获取正常，则存储也没问题
[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem set dong "diss and peace" diss and peace [root@etcd2 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem get dong diss and peace [root@etcd3 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem get dong diss and peace
（3）可以通过etcdctl member list查看谁是当前集群中选举出的leader
[root@etcd2 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem member list 31a87df7577ee4e3: name=etcd2 peerURLs=https://192.168.214.201:2380 clientURLs=https://192.168.214.201:2379 isLeader=false d009f1b31e51b9c3: name=etcd3 peerURLs=https://192.168.214.202:2380 clientURLs=https://192.168.214.202:2379 isLeader=false fb94879e6d597fdf: name=etcd1 peerURLs=https://192.168.214.200:2380 clientURLs=https://192.168.214.200:2379 isLeader=true
（4）验证leader切换。上步可以看出此时的leader为etcd1，停掉etcd1后，再次查看，leader变为了etcd2
[root@etcd1 ~]# systemctl stop etcd [root@etcd2 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem member list 31a87df7577ee4e3: name=etcd2 peerURLs=https://192.168.214.201:2380 clientURLs=https://192.168.214.201:2379 isLeader=true d009f1b31e51b9c3: name=etcd3 peerURLs=https://192.168.214.202:2380 clientURLs=https://192.168.214.202:2379 isLeader=false fb94879e6d597fdf: name=etcd1 peerURLs=https://192.168.214.200:2380 clientURLs=https://192.168.214.200:2379 isLeader=false
创建kubernetes网段，用于后续的flannel请求调用，通过其他etcd集群节点如果可以查看到这个网段，说明网段创建成功。
[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem mk /kubernetes/network/config '{ "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}'
{ "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }} #mk创建一个键值存储，同时会自动创建键值目录
[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem ls -r
/dong #ls -r列出键值存储的目录
/kubernetes
/kubernetes/network
/kubernetes/network/config
[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem get /kubernetes/network/config
{ "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }} #get查看键值存储
kubernetes二进制安装(一)环境说明与准备
kubernetes二进制安装(二)证书详解