《kubernetes-1.8.0》02-etcd群集搭建
《kubernetes-1.8.0》02-etcd群集搭建
—《kubernetes 1.8.0 测试环境安装部署》
— 时间：2017-11-22
一、etcd软件安装
通过在node-131、node-132、node-133上yum安装，也可使用mritd提供tarball中的etcd rpm包。
yum install -y etcd 当前版本为”etcd-3.2.701.el7”
二、分发etcd相关证书
分发证书
$ cd ~/etcd_ssl
$ for IP in `seq 131 133`;do
ssh root@172.18.169.$IP mkdir /etc/etcd/ssl
scp *.pem root@172.18.169.$IP:/etc/etcd/ssl
ssh root@172.18.169.$IP chown -R etcd:etcd /etc/etcd/ssl
ssh root@172.18.169.$IP chmod -R 644 /etc/etcd/ssl/*
ssh [email protected].$IP chmod 755 /etc/etcd/ssl
done设置etcd数据目录owner/group
for IP in `seq 131 133`;do
ssh root@172.18.169.$IP chown -R etcd:etcd /var/lib/etcd
done三、修改etcd配置文件
$ vim /etc/etcd/etcd.conf示例：
# [member]
ETCD_NAME=node-131
ETCD_DATA_DIR="/var/lib/etcd/node-131.etcd"
ETCD_WAL_DIR="/var/lib/etcd/wal"
ETCD_SNAPSHOT_COUNT="100"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://172.18.169.131:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.18.169.131:2379,http://127.0.0.1:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
#ETCD_CORS=""
# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.18.169.131:2380"
# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="node-131=https://172.18.169.131:2380,node-132=https://172.18.169.132:2380,node-133=https://172.18.169.133:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://172.18.169.131:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_STRICT_RECONFIG_CHECK="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
# [proxy]
#ETCD_PROXY="off"
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
# [security]
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_PEER_AUTO_TLS="true"
# [logging]
#ETCD_DEBUG="false"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
#ETCD_LOG_PACKAGE_LEVELS=""ETCD_NAME: etcd节点名称，如果是静态etcd cluster，必须与ETCD_INITIAL_CLUSTER中的名称进行对应。ETCD_INITIAL_CLUSTER_STATE: new为新建集群，如果是加入一个已经存在的etcd集群，需将该参数改为existingETCD_DATA_DIR=：存放etcd member等db数据ETCD_CLIENT_CERT_AUTH、ETCD_TRUSTED_CA_FILE、ETCD_CERT_FILE、ETCD_KEY_FILE等：为etcd TLS所需证书，制定之前创建的证书即可。- 更多etcd TLS 选项查看 etcd 3.2.7 document
node-132、node-133修改对应参数。
三、启动etcd服务并验证
node-131、node-132、node-133上执行：
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd检查节点状态：
$ export ETCDCTL_API=3
$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
endpoint health
https://172.18.169.133:2379 is healthy: successfully committed proposal: took = 2.016793ms
https://172.18.169.132:2379 is healthy: successfully committed proposal: took = 2.005839ms
https://172.18.169.131:2379 is healthy: successfully committed proposal: took = 1.167565ms检查etcd 版本：
[root@node-131 etcd_ssl]# etcdctl version
etcdctl version: 3.2.7
API version: 3.2加上TLS之后 etcd api verison 自动切换成了3.2。原先的cluster-health ls pwd 什么的都不管用了。
四、删除、添加etcd节点
查看群集成员：
$ export ETCDCTL_API=3
$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
member list
5d5554b1f11aba62, started, node-131, https://172.18.169.131:2380, https://172.18.169.131:2379
8b10a60fc4b98fcb, started, node-133, https://172.18.169.133:2380, https://172.18.169.133:2379
cd1bf9a8ae65b314, started, node-132, https://172.18.169.132:2380, https://172.18.169.132:2379删除note-133：
$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
member remove 8b10a60fc4b98fcb
Member 8b10a60fc4b98fcb removed from cluster 3697c33650b7b984
$ etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
member list
5d5554b1f11aba62, started, node-131, https://172.18.169.131:2380, https://172.18.169.131:2379
cd1bf9a8ae65b314, started, node-132, https://172.18.169.132:2380, https://172.18.169.132:2379
将node-133添加回etcd群集：
群集节点添加member：
etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
member add node-133 \
--peer-urls=https://172.18.169.133:2380
Member 17948fc49f73cbb9 added to cluster 3697c33650b7b984
ETCD_NAME="node-133"
ETCD_INITIAL_CLUSTER="node-133=https://172.18.169.133:2380,node-131=https://172.18.169.131:2380,node-132=https://172.18.169.132:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"--peer-urls：api 3.2后加member需要增加这个选项
修改member设置
##清空member上的数据目录
$ sudo systemctl start etcd
$ sudo rm -rf /var/lib/etcd/*
##修改member etcd配置文件修改
ETCD_INITIAL_CLUSTER_STATE="existing"
##启动服务
$ sudo systemctl start etcd验证节点是否加入：
$ sudo etcdctl \
--cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
member list
17948fc49f73cbb9, started, node-133, https://172.18.169.133:2380, https://172.18.169.133:2379
5d5554b1f11aba62, started, node-131, https://172.18.169.131:2380, https://172.18.169.131:2379
cd1bf9a8ae65b314, started, node-132, https://172.18.169.132:2380, https://172.18.169.132:2379
$ sudo etcdctl \
--cacert=/etc/etcd/ssl/etcd-root-ca.pem \
--cert=/etc/etcd/ssl/etcd.pem \
--key=/etc/etcd/ssl/etcd-key.pem \
--endpoints=https://172.18.169.131:2379,https://172.18.169.132:2379,https://172.18.169.133:2379 \
endpoint health
https://172.18.169.133:2379 is healthy: successfully committed proposal: took = 2.348909ms
https://172.18.169.132:2379 is healthy: successfully committed proposal: took = 2.139596ms
https://172.18.169.131:2379 is healthy: successfully committed proposal: took = 1.222221ms至此etcd TLS集群搭建完成
本系列其他内容：
01-环境准备
02-etcd群集搭建
03-kubectl管理工具
04-master搭建
05-node节点搭建
06-addon-calico
07-addon-kubedns
08-addon-dashboard
09-addon-kube-prometheus
10-addon-EFK
11-addon-Harbor
12-addon-ingress-nginx
13-addon-traefik
参考链接：
https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/
https://github.com/opsnull/follow-me-install-kubernetes-cluster
https://coreos.com/etcd/docs/3.2.7/index.html