【运维】K8S集群部署系列之ETCD集群搭建（三）
ETCD集群安全升级
文章目录
- ETCD集群安全升级
- 证书准备
- 证书拷贝
- 三步走升级为TLS安全集群
- Step1 开启集群外部`TLS`安全认证
- 修改master节点配置
- 修改master节点服务文件
- 重启master节点并验证
- 修改node1节点配置
- 修改node1节点服务文件
- 重启node1节点并验证
- 修改node2节点配置
- 修改node2节点服务文件
- 重启node2节点并验证
- Step2 开启客户端验证
- 修改master节点配置
- 修改master节点服务文件
- 重启master节点并验证
- 修改node1节点配置
- 修改node1节点服务文件
- 重启node1节点并验证
- 修改node2节点配置
- 修改node2节点服务文件
- 重启node2节点并验证
- Step3 开启集群内部`TLS`安全认证
- 查看集群节点标识
- 更新节点`peerURLs`链接为`https`方式
- 再次查看集群状态
- 节点开启内部`TLS`验证
- 查看当前的`ETCD`服务状态
- 修改master节点配置
- 修改node1节点配置
- 修改node2节点配置
- 修改服务文件
- 服务重启
- 再次查看的`ETCD`服务状态
【运维】K8S集群部署系列之ETCD集群搭建（一） 中已部署了由master、node1、node2三个节点组成的普通集群。创建TLS加密方式的ETCD安全集群可以采取删除旧集群重建和逐步升级的方式，相对于删除后重建的方式，逐步升级为安全集群可避免旧数据丢失，本文将采取逐步升级的方式。
证书准备
在【运维】K8S集群部署系列之ETCD集群搭建（二） 中我们已经制作了本文需要的相关证书。
证书拷贝
# 三个节点执行
mkdir -p /opt/etcd/pki
# 证书服务器执行
scp -P 22 *.pem [email protected]:/opt/etcd/pki/
scp -P 22 *.csr [email protected]:/opt/etcd/pki/
scp -P 22 *.pem [email protected]:/opt/etcd/pki/
scp -P 22 *.csr [email protected]:/opt/etcd/pki/
scp -P 22 *.pem [email protected]:/opt/etcd/pki/
scp -P 22 *.csr [email protected]:/opt/etcd/pki/
[root@master ~]# ls /opt/etcd/pki/
ca.csr ca-key.pem ca.pem etcd.csr etcdctl.csr etcdctl-key.pem etcdctl.pem etcd-key.pem etcd.pem peer.csr peer-key.pem peer.pem
三步走升级为TLS安全集群
Step1 开启集群外部TLS安全认证
外部访问即客户端访问服务端，
etcdctl和k8s的apiserver服务都属于外部客户端，该小节将用到服务器证书etcd.pem及其秘钥文件etcd-key.pem。
修改master节点配置
本节更新字段
ETCD_LISTEN_CLIENT_URLS和ETCD_ADVERTISE_CLIENT_URLS中的链接为https方式
新增字段ETCD_CERT_FILE和ETCD_KEY_FILE。
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.3:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.3:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.3:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
EOF
- 字段解释
ETCD_CERT_FILE，服务器证书，可以使用对等证书； ETCD_KEY_FILE，服务器证书私钥，可以使用对等证书私钥。
修改master节点服务文件
此处新增参数
--cert-file和--key-file
cat > vim /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
重启master节点并验证
-
重启master节点
systemctl daemon-reload && systemctl restart etcd -
集群状态查看
# 此时没加CA根证书，提示master节点不可达 etcdctl cluster-health # 加上CA根证书，集群验证通过,master节点链接变为https模式 etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-healthmember 8ada33a16cb8b5f9 is healthy: got healthy result from http://192.168.159.4:2379 member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379 member e689a191b9fab04f is healthy: got healthy result from http://192.168.159.5:2379 cluster is healthy
修改node1节点配置
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.4:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.4:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
EOF
修改node1节点服务文件
同master设置
重启node1节点并验证
-
重启master节点
systemctl daemon-reload && systemctl restart etcd -
集群状态查看
etcdctl cluster-health # 此时没加CA根证书，提示master节点不可达 etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群验证通过,节点链接变为https模式member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379 member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379 member e689a191b9fab04f is healthy: got healthy result from http://192.168.159.5:2379 cluster is healthy
修改node2节点配置
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.5:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.5:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
EOF
修改node2节点服务文件
同master设置
重启node2节点并验证
-
重启master节点
systemctl daemon-reload && systemctl restart etcd -
集群状态查看
etcdctl cluster-health # 此时没加CA根证书，提示master节点不可达
etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群验证通过,节点链接变为https模式
member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
cluster is healthy
Step2 开启客户端验证
即服务端对
etcdctl等客户端的验证。
修改master节点配置
本节新增字段
ETCD_CLIENT_CERT_AUTH和ETCD_TRUSTED_CA_FILE
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.3:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.3:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.3:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF
- 字段解释
ETCD_CLIENT_CERT_AUTH，开启客户端证书验证 ETCD_TRUSTED_CA_FILE，用于验证客户端验证的CA根证书
修改master节点服务文件
本节新增参数
--client-cert-auth和--trusted-ca-file
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
重启master节点并验证
-
重启master节点
systemctl daemon-reload && systemctl restart etcd -
集群状态查看
# 此时没加CA根证书，提示master节点不可达 etcdctl cluster-health # 加上CA根证书，master不可访问，因为没有配置客户端证书 etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群正常,此处也可以使用对等证书及其私钥进行验证，只要有客户端验证功能即可 etcdctl --ca-file=/opt/etcd/pki/ca.pem --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-healthmember 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379 member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379 member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379 cluster is healthy
修改node1节点配置
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.4:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.4:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF
修改node1节点服务文件
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
重启node1节点并验证
-
重启master节点
systemctl daemon-reload && systemctl restart etcd -
集群状态查看
# 此时没加CA根证书，提示master节点不可达 etcdctl cluster-health # 加上CA根证书，master不可访问，因为没有配置客户端证书 etcdctl -ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群正常,也可以使用对等证书及其私钥进行验证 etcdctl --ca-file=/opt/etcd/pki/ca.pem --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-healthmember 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379 member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379 member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379 cluster is healthy
修改node2节点配置
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.159.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.5:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.159.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.5:2379"
ETCD_INITIAL_CLUSTER="etcd-1=http://192.168.159.3:2380,etcd-2=http://192.168.159.4:2380,etcd-3=http://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF
修改node2节点服务文件
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \ # 开启客户端验证
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE} # 生成客户端证书的CA证书
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
重启node2节点并验证
-
重启master节点
systemctl daemon-reload && systemctl restart etcd -
集群状态查看
# 此时没加CA根证书，提示master节点不可达 etcdctl cluster-health # 加上CA根证书，master不可访问，因为没有配置客户端证书 etcdctl --ca-file /opt/etcd/pki/ca.pem cluster-health # 加上CA根证书，集群正常,也可以使用对等证书及其私钥进行验证 etcdctl --ca-file=/opt/etcd/pki/ca.pem --cert-file=etcdctl.pem --key-file=etcdctl-key.pem cluster-healthmember 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379 member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379 member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379 cluster is healthy
Step3 开启集群内部TLS安全认证
开启集群节点服务器间的内部通信
TLS安全认证。
查看集群节点标识
etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member list
可以看到此时peerURLs任然是http的方式。
8ada33a16cb8b5f9: name=etcd-2 peerURLs=http://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=false
df5c33b8666738a6: name=etcd-1 peerURLs=http://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=true
e689a191b9fab04f: name=etcd-3 peerURLs=http://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
更新节点peerURLs链接为https方式
为了避免日志中出现无关报错，先将
peerURLs链接更新为https是必要的；
命令调用方式：etcdctl member update。
etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member update df5c33b8666738a6 https://192.168.159.3:2380
etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member update 8ada33a16cb8b5f9 https://192.168.159.4:2380
etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member update e689a191b9fab04f https://192.168.159.5:2380
注意：
1、由于已开启集群的服务端和客户端验证，因此需要加上相关证书进行调用；
2、节点的memberID需要与上一步查看的必须一致；
3、节点的peerURLs的端口号和IP地址与上一步查看的必须一致。
再次查看集群状态
此时
peerURLs链接全部为https方式。
[root@master pki]# etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem member list
8ada33a16cb8b5f9: name=etcd-2 peerURLs=https://192.168.159.4:2380 clientURLs=https://192.168.159.4:2379 isLeader=false
df5c33b8666738a6: name=etcd-1 peerURLs=http://192.168.159.3:2380 clientURLs=https://192.168.159.3:2379 isLeader=true
e689a191b9fab04f: name=etcd-3 peerURLs=http://192.168.159.5:2380 clientURLs=https://192.168.159.5:2379 isLeader=false
[root@master pki]# etcdctl -ca-file=/opt/etcd/pki/ca.pem -cert-file=etcdctl.pem -key-file=etcdctl-key.pem cluster-health
member 8ada33a16cb8b5f9 is healthy: got healthy result from https://192.168.159.4:2379
member df5c33b8666738a6 is healthy: got healthy result from https://192.168.159.3:2379
member e689a191b9fab04f is healthy: got healthy result from https://192.168.159.5:2379
cluster is healthy
节点开启内部TLS验证
通过上述操作集群内部
https通信并没有真正建立，因为PEER_URLS的侦听地址和相关证书还没有配置；
如果单个节点的PEER_URLS开启https，则其余节点都需要配置相应证书和修改集群客户端侦听地址ETCD_INITIAL_CLUSTER，
集群内部才能正确通信。
查看当前的ETCD服务状态
可以看到日志中有
TLS相关的错误信息。
[root@master pki]# systemctl status etcd -l
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since 四 2019-08-08 14:11:42 CST; 8min ago
Main PID: 2720 (etcd)
CGroup: /system.slice/etcd.service
└─2720 /home/k8s/etcd/etcd --name=etcd-1 --data-dir=/opt/etcd/data --listen-peer-urls=https://192.168.159.3:2380 --listen-client-urls=https://192.168.159.3:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.159.3:2379 --initial-advertise-peer-urls=https://192.168.159.3:2380 --initial-cluster=etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/pki/etcd.pem --key-file=/opt/etcd/pki/etcd-key.pem --client-cert-auth=true --trusted-ca-file=/opt/etcd/pki/ca.pem
8月 08 14:11:42 master etcd[2720]: ready to serve client requests
8月 08 14:11:42 master etcd[2720]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
8月 08 14:11:42 master systemd[1]: Started Etcd Server.
8月 08 14:11:42 master etcd[2720]: rejected connection from "192.168.159.3:46646" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
8月 08 14:11:42 master etcd[2720]: WARNING: 2019/08/08 14:11:42 Failed to dial 192.168.159.3:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
8月 08 14:11:42 master etcd[2720]: peer e689a191b9fab04f became active
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream MsgApp v2 writer)
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream MsgApp v2 reader)
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream Message reader)
8月 08 14:11:42 master etcd[2720]: established a TCP streaming connection with peer e689a191b9fab04f (stream Message writer)
修改master节点配置
本节更新字段
ETCD_LISTEN_PEER_URLS、ETCD_INITIAL_ADVERTISE_PEER_URLS和ETCD_INITIAL_CLUSTER;
新增字段ETCD_PEER_CERT_FILE、ETCD_PEER_KEY_FILE、ETCD_PEER_CLIENT_CERT_AUTH和ETCD_PEER_TRUSTED_CA_FILE
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.3:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.3:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.3:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.3:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
ETCD_PEER_CERT_FILE="/opt/etcd/pki/peer.pem"
ETCD_PEER_KEY_FILE="/opt/etcd/pki/peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF
- 字段解释
ETCD_PEER_CERT_FILE， 服务端内部通信对等证书 ETCD_PEER_KEY_FILE， 服务端内部通信对等证书私钥 ETCD_PEER_CLIENT_CERT_AUTH，开启内部通信TLS验证 ETCD_PEER_TRUSTED_CA_FILE， 用于验证对等证书的CA根证书
修改node1节点配置
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.4:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.4:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.4:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.4:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
ETCD_PEER_CERT_FILE="/opt/etcd/pki/peer.pem"
ETCD_PEER_KEY_FILE="/opt/etcd/pki/peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF
修改node2节点配置
cat > /opt/etcd/etc/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.159.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.159.5:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.159.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.159.5:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#[Security]
ETCD_CERT_FILE="/opt/etcd/pki/etcd.pem"
ETCD_KEY_FILE="/opt/etcd/pki/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
ETCD_PEER_CERT_FILE="/opt/etcd/pki/peer.pem"
ETCD_PEER_KEY_FILE="/opt/etcd/pki/peer-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/opt/etcd/pki/ca.pem"
EOF
修改服务文件
本节在三个节点同时做以下操作。
新增参数peer-cert-file、--peer-key-file、--peer-client-cert-auth和--peer-trusted-ca-file
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/etc/etcd.conf
ExecStart=/home/k8s/etcd/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \
--cert-file=${ETCD_CERT_FILE} \
--key-file=${ETCD_KEY_FILE} \
--client-cert-auth=${ETCD_CLIENT_CERT_AUTH} \
--trusted-ca-file=${ETCD_TRUSTED_CA_FILE} \
--peer-cert-file=${ETCD_PEER_CERT_FILE} \
--peer-key-file=${ETCD_PEER_KEY_FILE} \
--peer-client-cert-auth=${ETCD_PEER_CLIENT_CERT_AUTH} \
--peer-trusted-ca-file=${ETCD_PEER_TRUSTED_CA_FILE}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
服务重启
systemctl daemon-reload && systemctl restart etcd
再次查看的ETCD服务状态
日志中没有再出现错误信息。
[root@master pki]# systemctl status etcd.service -l
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since 四 2019-08-08 14:28:19 CST; 20s ago
Main PID: 2812 (etcd)
CGroup: /system.slice/etcd.service
└─2812 /home/k8s/etcd/etcd --name=etcd-1 --data-dir=/opt/etcd/data --listen-peer-urls=https://192.168.159.3:2380 --listen-client-urls=https://192.168.159.3:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.159.3:2379 --initial-advertise-peer-urls=https://192.168.159.3:2380 --initial-cluster=etcd-1=https://192.168.159.3:2380,etcd-2=https://192.168.159.4:2380,etcd-3=https://192.168.159.5:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/pki/etcd.pem --key-file=/opt/etcd/pki/etcd-key.pem --client-cert-auth=true --trusted-ca-file=/opt/etcd/pki/ca.pem --peer-cert-file=/opt/etcd/pki/peer.pem --peer-key-file=/opt/etcd/pki/peer-key.pem --peer-client-cert-auth=true --peer-trusted-ca-file=/opt/etcd/pki/ca.pem
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 is starting a new election at term 167
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 became candidate at term 168
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 received MsgVoteResp from df5c33b8666738a6 at term 168
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 [logterm: 167, index: 104] sent MsgVote request to e689a191b9fab04f at term 168
8月 08 14:28:21 master etcd[2812]: df5c33b8666738a6 [logterm: 167, index: 104] sent MsgVote request to 8ada33a16cb8b5f9 at term 168
8月 08 14:28:21 master etcd[2812]: raft.node: df5c33b8666738a6 lost leader 8ada33a16cb8b5f9 at term 168
8月 08 14:28:22 master etcd[2812]: df5c33b8666738a6 [term: 168] received a MsgVote message with higher term from 8ada33a16cb8b5f9 [term: 170]
8月 08 14:28:22 master etcd[2812]: df5c33b8666738a6 became follower at term 170
8月 08 14:28:22 master etcd[2812]: df5c33b8666738a6 [logterm: 167, index: 104, vote: 0] cast MsgVote for 8ada33a16cb8b5f9 [logterm: 167, index: 104] at term 170
8月 08 14:28:22 master etcd[2812]: raft.node: df5c33b8666738a6 elected leader 8ada33a16cb8b5f9 at term 170
至此集群已经完全升级为
TLS安全通信方式，在生产环境中，该方式有效实现平滑升级，避免集群旧数据的丢失。
下一篇将继续介绍集群的扩容和节点删除操作。