不知在哪篇技术文档中看到，kubernetes master和etcd分开部署模式，因为集群的状态都保存在etcd中，这样当kubernetes master挂掉后，通过API Server交互的Scale等功能无法使用外，其他已经部署的Pod仍然能继续工作。
基于这种考虑，通过yum以及修改etcd.conf方式部署了一个三节点的etcd集群，但对于企业使用而言，虽然在局域网内访问，多数情况下还是需要配置安全证书，就好像很多政府部门因为三级等保的要求必须在weblogic中配置ssl一样，自己尝试在之前的环境中通过修改conf文件下配置，启动时遭遇各种问题失败，但同样的证书后修改为命令行方式配置后以及手工安装etcd后部署成功。记录如下:
安装cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
证书相关处理
证书名称 |
配置文件 |
用途 |
etcd-root-ca.pem |
etcd-root-ca-csr.json |
etcd 根 CA 证书 |
etcd.pem |
etcd-gencert.json、etcd-csr.json |
etcd 集群证书 |
Etcd 证书生成所需配置文件如下:
etcd-root-ca-csr.json
{ "key": {
"algo": "rsa",
"size": 4096
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"L": "Beijing",
"ST": "Beijing",
"C": "CN"
}
],
"CN": "etcd-root-ca"
}
etcd-gencert.json
{
"signing": {
"default": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
etcd-csr.json
{
"key": {
"algo": "rsa",
"size": 4096
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"L": "Beijing",
"ST": "Beijing",
"C": "CN"
}
],
"CN": "etcd",
"hosts": [
"127.0.0.1",
"localhost",
"192.168.0.153",
"192.168.0.154",
"192.168.0.164",
"master",
"node1",
"node2"
]
}
最后生成 Etcd 证书
cfssl gencert --initca=true etcd-root-ca-csr.json | cfssljson --bare etcd-root-ca
cfssl gencert --ca etcd-root-ca.pem --ca-key etcd-root-ca-key.pem --config etcd-gencert.json etcd-csr.json | cfssljson --bare etcd
生成的证书列表如下
三、部署 HA ETCD
安装前准备
关闭 selinux, setenforce 0
关闭防火墙， systemctl stop firewalld; iptables -F
ntpdate 时间同步
ntpdate time1.aliyun.com
安装 Etcd
以下操作都是在master节点上操作
ETCD 直接采用 rpm 安装，RPM 可以从 Fedora 官方仓库 获取 spec 文件自己 build，或者直接从 rpmFind 网站 搜索
下载 rpm包
wget ftp://195.220.108.108/linux/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/e/etcd-3.2.7-1.fc28.x86_64.rpm
#分发并安装
I="192.168.0.153 192.168.0.154 192.168.0.164"
for IP in $I; do
etcd-3.2.7-1.fc28.x86_64.rpm root@$IP:~
ssh root@$IP rpm -ivh etcd-3.2.7-1.fc28.x86_64.rpm
done
分发证书
I="192.168.0.153 192.168.0.154 192.168.0.164"
for IP in $I; do
ssh root@$IP mkdir /etc/etcd/ssl/
scp *.pem root@$IP:/etc/etcd/ssl/
ssh root@$IP chown -R etcd:etcd /etc/etcd/ssl/
ssh root@$IP chmod -R 755 /etc/etcd/
done
修改配置
rpm 安装好以后直接修改 /etc/etcd/etcd.conf 配置文件即可，其中单个节点配置如下(其他节点只是名字和 IP 不同)
# [member]
ETCD_NAME=etcd0
ETCD_DATA_DIR="/var/lib/etcd/etcd0.etcd"
ETCD_WAL_DIR="/var/lib/etcd/wal"
ETCD_SNAPSHOT_COUNT="100"
ETCD_HEARTBEAT_INTERVAL="100"
ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.0.153:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.153:2379,http://127.0.0.1:2379"
ETCD_MAX_SNAPSHOTS="5"
ETCD_MAX_WALS="5"
#ETCD_CORS=""
# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.153:2380"
# if you use different ETCD_NAME (e.g. test), set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd0=https://192.168.0.153:2380,etcd1=https://192.168.0.154:2380,etcd2=https://192.168.0.164:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.153:2379"
#ETCD_DISCOVERY=""
#ETCD_DISCOVERY_SRV=""
#ETCD_DISCOVERY_FALLBACK="proxy"
#ETCD_DISCOVERY_PROXY=""
#ETCD_STRICT_RECONFIG_CHECK="false"
#ETCD_AUTO_COMPACTION_RETENTION="0"
# [proxy]
#ETCD_PROXY="off"/
#ETCD_PROXY_FAILURE_WAIT="5000"
#ETCD_PROXY_REFRESH_INTERVAL="30000"
#ETCD_PROXY_DIAL_TIMEOUT="1000"
#ETCD_PROXY_WRITE_TIMEOUT="5000"
#ETCD_PROXY_READ_TIMEOUT="0"
# [security]
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-root-ca.pem"
ETCD_PEER_AUTO_TLS="true"
# [logging]
#ETCD_DEBUG="false"
# examples for -log-package-levels etcdserver=WARNING,security=DEBUG
#ETCD_LOG_PACKAGE_LEVELS=""
node 节点要修改的地方：
ETCD_NAME
ETCD_LISTEN_PEER_URLS
ETCD_LISTEN_CLIENT_URLS
ETCD_INITIAL_ADVERTISE_PEER_URLS
ETCD_ADVERTISE_CLIENT_URLS
修改完成后，还需要修改/usr/lib/systemd/system/etcd.service文件内容如下：
1. [Unit]
2. Description=Etcd Server
3. After=network.target
4. After=network-online.target
5. Wants=network-online.target
6. [Service]
7. Type=notify
8. WorkingDirectory=/var/lib/etcd/
9. EnvironmentFile=-/etc/etcd/etcd.conf
10. User=etcd
11. # set GOMAXPROCS to number of processors
12. ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
13. --name=\"${ETCD_NAME}\" \
14. --data-dir=\"${ETCD_DATA_DIR}\" \
15. --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
16. --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
17. --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
18. --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" \
19. --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" \
20. --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
21. Restart=on-failure
22. LimitNOFILE=65536
23. [Install]
24. WantedBy=multi-user.target
4、启动验证
配置修改后在每个节点进行启动即可，注意，Etcd 各个节点间必须保证时钟同步，否则会造成启动失败等错误
systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
启动成功后验证节点状态
export ETCDCTL_API=3
etcdctl --cacert=/etc/etcd/ssl/etcd-root-ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.0.153:2379,https://192.168.0.154:2379,https://192.168.0.164:2379 endpoint health
本文出自https://www.cnblogs.com/Tempted/p/7737361.html
参考http://www.361way.com/etcd-cluster/5468.html