K8S部署步骤：6-部署etcd集群

kuberntes系统使用etcd存储所有数据，本文档介绍部署一个三节点高可用的etcd集群的步骤，这三个节点复用kubernetes master机器，分别命名为ERP-KUBE-TEST、ERP-KUBE-TEST2、ERP-KUBE-TEST3：

• 10.61.16.39:ERP-KUBE-TEST
• 10.61.16.29:ERP-KUBE-TEST2
• 10.61.16.28:ERP-KUBE-TES3

导入环境变量

$ source /root/local/bin/environment.sh

下载二进制文件

到https://github.com/coreos/etcd/releases页面下载最新版本的二进制文件：

$ wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
$ tar -xvf etcd-v3.2.18-linux-amd64.tar.gz
$ sudo mv etcd-v3.2.18-linux-amd64/etcd* /root/local/bin

创建TLS秘钥和证书

为了保证通信安全，客户端（如etcdctl）与etcd集群、etcd集群之间的通信需要使用TLS加密，本节创建etcd TLS加密所需的证书和私钥。

创建etcd证书签名请求：

$ cat > etcd-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "${NODE_IP}"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

hosts字段指定授权使用该证书的etcd节点IP；

生成etcd证书和私钥：

$ cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem \
  -ca-key=/etc/kubernetes/ssl/ca-key.pem \
  -config=/etc/kubernetes/ssl/ca-config.json \
  -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
$ ls etcd*
# etcd.csr  etcd-csr.json  etcd-key.pem etcd.pem
$ sudo mkdir -p /etc/etcd/ssl
$ sudo mv etcd*.pem /etc/etcd/ssl
$ sudo rm -rf etcd.csr etcd-csr.json

创建etcd的systemd unit文件

# 必须先创建工作目录
$ sudo mkdir -p /var/lib/etcd
$ cat > etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/root/local/bin/etcd \\
  --name=${NODE_NAME} \\
  --cert-file=/etc/etcd/ssl/etcd.pem \\
  --key-file=/etc/etcd/ssl/etcd-key.pem \\
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --initial-advertise-peer-urls=https://${NODE_IP}:2380 \\
  --listen-peer-urls=https://${NODE_IP}:2380 \\
  --listen-client-urls=https://${NODE_IP}:2379,http://127.0.0.1:2379 \\
  --advertise-client-urls=https://${NODE_IP}:2379 \\
  --initial-cluster-token=etcd-cluster-0 \\
  --initial-cluster=${ETCD_NODES} \\
  --initial-cluster-state=new \\
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

• 指定etcd的工作目录和数据目录为/var/lib/etcd，需在启动服务前创建这个目录；
• 为了保证通信安全，需要指定etcd的公私钥(cert-file和key-file)、Peers通信的公私钥CA证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；
• –initial-cluster-state值为new时，–name的参数值必须位于–initial-cluster 列表中；

启动etcd服务

$ sudo mv etcd.service /etc/systemd/system/
$ sudo systemctl daemon-reload
$ sudo systemctl enable etcd
$ sudo systemctl start etcd
$ sudo systemctl status etcd

最先启动的etcd进程会卡住一段时间，等待其它节点上的etcd进程加入集群，为正常现象。

在所有的etcd节点重复上面的步骤，直到所有机器的etcd服务都已启动。

验证服务

部署完etcd集群后，在任一etcd集群节点上执行如下命令：

$ for ip in ${NODE_IPS}; do
  ETCDCTL_API=3 /root/local/bin/etcdctl \
  --endpoints=https://${ip}:2379  \
  --cacert=/etc/kubernetes/ssl/ca.pem \
  --cert=/etc/etcd/ssl/etcd.pem \
  --key=/etc/etcd/ssl/etcd-key.pem \
  endpoint health; done
# https://10.211.55.14:2379 is healthy: successfully committed proposal: took = 2.435191ms
# https://10.211.55.12:2379 is healthy: successfully committed proposal: took = 2.311742ms
# https://10.211.55.15:2379 is healthy: successfully committed proposal: took = 1.685821ms