前言

Kubernetes 系统使用 Etcd 存储集群配置和一些状态信息。如果单一部署,etcd如果挂了会导致集群异常甚至集群崩溃无法恢复。所以单点部署是很风险的。所以我们要部署Etcd集群

项目地址:https://github.com/etcd-io/etcd

Etcd集群详解

环境说明:

etcd1 192.168.214.200

etcd2 192.168.214.201

etcd3 192.168.214.202

三台节点创建相关目录

[root@etcd1 ~]# mkdir -p /data/etcd
[root@etcd1 ~]# mkdir -p /opt/kubernetes/{bin,conf,ssl}

将我们之前生成的证书分发到节点的/opt/kubernetes/ssl/目录下,详见上篇文档《证书详解》

etcd集群用到的证书为ca.pem、kubernetes-key.pem、kubernetes.pem

[root@master1 ssl]# scp *.pem etcd1:/opt/kubernetes/ssl
[root@master1 ssl]# scp *.pem etcd2:/opt/kubernetes/ssl
[root@master1 ssl]# scp *.pem etcd3:/opt/kubernetes/ssl

三台etcd节点下载并解压二进制文件

[root@etcd1 ~]# wget  https://github.com/coreos/etcd/releases/download/v3.2.11/etcd-v3.2.11-linux-amd64.tar.gz
[root@etcd1 ~]# tar xzvf etcd-v3.2.11-linux-amd64.tar.gz
[root@etcd1 ~]# cp etcd-v3.2.11-linux-amd64/etcd* /opt/kubernetes/bin/

创建etcd.service文件

在/usr/lib/systemd/system/下创建etcd.service

[root@etcd1 system]# vim etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=/data/etcd/
EnvironmentFile=-/opt/kubernetes/conf/etcd.conf
ExecStart=/opt/kubernetes/bin/etcd \
--name=etcd1 \
--cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--peer-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--peer-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--trusted-ca-file=/opt/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/ssl/ca.pem \
--initial-advertise-peer-urls=https://192.168.214.200:2380 \
--listen-peer-urls=https://192.168.214.200:2380 \
--listen-client-urls=https://192.168.214.200:2379,http://127.0.0.1:2379 \
--advertise-client-urls=https://192.168.214.200:2379 \
--initial-cluster-token=etcd-cluster-0 \
--initial-cluster=etcd1=https://192.168.214.200:2380,etcd2=https://192.168.214.201:2380,etcd3=https://192.168.214.202:2380 \
--initial-cluster-state=new \
--data-dir=/data/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

说明:

自行根据节点IP设置其他节点

指定 etcd 的工作目录和数据目录为 /data/etcd,需在启动服务前创建这个目录;

--initial-cluster-state 值为 new 时,--name 的参数值必须位于 --initial-cluster 列表中;

为了保证通信安全,需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书(trusted-ca-file);

增加环境变量

[root@etcd1 ~]# vim .bash_profile 
export PATH=/opt/kubernetes/bin:$PATH
[root@etcd1 ~]# source .bash_profile

启动etcd服务

[root@etcd1 ~]# systemctl daemon-reload
[root@etcd1 ~]# systemctl enable etcd
[root@etcd1 ~]# systemctl start etcd

集群验证

(1)验证集群状态

[root@etcd1 ssl]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem cluster-health
member 31a87df7577ee4e3 is healthy: got healthy result from https://192.168.214.201:2379
member d009f1b31e51b9c3 is healthy: got healthy result from https://192.168.214.202:2379
member fb94879e6d597fdf is healthy: got healthy result from https://192.168.214.200:2379
cluster is healthy

(2)通过集群节点etcd1设置一个值,到etcd2和etcd3获取,如果获取正常,则存储也没问题

[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem set dong "diss and peace"
diss and peace
[root@etcd2 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem get dong
diss and peace
[root@etcd3 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem get dong
diss and peace

(3)可以通过etcdctl member list查看谁是当前集群中选举出的leader

[root@etcd2 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem member list
31a87df7577ee4e3: name=etcd2 peerURLs=https://192.168.214.201:2380 clientURLs=https://192.168.214.201:2379 isLeader=false
d009f1b31e51b9c3: name=etcd3 peerURLs=https://192.168.214.202:2380 clientURLs=https://192.168.214.202:2379 isLeader=false
fb94879e6d597fdf: name=etcd1 peerURLs=https://192.168.214.200:2380 clientURLs=https://192.168.214.200:2379 isLeader=true

(4)验证leader切换。上步可以看出此时的leader为etcd1,停掉etcd1后,再次查看,leader变为了etcd2

[root@etcd1 ~]# systemctl stop etcd
[root@etcd2 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem member list
31a87df7577ee4e3: name=etcd2 peerURLs=https://192.168.214.201:2380 clientURLs=https://192.168.214.201:2379 isLeader=true
d009f1b31e51b9c3: name=etcd3 peerURLs=https://192.168.214.202:2380 clientURLs=https://192.168.214.202:2379 isLeader=false
fb94879e6d597fdf: name=etcd1 peerURLs=https://192.168.214.200:2380 clientURLs=https://192.168.214.200:2379 isLeader=false

创建kubernetes网段,用于后续的flannel请求调用,通过其他etcd集群节点如果可以查看到这个网段,说明网段创建成功。

[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem     mk /kubernetes/network/config '{ "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}'
{ "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}           #mk创建一个键值存储,同时会自动创建键值目录
[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem   ls -r
/dong             #ls -r列出键值存储的目录
/kubernetes
/kubernetes/network
/kubernetes/network/config
[root@etcd1 ~]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/kubernetes.pem --key-file=/opt/kubernetes/ssl/kubernetes-key.pem   get /kubernetes/network/config
{ "Network": "172.20.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}    #get查看键值存储

kubernetes二进制安装(一)环境说明与准备

kubernetes二进制安装(二)证书详解