Kubernetes集群部署—部署etcd集群(二)

文章目录

    • 1 准备cfssl证书生成工具
    • 2 生成Etcd证书
        • 2.1 自签证书颁发机构(CA)
        • 2.2 使用自签CA签发Etcd HTTPS证书
    • 3 从Github下载etcd二进制文件
    • 4 部署Etcd集群
        • 4.1 创建工作目录并解压二进制包
        • 4.2 创建etcd配置文件
        • 4.3 systemd管理etcd
        • 4.4 拷贝刚才生成的证书
        • 4.5 将上面节点1所有生成的文件拷贝到节点2和节点3
        • 4.6 启动etcd服务并设置开机启动
        • 4.7 查看集群状态

Etcd 是一个分布式键值存储系统,Kubernetes使用Etcd进行数据存储,所以先准备一个Etcd数据库,为解决Etcd单点故障,应采用集群方式部署,这里使用3台组建集群,可容忍1台机器故障,当然,你也可以使用5台组建集群,可容忍2台机器故障。

节点名称 IP
etcd-1 10.20.17.20
etcd-2 10.20.17.21
etcd-3 10.20.17.22

注:为了节省机器,这里与K8s节点机器复用。也可以独立于k8s集群之外部署,只要apiserver能连接到就行

1 准备cfssl证书生成工具

cfssl是一个开源的证书管理工具,使用json文件生成证书,相比openssl更方便使用。

找任意一台服务器操作,这里用Master节点。

# mkdir -p /opt/tools
# mkdir -p /opt/tools/cfssl
# cd /opt/tools/cfssl/
# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2 生成Etcd证书

2.1 自签证书颁发机构(CA)

创建工作目录:

# mkdir -p /root/TLS/{etcd,k8s}
# mkdir -p /root/TLS/{etcd,k8s}
# cd /root/TLS/etcd

自签CA:

cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat > ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成证书:

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# ls *pem
ca-key.pem  ca.pem
2.2 使用自签CA签发Etcd HTTPS证书

创建证书申请文件:

cat > server-csr.json << EOF
{
    "CN": "etcd",
    "hosts": [
    "10.20.17.20",
    "10.20.17.21",
    "10.20.17.22"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF

注:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,一个都不能少!为了方便后期扩容可以多写几个预留的IP。

生成证书:

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
# ls server*pem
server-key.pem  server.pem

3 从Github下载etcd二进制文件

下载地址:https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

4 部署Etcd集群

以下在etcd 节点1上操作,为简化操作,待会将节点1生成的所有文件拷贝到节点2和节点3。

4.1 创建工作目录并解压二进制包
# mkdir -p /opt/etcd/{bin,cfg,ssl} 
# tar zxvf /opt/tools/etcd-v3.4.9-linux-amd64.tar.gz
# mv /opt/tools/etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
4.2 创建etcd配置文件
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.20.17.20:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.20.17.20:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.20.17.20:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.20.17.20:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://10.20.17.20:2380,etcd-2=https://10.20.17.21:2380,etcd-3=https://10.20.17.22:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
  • ETCD_NAME:节点名称,集群中唯一
  • ETCD_DATA_DIR:数据目录
  • ETCD_LISTEN_PEER_URLS:集群通信监听地址
  • ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
  • ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
  • ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
  • ETCD_INITIAL_CLUSTER:集群节点地址
  • ETCD_INITIAL_CLUSTER_TOKEN:集群Token
  • ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群
4.3 systemd管理etcd
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
4.4 拷贝刚才生成的证书
# cp /root/TLS/etcd/ca*pem /root/TLS/etcd/server*pem /opt/etcd/ssl/
4.5 将上面节点1所有生成的文件拷贝到节点2和节点3
scp -r /opt/etcd/ k8s-node1:/opt/
scp /usr/lib/systemd/system/etcd.service k8s-node1:/usr/lib/systemd/system/

scp -r /opt/etcd/ k8s-node2:/opt/
scp /usr/lib/systemd/system/etcd.service k8s-node2:/usr/lib/systemd/system/

然后在节点2和节点3分别修改etcd.conf配置文件中的节点名称和当前服务器IP:

[root@k8s-node1 bin]# vim /opt/etcd/cfg/etcd.conf

#[Member]
ETCD_NAME="etcd-1"  # 修改此处,节点2改为etcd-2,节点3改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.20.17.20:2380"  # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://10.20.17.20:2379"  # 修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.20.17.20:2380"  # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://10.20.17.20:2379"  # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://10.20.17.20:2380,etcd-2=https://10.20.17.21:2380,etcd-3=https://10.20.17.22:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
4.6 启动etcd服务并设置开机启动
systemctl daemon-reload    # 配置生效
systemctl start etcd       # 启动etcd
systemctl status etcd      # 查看启动状态
systemctl enable etcd      # 设置开机启动

注意:以上三个节点需要先启动其他节点的 etcd 而不是 master 的etcd !!!

启动master节点的etcd服务后,发现服务无法启动,可使用 journalctl -xe 命令或查看系统日志cat /var/log/messages
看到以下关于etcd的报错信息:

"msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"427a09770fe3b784","rtt":"0s","error":"dial tcp 10.20.17.21:2380: connect: connection refused"

报错原因:分析是因为etcd1的配置文件/etc/systemd/system/etcd.service 启动脚本中的ETCD_INITIAL_CLUSTER_STATE是new,而在配置中ETCD_INITIAL_CLUSTER写入了etcd2/3的IP:PORT,这时etcd1尝试去连接etcd2、etcd3,但是etcd2、3的etcd服务此时还未启动,因此需要先启动etcd2和3的etcd服务,再去启动etcd1。

4.7 查看集群状态
# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://10.20.17.2017.21:2379,https://10.20.17.22:2379" endpoint health

https://10.20.17.20:2379 is healthy: successfully committed proposal: took = 11.989312ms
https://10.20.17.21:2379 is healthy: successfully committed proposal: took = 12.942844ms
https://10.20.17.22:2379 is healthy: successfully committed proposal: took = 29.3212ms

如果输出上面信息,就说明集群部署成功。如果有问题第一步先看日志:/var/log/message 或 journalctl -u etcd

你可能感兴趣的:(Kubernetes,kubernetes)