原创内容,转载请注明出处
博主地址:https://aronligithub.github.io/
前言
在经过上一篇章关于etcd单台部署,启用https以及ca自签名,这个篇章就是介绍以及演示三台etcd部署以及使用CFSSL来生成CA证书
环境要求
1、三台安装centos7的服务器
2、具备访问互联网
3、关闭服务器的防火墙以及selinux
服务器拓扑
服务器部署组件以及IP清单
host name | 服务器IP | etcd节点name |
---|---|---|
server81 | 172.16.5.81 | infra1 |
server86 | 172.16.5.86 | infra2 |
server87 | 172.16.5.87 | infra3 |
CFSSL工具的安装
下载CFSSL的可执行二进制文件
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl*
下载执行过程截图如下
将这三个二进制可执行文件,修改名称复制到/usr/local/bin/下
cp -v cfssl_linux-amd64 /usr/local/bin/cfssl
cp -v cfssljson_linux-amd64 /usr/local/bin/cfssljson
cp -v cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
ls /usr/local/bin/cfssl*
使用CFSSL创建CA证书以及etcd的TLS认证证书
创建 CA (Certificate Authority)
创建 CA 配置文件(ca-config.json)
[root@server81 etcdSSl]# vim ca-config.json
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"etcd": {
"usages": [
"signing",
"key encipherment",
"default": {
"expiry": "876000h"
},
"profiles": {
"etcd": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "876000h"
}
}
}
}
"字段说明"
"ca-config.json":可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;后续在签名证书时使用某个 profile;
"signing":表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE;
"server auth":表示client可以用该 CA 对server提供的证书进行验证;
"client auth":表示server可以用该CA对client提供的证书进行验证;
创建 CA 证书签名请求(ca-csr.json)
[root@server81 etcdSSl]# vim ca-csr.json
{
"CN": "etcd",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shenzhen",
"L": "shenzhen",
"O": "etcd",
"OU": "System"
}
]
}
"CN":Common Name,etcd 从证书中提取该字段作为请求的用户名 (User Name);浏览器使用该字段验证网站是否合法;
"O":Organization,etcd 从证书中提取该字段作为请求用户所属的组 (Group);
这两个参数在后面的kubernetes启用RBAC模式中很重要,因为需要设置kubelet、admin等角色权限,那么在配置证书的时候就必须配置对了,具体后面在部署kubernetes的时候会进行讲解。
"在etcd这两个参数没太大的重要意义,跟着配置就好。"
生成 CA 证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
说明:生成 "ca-csr.json ca-key.pem ca.pem" 三个文件
创建etcd的TLS认证证书
创建 etcd证书签名请求(etcd-csr.json)
[root@server81 etcdSSl]# vim etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"172.16.5.81",
"172.16.5.86",
"172.16.5.87"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "shenzhen",
"L": "shenzhen",
"O": "etcd",
"OU": "System"
}
]
}
[^_^]:
如果 hosts 字段不为空,则需要指定授权使用该证书的 IP 或域名列表。
[>_<]:
该证书被 etcd 集群使用,所以填写三台etcd服务器的IP即可。
生成 etcd证书和私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd
[^_^]:
生成 "etcd-csr.json etcd-key.pem etcd.pem" 三个文件。
将TLS 认证文件拷贝至证书目录下
mkdir -p /etc/etcd/etcdSSL
cp * /etc/etcd/etcdSSL
[^_^]:
存在CA证书的路径"/etc/etcd/etcdSSL"是自定义的,看个人习惯创建即可。
Server81 安装etcd服务
yum install -y etcd
配置 etcd 的 service文件(/usr/lib/systemd/system)
root@server81 ~]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
--name ${ETCD_NAME} \
--cert-file=/etc/etcd/etcdSSL/etcd.pem \
--key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \
--peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster infra1=https://172.16.5.81:2380,infra2=https://172.16.5.86:2380,infra3=https://172.16.5.87:2380 \
--initial-cluster-state new \
--data-dir=${ETCD_DATA_DIR}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
参数说明:
1、指定 etcd 的工作目录为 /var/lib/etcd,数据目录为 /var/lib/etcd,需在启动服务前创建这两个目录;
在配置中的命令是这条:
WorkingDirectory=/var/lib/etcd/
2、为了保证通信安全,需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书(trusted-ca-file);
在配置中添加etcd证书的命令是以下:
--cert-file=/etc/etcd/etcdSSL/etcd.pem \
--key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \
--peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
3、配置etcd的endpoint:
--initial-cluster infra1=https://172.16.5.81:2380,infra2=https://172.16.5.86:2380,infra3=https://172.16.5.87:2380 \
4、配置etcd的监听服务集群:
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
5、配置etcd创建的集群为新集群,则定义集群状态为new
--initial-cluster-state 值为 new
6、定义etcd节点的名称,该名称等下从配置文件中获取:
--name ${ETCD_NAME} \
其中配置文件:EnvironmentFile=-/etc/etcd/etcd.conf
etcd的配置文件(/etc/etcd/etcd.conf)
[root@server81 ~]# vim /etc/etcd/etcd.conf
#[member]
ETCD_NAME=infra1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.5.81:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.5.81:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.5.81:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.5.81:2379"
这是172.16.5.81节点的配置,如果配置其他etcd节点只要将上面的IP地址改成相应节点的IP地址即可。
启动 etcd 服务
systemctl daemon-reload
systemctl start etcd
systemctl status etcd
到了这里基本etcd的证书以及Server81的etcd节点服务已经配置好了,下面就是快速部署配置一下Server86/87两台服务器的etcd证书以及服务。
Server86安装etcd服务
yum install -y etcd
配置 etcd 的 service文件(/usr/lib/systemd/system)
[root@server86 ~]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
--name ${ETCD_NAME} \
--cert-file=/etc/etcd/etcdSSL/etcd.pem \
--key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \
--peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster infra1=https://172.16.5.81:2380,infra2=https://172.16.5.86:2380,infra3=https://172.16.5.87:2380 \
--initial-cluster-state new \
--data-dir=${ETCD_DATA_DIR}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
etcd的配置文件(/etc/etcd/etcd.conf)
[root@server86 ~]# vim /etc/etcd/etcd.conf
#[member]
ETCD_NAME=infra2
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.5.86:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.5.86:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.5.86:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.5.86:2379"
这是172.16.5.86节点的配置,如果配置其他etcd节点只要将上面的IP地址改成相应节点的IP地址即可。
还要将ETCD_NAME改一下名称。
从Server81服务器拷贝etcd的TLS证书文件至Server86服务器
[root@server81 etc]# cd etcd/
[root@server81 etcd]# ls
etcd.conf etcdSSL
[root@server81 etcd]#
[root@server81 etcd]# pwd
/etc/etcd
[root@server81 etcd]#
[root@server81 etcd]# scp -r etcdSSL root@server86:/etc/etcd
ca-config.json 100% 288 82.3KB/s 00:00
ca.csr 100% 997 283.1KB/s 00:00
ca-csr.json 100% 205 60.5KB/s 00:00
ca-key.pem 100% 1675 395.4KB/s 00:00
ca.pem 100% 1350 356.7KB/s 00:00
etcd.csr 100% 1066 50.8KB/s 00:00
etcd-csr.json 100% 296 40.1KB/s 00:00
etcd-key.pem 100% 1675 356.8KB/s 00:00
etcd.pem 100% 1436 341.3KB/s 00:00
[root@server81 etcd]#
启动 etcd 服务
systemctl daemon-reload
systemctl start etcd
systemctl status etcd
Server87安装etcd服务
yum install -y etcd
配置 etcd 的 service文件(/usr/lib/systemd/system)
[root@server87 ~]# vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd \
--name ${ETCD_NAME} \
--cert-file=/etc/etcd/etcdSSL/etcd.pem \
--key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--peer-cert-file=/etc/etcd/etcdSSL/etcd.pem \
--peer-key-file=/etc/etcd/etcdSSL/etcd-key.pem \
--trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--peer-trusted-ca-file=/etc/etcd/etcdSSL/ca.pem \
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster infra1=https://172.16.5.81:2380,infra2=https://172.16.5.86:2380,infra3=https://172.16.5.87:2380 \
--initial-cluster-state new \
--data-dir=${ETCD_DATA_DIR}
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
etcd的配置文件(/etc/etcd/etcd.conf)
[root@server87 ~]# vim /etc/etcd/etcd.conf
#[member]
ETCD_NAME=infra3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.5.87:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.5.87:2379"
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.5.87:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.5.87:2379"
这是172.16.5.87节点的配置,如果配置其他etcd节点只要将上面的IP地址改成相应节点的IP地址即可。
还要将ETCD_NAME改一下名称。
从Server81服务器拷贝etcd的TLS证书文件至Server87服务器
[root@server81 etcd]# scp -r etcdSSL root@server87:/etc/etcd
ca-config.json 100% 288 109.8KB/s 00:00
ca.csr 100% 997 417.5KB/s 00:00
ca-csr.json 100% 205 79.4KB/s 00:00
ca-key.pem 100% 1675 575.7KB/s 00:00
ca.pem 100% 1350 485.5KB/s 00:00
etcd.csr 100% 1066 289.0KB/s 00:00
etcd-csr.json 100% 296 120.7KB/s 00:00
etcd-key.pem 100% 1675 601.9KB/s 00:00
etcd.pem 100% 1436 104.4KB/s 00:00
[root@server81 etcd]#
启动 etcd 服务
systemctl daemon-reload
systemctl start etcd
systemctl status etcd
最后回到server81服务器,检查一下etcd集群是否正常
检查刚才启动etcd失败的Server81的节点
[root@server81 etcd]# service etcd status
Redirecting to /bin/systemctl status etcd.service
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-08-14 16:44:46 HKT; 10min ago
Main PID: 13905 (etcd)
CGroup: /system.slice/etcd.service
└─13905 /usr/bin/etcd --name infra1 --cert-file=/etc/etcd/etcdSSL/etcd.pem --key-file=/etc/etcd/etcdSSL/etcd-key.pem --peer-cert-file=/etc/etcd/etcdSSL/etcd.pem
Aug 14 16:54:14 server81 etcd[13905]: established a TCP streaming connection with peer 9281d0058fa51de0 (stream Message writer)
Aug 14 16:54:14 server81 etcd[13905]: established a TCP streaming connection with peer 9281d0058fa51de0 (stream MsgApp v2 writer)
Aug 14 16:54:14 server81 etcd[13905]: health check for peer 9281d0058fa51de0 could not connect: dial tcp 172.16.5.87:2380: getsockopt: connection refused
Aug 14 16:54:15 server81 etcd[13905]: updating the cluster version from 3.0 to 3.2
Aug 14 16:54:15 server81 etcd[13905]: updated the cluster version from 3.0 to 3.2
Aug 14 16:54:15 server81 etcd[13905]: enabled capabilities for version 3.2
Aug 14 16:54:20 server81 etcd[13905]: 8da195f9b25012c9 [term: 1972] received a MsgApp message with higher term from 9281d0058fa51de0 [term: 1973]
Aug 14 16:54:20 server81 etcd[13905]: 8da195f9b25012c9 became follower at term 1973
Aug 14 16:54:20 server81 etcd[13905]: raft.node: 8da195f9b25012c9 changed leader from 8da195f9b25012c9 to 9281d0058fa51de0 at term 1973
Aug 14 16:54:20 server81 etcd[13905]: 8da195f9b25012c9 [logterm: 1973, index: 13, vote: 0] rejected MsgVote from 9281d0058fa51de0 [logterm: 1972, index: 12] at term 1973
[root@server81 etcd]#
检查etcd集群服务的健康
etcdctl \
--ca-file=/etc/etcd/etcdSSL/ca.pem \
--cert-file=/etc/etcd/etcdSSL/etcd.pem \
--key-file=/etc/etcd/etcdSSL/etcd-key.pem \
cluster-health
最后的总结
从上面的流程对于新手基本都是可以复制黏贴就可以完成的了。但是为了能够快速执行etcd集群一键部署,我在后面写好了离线etcd三台的自动化集群部署脚本,如下图所示:
在这里已经部署好etcd集群服务了,那么在此基础上,我们下一篇章就开始部署kubernetes的服务部署了。
如果你想要看我写的总体系列文章目录介绍,可以点击kuberntes以及运维开发文章目录介绍
关注微信公众号,回复【资料】、Python、PHP、JAVA、web,则可获得Python、PHP、JAVA、前端等视频资料。