k8s二进制（ETCD的部署安装）
| 角色 | ip | 组件 |
|---|---|---|
| k8s-master | 192.168.11.169 | kube-apiserver,kube-controller-manager,kube-scheduler,etcd |
| k8s-node1 | 192.168.11.164 | kubelet,kube-proxy,docker,etcd |
| k8s-node2 | 192.168.11.166 | kubelet,kube-proxy,docker,etcd |
1、为etcd签发证书
1、证书的下载(任意机器上执行都可以)
#工作目录
mkdir -vp /usr/local/kubernetes/{etcd,cfssl}
#进入带cfssl证书目录下
mkdir -vp /opt/etcd/{bin,ssl,cfg}
cd /usr/local/kubernetes/cfssl
#github二进制包下载地址：https://github.com/cloudflare/cfssl/releases
#wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
#wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
#wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64
# 生成证书
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# 利用Json生成证书
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# 查看证书信息的工具
https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
#其他版本下载地址：https://github.com/cloudflare/cfssl/tags
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
#\cp -R cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 /usr/local/bin/
#mv cfssl_linux-amd64 /usr/local/bin/cfssl
#mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
#mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
\cp -R cfssl_linux-amd64 /usr/local/bin/cfssl
\cp -R cfssljson_linux-amd64 /usr/local/bin/cfssljson
\cp -R cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
#查看版本信息
cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6
2、生成自签证书颁发机构（CA）
1、配置CA证书请求文件
cd /usr/local/kubernetes/etcd
#创建ca-csr.json
cat > ca-csr.json <<"EOF"
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "kubemsb",
"OU": "CN"
}
],
"ca": {
"expiry": "87600h"
}
}
EOF
2、生成CA
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#查看是否生成成功
ls *pem ca*csr
#显示ca-key.pem ca.pem这2个文件即生成成功
3、配置ca证书策略
#创建ca-config.json
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
- server auth 表示client可以对使用该ca对server提供证书进行验证
- client auth 表示server可以使用该ca对client提供的证书进行验证
3、使用自签CA 签发Etcd HTTPS 证书
1、创建证书申请文件
cd /usr/local/kubernetes/etcd
cat > etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.11.169",
"192.168.11.166",
"192.168.11.164"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C": "CN",
"ST": "Beijing",
"L": "Beijing",
"O": "kubemsb",
"OU": "CN"
}]
}
EOF
以上ip修改为自己集群环境的ip
2、生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
#查看证书是否生成成功
ls etcd*pem
2、部署etcd集群
1、下载etcd
cd /usr/local/kubernetes/etcd/
#下载etcd文件
wget https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz
#wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz
#系统对应的版本https://github.com/etcd-io/etcd/releases/download/v3.5.7/etcd-v3.5.7-linux-amd64.tar.gz
#解压缩
tar zxvf etcd-v3.5.2-linux-amd64.tar.gz
#将下载下来的二进制可执行文件复制到etcd的工作目录中(用于复制到其他服务器上，部署使用)
\cp -R etcd-v3.5.2-linux-amd64/{etcd,etcdctl} /opt/etcd/bin
核心是复制etcd,etcdctl可执行文件到指定目录中,进行后续统一的远程传递。以及etcd集群的搭建
2、创建etcd.conf配置文件
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#填写当前节点的ip地址即可
ETCD_LISTEN_PEER_URLS="https://192.168.11.169:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.11.169:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.11.169:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.11.169:2379"
#配置所有etcd集群对应的ip服务配置项
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.11.169:2380,etcd-2=https://192.168.11.166:2380,etcd-3=https://192.168.11.164:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
- 配置配置选项说明
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new 是新集群，existing 表示加入已有集群
3、systemd 管理etcd
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/etcd.pem \
--key-file=/opt/etcd/ssl/etcd-key.pem \
--peer-cert-file=/opt/etcd/ssl/etcd.pem \
--peer-key-file=/opt/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
4、拷贝生成的证书到ssl工作目录下
cd /usr/local/kubernetes/etcd
#复制ca办法机构和证书文件到etcd的ssl目录下准备生成etcd集群
\cp -R /usr/local/kubernetes/etcd/ca*.pem etcd*.pem /opt/etcd/ssl/
cd /opt/etcd
tree .
[root@k8s-master1 etcd]# tree .
.
├── bin
│ ├── etcd
│ └── etcdctl
├── cfg
│ └── etcd.conf
└── ssl
├── ca-key.pem
├── ca.pem
├── etcd-key.pem
└── etcd.pem
3 directories, 7 files
[root@k8s-master1 etcd]#
5、将节点etcd-1上的文件拷贝到节点2、3上
- 说明
1、 etcd-1上面的文件是已经生成好的,直接传输到需要部署etcd的服务器上即可
2、一定记得修改对应的ip配置等信息
3、节点etcd-2和etcd-3文件一直，需要更改为自己的ip地址
1、复制文件
#在master其他节点创建工作目录
#mkdir -vp /opt/etcd
#节点1复制到节点2上
scp -r /opt/etcd/* [email protected]:/opt/etcd
scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/
#节点1复制到节点3上
scp -r /opt/etcd/* [email protected]:/opt/etcd
scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/
2、修改对应的ip地址
vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 修改此处，节点2 改为etcd-2，节点3 改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#将192.168.11.169修改为当前服务器的ip地址
ETCD_LISTEN_PEER_URLS="https://192.168.11.169:2380" # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.11.169:2379" # 修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.11.169:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.11.169:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.11.169:2380,etcd-2=https://192.168.11.166:2380,etcd-3=https://192.168.11.164:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
6、启动并设置开机启动
节点都执行启动命令
systemctl daemon-reload
#启动etcd
systemctl start etcd
#开机启动
systemctl enable etcd
#查看启动状态
systemctl status etcd
7、记录错误
1、某一台服务一直启动不起起来
request sent was ignored (cluster ID mismatch: remote[9e5055697e
解决办法是：在配置的ETCD_DATA_DIR删除下面的文件
8、集群状态查看
- 里面的ip由于做了很多次、ip修改为自己的正确ip
1、验证etcd集群状态
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 endpoint health
2、集群状态性能
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 check perf
3、集群状态成员列表
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 member list
4、查看集群状态(主从状态)
ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 endpoint status
