安装前规划
etcd1服务器 172.16.0.5
etcd
haproxy(apiserver做负载)
keepalive
chronyclient
etcd2服务器 172.16.0.6
etcd
haproxy
keepalive
chronyclient
etcd3服务器 172.16.0.7
etcd
haproxy
keepalive
chronyclient
master1 服务器 172.16.0.2
kube-apiserver
kube-schedule
kube-controll
docker(私有仓库)
chronyserver(时间服务器)
master2 172.16.0.3
kube-apiserver
kube-schedule
kube-controll
chronyclient
master3 172.16.0.4
kube-apiserver
kube-schedule
kube-controll
chronyclient
node1 172.16.0.8
kubelet
kube-proxy
docker
chronyclient
node2 172.16.0.9
kubelet
kube-proxy
docker
chronyclient
yum镜像
1.用国内源,才可以安装下面的软件
cat </etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF
重建yum缓存
yum -y install epel-release
yum clean all
yum makecache
centos 7.5
docker 1.11.3
kubernetes 1.11.0(tar包安装)
chrony(yum 安装)
ansible(yum 安装)
cfssl(yum 安装)
haproxy(yum 安装)
keepalived(yum 安装)
flannel(yum 安装)
master集群
172.16.0.2 master1+docker(私有仓库)
172.16.0.3 master2
172.16.0.4 master3
apiserver-vip
172.16.0.100
etcd集群
172.16.0.5 etcd1+haproxy+keepalived
172.16.0.6 etcd2+haproxy+keepalived
172.16.0.7 etcd3+haproxy+keepalived
node节点
172.16.0.8 node1
172.16.0.9 node2
kube-apiserver 6443
kube-controller-manager 10252
kbe-scheduler 10251
kubelet 10250
0.ansible批量管理工具安装
测试环境只在master1服务端安装ansible,用于统一管理服务器
yum -y install ansible
1.chrony 时间服务器
https://github.com/gjmzj/kubeasz/blob/master/docs/guide/chrony.md
在安装k8s集群前需确保各节点时间同步,不然程序启动后会报时间不同步;chrony 性能比ntp好且配置管理方便;它既可作时间服务器服务端,也可作客户端
所有服务器端安装以下时间同步工具
yum -y install chrony
服务端配置
vim /etc/chrony.conf
配置时间源,国内可以增加阿里的时间源 ntp1.aliyun.com
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server 172.16.0.2 iburst
配置允许同步的客户端网段
allow 172.16.0.0/24
配置离线也能作为源服务器
local stratum 10
客户端配置
vim /etc/chrony.conf
配置时间源,国内可以增加阿里的时间源 chrony服务器ip地址,其他server删除,其他配置保持不变
server 172.16.0.2 iburst
所有机器启动服务
systemctl enable chronyd
systemctl start chronyd
1.1系统参数优化
优化内容包括,在此处不想多说,百度吧:
关闭不必要服务
系统参数优化
帐号登录安全优化
系统软件升级到最新版本,保持系统干净
2.cfssl搭建本地CA服务器
0.下载安装cfssl的二进制工具即可,其实就是三个命令
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfsslinfo
yum方式安装cfssl工具
[root@master1 ~]#yum -y install cfssl
生成默认配置文件文件,csr和config
[root@master1 ~]# cfssl print-defaults list
Default configurations are available for:
csr
config
1.创建TLS证书和秘钥
这一步是在安装配置kubernetes的所有步骤中最容易出错也最难于排查问题的一步,而这却刚好是第一步,也是最重要一步
kubernetes 系统的各组件需要使用TLS 证书对通信进行加密,本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 和其它证书;
生成的 CA 证书和秘钥文件如下:
ca-key.pem
ca.pem
kubernetes-key.pem
kubernetes.pem
kube-proxy.pem
kube-proxy-key.pem
admin.pem
admin-key.pem
etcd.pem
etcd-key.pem
使用证书的组件如下:
etd:使用 ca.pem、etcd-key.pem、etcd.pem
kube-apiserver:使用 ca.pem、kubernetes-key.pem、kubernetes.pem
kubelet:使用 ca.pem
kube-proxy:使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem
kubectl:使用 ca.pem、admin-key.pem、admin.pem
kube-controller-manager:使用 ca-key.pem、ca.pem
注意: 以下操作都在 172.16.0.2 master1主机上执行,然后分发到集群所有主机,证书只需要创建一次即可,以后在向集群中添加新节点时只要将 /etc/kubernetes/pki/ 目录下的证书拷贝到新节点上即可。
2.创建ca证书签名请求
cfssl print-defaluts ca> ca.-csr.json 生成一个默认csr创建ca证书签名请求文件
!vim ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
生成CA认证服务器自身的证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls
ca.pem ca-key.pem ca.csr 生成三个文件
配置证书生成策略,让ca知道颁发server client peer类型证书
cfssl print-defaluts config> ca-config.json 生成模板文件
vim ca-config.json
{
"signing": {
"default": {
"expiry": "168h"
},
"profiles": {
"server": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
},
"peer": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth"
"client auth"
]
}
}
}
}
生成etcd的csr配置文件,专门给etcd服务使用
vim etcd-csr.json
{
"CN": "etcd",
"hosts": [
"172.16.0.5", etcd服务器ip地址
"172.16.0.6", etcd服务器ip地址
"172.16.0.7", etcd服务器ip地址
"127.0.0.1" 最好把此ip也加上,不然在后面启动Etcd会时报错
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
etcd生成对等证书"peer",在profile参数中指定
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-csr.json | cfssljson -bare etcd
把生成的所有pem文件拷贝到所有etcd服务器的/etc/kubernetes/pki目录下面
生成kubernetes证书
vim kubernetes-csr.json
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"172.16.0.2",
"172.16.0.3",
"172.16.0.4",
"172.16.0.5",
"172.16.0.6",
"172.16.0.7",
"172.16.0.8",
"172.16.0.9",
"172.16.0.100",
"169.169.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "XS",
"O": "k8s",
"OU": "System"
}
]
}
如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表,由于该证书后续被 etcd 集群和 kubernetes master 集群使用,
所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP,如 169.169.0.1)。
生成kubernetes证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer kubernetes-csr.json | cfssljson -bare kubernetes
创建 admin 证书签名请求文件 admin-csr.json:kubectl的权限是admin,具有访问k8s所有api的权限
cat > admin-csr.json << EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
生成admin证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json|cfssljson -bare admin
创建 kube-proxy 证书签名请求文件 kube-proxy-csr.json:
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成kube-proxy证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
校验证书
openssl方式
openssl x509 -noout -text -in kubernetes.pem
cfssl-certinfo -cert kubernetes.pem
分发证书
将生成的证书和秘钥文件(后缀名为.pem)拷贝到所有机器的 /etc/kubernetes/ssl 目录下备用;
scp *.pem root@172.16.0.3-9:/etc/kubernetes/pki/
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------到此CA配置完毕
此文件在apiserver启动时需要调用
创建 TLS Bootstrapping Token
Token可以是任意的包含128 bit的字符串,可以使用安全的随机数发生器生成。
export BOOTSTRAP_TOKEN={BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
注意: 请检查 token.csv 文件,确认其中的 {KUBE_APISERVER}
--kubeconfig=bootstrap.kubeconfig
设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap
--token=${BOOTSTRAP_TOKEN}
--kubeconfig=bootstrap.kubeconfig
设置上下文参数
kubectl config set-context default
--cluster=kubernetes
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
--embed-certs 为 true 时表示将 certificate-authority 证书写入到生成的 bootstrap.kubeconfig 文件中;
设置客户端认证参数时没有指定秘钥和证书,后续由 kube-apiserver 自动生成;
3.安装etcd集群
安装etcd
yum -y install etcd
用yum安装的etcd主要有二个配置文件
/etc/etcd/etcd.conf 参数配置文件
/usr/lib/systemd/system/etcd.service 启动配置文件
vim /etc/etcd/etcd.conf 修改以下文件
ETCD_DATA_DIR=" var="" lib="" etcd="" etcd1.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.0.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379,https://172.16.0.5:2379"
ETCD_NAME="etcd1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.0.5:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.0.5:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://172.16.0.5:2380,etcd2=https://172.16.0.6:2380,etcd3=https://172.16.0.7:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_CERT_FILE="/etc/kubernetes/pki/etcd.pem"
ETCD_KEY_FILE="/etc/kubernetes/pki/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/kubernetes/pki/ca.pem"
ETCD_PEER_CERT_FILE="/etc/kubernetes/pki/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/kubernetes/pki/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/pki/ca.pem"
etcd用yum安装的配置文件/etc/etcd/etcd.conf后不能直接配置好后就启动,必须要在etcd启动脚本etcd.service中引用/etc/etcd/etcd.conf里面的变量名后再启动才可以,例如上面为例
vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
set GOMAXPROCS to number of processors
ExecStart=/usr/bin/etcd
--name={ETCD_DATA_DIR}
--listen-client-urls={ETCD_CERT_FILE}
--key-file={ETCD_PEER_CERT_FILE}
--peer-key-file={ETCD_TRUSTED_CA_FILE}
--peer-trusted-ca-file={ETCD_INITIAL_ADVERTISE_PEER_URLS}
--listen-peer-urls={ETCD_LISTEN_CLIENT_URLS}
--advertise-client-urls={ETCD_INITIAL_CLUSTER_TOKEN}
--initial-cluster={ETCD_INITIAL_CLUSTER_STATE}
Restart=on-failure
LimitNOFILE=65536
[Install]
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
以上配置没有问题后启动程序
systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd
查看启用了ca认证的etcd集群状态
etcdctl --ca-file=/etc/kubernetes/pki/ca.pem --cert-file=/etc/kubernetes/pki/kubernetes.pem --key-file=/etc/kubernetes/pki/kubernetes-key.pem --endpoints=https://172.16.0.5:2379,https://172.16.0.6:2379,https://172.16.0.7:2379 cluster-health
出现以下为正常
member 19fb9c98c9807abe is healthy: got healthy result from https://172.16.0.6:2379
member 7b75c34d64762dc5 is healthy: got healthy result from https://172.16.0.7:2379
member a5ff6d2872667bd3 is healthy: got healthy result from https://172.16.0.5:2379
cluster is healthy
etcdctl --ca-file=/etc/kubernetes/pki/ca.pem --cert-file=/etc/kubernetes/pki/kubernetes.pem --key-file=/etc/kubernetes/pki/kubernetes-key.pem --endpoints=https://172.16.0.5:2379,https://172.16.0.6:2379,https://172.16.0.7:2379 member list
出现以下为正常
19fb9c98c9807abe: name=etcd2 peerURLs=https://172.16.0.6:2380 clientURLs=https://127.0.0.1:2379,https://172.16.0.6:2379 isLeader=true
7b75c34d64762dc5: name=etcd3 peerURLs=https://172.16.0.7:2380 clientURLs=https://127.0.0.1:2379,https://172.16.0.7:2379 isLeader=false
a5ff6d2872667bd3: name=etcd1 peerURLs=https://172.16.0.5:2380 clientURLs=https://127.0.0.1:2379,https://172.16.0.5:2379 isLeader=false
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------etcd集群配置完毕
4.haproxy+keepalived
部署 haproxy+keepalived
本次部署一个三节点高可用 haproxy+keepalived 集群
分别为:
172.16.0.2 172.16.0.3 172.16.0.4 vip:172.16.0.100
安装 haproxy+keepalived
yum install -y haproxy keepalived
注: 3台 haproxy+keepalived 节点都需安装
配置 keepalived
节点1 172.16.0.2 配置文件 vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
test@sina.com
}
notification_email_from admin@test.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_MASTER
}
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 3
}
vrrp_instance VI_1 {
state MASTER # 如果配置主从,从服务器改为BACKUP即可
interface eth0 #网卡为真实网卡,不然启动报错
virtual_router_id 60
priority 100 # 从服务器设置小于100的数即可
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.0.100/24
}
track_script {
check_haproxy
}
}
节点2 172.16.0.3 配置文件 vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
test@sina.com
}
notification_email_from admin@test.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_MASTER
}
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 3
}
vrrp_instance VI_1 {
state BACKUP # 如果配置主从,从服务器改为BACKUP即可
interface eth0 #网卡为真实网卡,不然启动报错
virtual_router_id 60
priority 90 # 从服务器设置小于100的数即可
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.0.100/24
}
track_script {
check_haproxy
}
}
节点3 172.16.0.3配置文件 vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
test@sina.com
}
notification_email_from admin@test.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_MASTER
}
vrrp_script check_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 3
}
vrrp_instance VI_1 {
state BACKUP # 如果配置主从,从服务器改为BACKUP即可
interface eth0 #网卡为真实网卡,不然启动报错
virtual_router_id 60
priority 80 # 从服务器设置小于100的数即可
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.0.3/24
}
track_script {
check_haproxy
}
}
检测脚本 vi /etc/keepalived/check_haproxy.sh!/bin/bash
flag=?)
if [[ $flag != 0 ]];then
echo "haproxy is down,close the keepalived"
systemctl stop keepalived
fi
修改keepalived启动文件 vi /usr/lib/systemd/system/keepalived.service 以下部分:
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target haproxy.service
Requires=haproxy.service
keepalived配置文件三台主机基本一样,除了state,主节点配置为MASTER,备节点配置BACKUP,优化级参数priority,主节点设置最高,备节点依次递减
自定义的检测脚本作用是检测本机haproxy服务状态,如果不正常就停止本机keepalived,释放VIP
这里没有考虑keepalived脑裂的问题,后期可以在脚本中加入相关检测
配置 haproxy
3台节点配置一模一样 配置文件 vim /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode tcp
log global
option tcplog
option dontlognull
option redispatch
retries 3
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
maxconn 3000
listen stats
mode http
bind :10086 #http 状态检查端口 http://ip:10086:/admin?stats
stats enable
stats uri /admin?stats
stats auth admin:admin
stats admin if TRUE
frontend k8s_http *:8080
mode tcp
maxconn 2000
default_backend http_sri
backend http_sri
balance roundrobin
server s1 172.16.0.2:8080 check inter 10000 fall 2 rise 2 weight 1
server s2 172.16.0.3:8080 check inter 10000 fall 2 rise 2 weight 1
server s3 172.16.0.4:8080 check inter 10000 fall 2 rise 2 weight 1
frontend k8s_https *:6443
mode tcp
maxconn 2000
default_backend https_sri
backend https_sri
balance roundrobin
server s1 172.16.0.2:6443 check inter 10000 fall 2 rise 2 weight 1
server s2 172.16.0.3:6443 check inter 10000 fall 2 rise 2 weight 1
server s3 172.16.0.4:6443 check inter 10000 fall 2 rise 2 weight 1
listen stats定义了haproxy自身状态查看地址,在里面可以看到haproy目前的各种状态
frontend 定义了前端提供服务的端口等信息
backend 定义了后端真实服务器的信息
启动 haproxy+keepalived
3个节点都启动
systemctl daemon-reload
systemctl enable haproxy
systemctl enable keepalived
systemctl start haproxy
systemctl start keepalived
如果没有什么报错,那应该就可以在主节点 192.168.223.201 上面看到ens33网卡已绑定VIP: 192.168.223.200
启动成功后到三台服务器上执行以下命令,查看是否有vip.然后,停掉有vip的haproxy,看下vip会不会飘移到别的机器,如果有说明搭建成功
ip a
1: lo:mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33:mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:2b:74:46 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.10/24 brd 192.168.223.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.223.200/24 scope global secondary ens33
valid_lft forever preferred_lft forever
inet6 fe80::435e:5e98:6d14:6c40/64 scope link
valid_lft forever preferred_lft forever
-----------------------------------------------------------------------------------------------------------------------------------------------------到此haproxy+keepalived 配置完毕