Kubernetes(k8s)1.14 离线版集群 - 高可用kube-apiserver+部署KeepLived

声明：
如果您有更好的技术与作者分享，或者商业合作；
请访问作者个人网站 http://www.esqabc.com/view/message.html 留言给作者。
如果该案例触犯您的专利，请在这里：http://www.esqabc.com/view/message.html 留言给作者说明原由
作者一经查实，马上删除。

.
.

1、搭建前说明：

.
a、前提提条件、服务器，请查看这个地址：https://blog.csdn.net/esqabc/article/details/102726771
.

.

2、搭建kube-apiserver 高可用

• 使用Nginx 4层实现k8s节点(master节点和worker节点)高可用，访问kube-apiserver的步骤
• 注意：搭建服务器，没有特殊说明，一般默认在：k8s-01 操作

a、下载编译nginx
（1）把下载好的文件上传到：/opt/k8s/work

[root@k8s-01 ~]# cd /opt/k8s/work
解压
[root@k8s-01 work]# tar -xzvf nginx-1.15.3.tar.gz
编译
[root@k8s-01 work]# cd /opt/k8s/work/nginx-1.15.3
[root@k8s-01 nginx-1.15.3]# mkdir nginx-prefix
.
注意，下面命令有一个点的
.
[root@k8s-01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module
[root@k8s-01 nginx-1.15.3]# make && make install

说明一下：

• without-http_scgi_module --without-http_fastcgi_module
• with-stream：开启 4 层透明转发(TCP Proxy)功能；
• without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；

（2）创建目录结构

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}
  done

（3）分发到其他主机

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${NODE_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    sleep 3
  done

注意：如果出现下面错误，执行上面命令重复执行一次就OK：

出现下图就代表成功：

（4）配置Nginx文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > kube-nginx.conf < 添加下面内容：

worker_processes 1;
events {
    worker_connections  1024;
}
stream {
    upstream backend {
        hash $remote_addr consistent;
        server 172.26.16.249:6443        max_fails=3 fail_timeout=30s;
        server 172.26.16.250:6443        max_fails=3 fail_timeout=30s;
        server 172.26.16.251:6443        max_fails=3 fail_timeout=30s;
    }
    server {
        listen *:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

注意：只需要修改server 内容即可
.
（5）分发配置文件

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
  done

（6）配置Nginx启动文件

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > kube-nginx.service < 添加下面内容：

[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

（7）分发nginx启动文件

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
  done

（8）启动 kube-nginx 服务

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl start kube-nginx"
  done

（9）检查 kube-nginx 服务运行状态

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in ${MASTER_IPS[@]}
  do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
  done

出现下图，就说明正常

3、KeepLived 部署

在所有master节点安装keeplived
a、安装keepalived

[root@k8s-01 ~]# yum install -y keepalived

b、配置keeplive服务

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# cat > /etc/keepalived/keepalived.conf < 添加下面内容

! Configuration File for keepalived
global_defs {
   router_id 172.26.16.249
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 172.26.16.249
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        172.26.16.252
    }
}
EOF	

注意：## 172.26.16.249 为当前节点，172.26.16.253为node节点
.
c、将配置拷贝到其他节点，并替换相关IP

[root@k8s-01 ~]# cd /opt/k8s/work

for node_ip in 172.26.16.249 172.26.16.250 172.26.16.251
  do
    echo ">>> ${node_ip}"
    scp  /etc/keepalived/keepalived.conf $node_ip:/etc/keepalived/keepalived.conf
  done

d、替换IP

	ssh root@172.26.16.250 sed -i 's#172.26.16.249#172.26.16.250#g'  /etc/keepalived/keepalived.conf

	ssh root@172.26.16.251 sed -i 's#172.26.16.249#172.26.16.251#g'  /etc/keepalived/keepalived.conf

注意：不需修改172.26.16.249，只需修改其他IP即可

e、创建健康检查脚本

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# vi /opt/check_port.sh
添加下面内容：

CHK_PORT=$1
 if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                exit 1
        fi
 else
        echo "Check Port Cant Be Empty!"
 fi

f、启动keeplived

[root@k8s-01 ~]# cd /opt/k8s/work

for NODE in k8s-01 k8s-02 k8s-03; do
    echo "--- $NODE ---"
    scp -r /opt/check_port.sh  $NODE:/etc/keepalived/
    ssh $NODE 'systemctl enable --now keepalived'
done

g、查看是否成功

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 ~]# ping 172.26.16.252

h、检查是否启动成功（分别在其他服务器执行）

[root@k8s-01 ~]# cd /opt/k8s/work
[root@k8s-01 work]# ps -ef|grep keep

如果没有启动，请执行下面的命令：
[root@k8s-01 ~]# systemctl start keepalived