手动搭建k8s-1.16.6高可用集群之部署worker节点-apiserver高可用

本文档讲解使用 nginx 4 层透明代理功能实现 Kubernetes worker 节点组件高可用访问 kube-apiserver 集群的步骤。

注意：如果没有特殊指明，本文档的所有操作均在 k8s-01 节点上执行。

一、基于 nginx 代理的 kube-apiserver 高可用方案

• 控制节点的 kube-controller-manager、kube-scheduler 是多实例部署且连接本机的 kube-apiserver，所以只要有一个实例正常，就可以保证高可用；
• 集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；
• 在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；
• kubelet、kube-proxy 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；

二、下载和编译nginx

下载源码：

cd /opt/k8s/work
wget http://nginx.org/download/nginx-1.15.3.tar.gz
tar -zxf nginx-1.15.3.tar.gz

配置编译参数：

cd /opt/k8s/work/nginx-1.15.3
mkdir nginx-prefix
yum install -y gcc make
./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module

• --with-stream：开启 4 层透明转发(TCP Proxy)功能；
• --without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；

编译和安装：

cd /opt/k8s/work/nginx-1.15.3
make && make install

三、验证编译的nginx

cd /opt/k8s/work/nginx-1.15.3
./nginx-prefix/sbin/nginx -v

输出：

nginx version: nginx/1.15.3

四、安装和部署nginx

创建目录结构：

cat > deploy.sh << "EOF"
#!/bin/bash
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh 
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
done
EOF

拷贝二进制程序：

cat > deploy.sh << "EOF"
#!/bin/bash
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh 
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"
    scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx
    ssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"
done
EOF

• 重命名二进制文件为 kube-nginx；

配置 nginx，开启 4 层透明转发功能：

cd /opt/k8s/work

cat > kube-nginx.conf << \EOF
worker_processes 1;

events {
    worker_connections  1024;
}

stream {
    upstream backend {
        hash $remote_addr consistent;
        server 192.168.0.71:6443        max_fails=3 fail_timeout=30s;
        server 192.168.0.72:6443        max_fails=3 fail_timeout=30s;
        server 192.168.0.73:6443        max_fails=3 fail_timeout=30s;
    }

    server {
        listen 127.0.0.1:8443;
        proxy_connect_timeout 1s;
        proxy_pass backend;
    }
}
EOF

• upstream backend 中的 server 列表为集群中各 kube-apiserver 的节点 IP，需要根据实际情况修改；

分发配置文件：

cat > deploy.sh << "EOF"
#!/bin/bash
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh 
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp kube-nginx.conf  root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf
done
EOF

五、配置 systemd unit 文件，启动服务

配置 kube-nginx systemd unit 文件：

cd /opt/k8s/work

cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

分发 systemd unit 文件：

cat > deploy.sh << "EOF"
#!/bin/bash
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh 
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp kube-nginx.service  root@${node_ip}:/etc/systemd/system/
done
EOF

启动 kube-nginx 服务：

cat > deploy.sh << "EOF"
#!/bin/bash
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh 
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"
done
EOF

六、检查 kube-nginx 服务运行状态

cat > deploy.sh << "EOF"
#!/bin/bash
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh 
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"
done
EOF

输出内容如下：

$ ./deploy.sh 
>>> 192.168.0.71
   Active: active (running) since Fri 2020-04-24 15:19:31 CST; 1min 9s ago
>>> 192.168.0.72
   Active: active (running) since Fri 2020-04-24 15:19:31 CST; 1min 9s ago
>>> 192.168.0.73
   Active: active (running) since Fri 2020-04-24 15:19:31 CST; 1min 9s ago

确保状态为 active (running)，否则通过如下命令查看日志，确认原因：

journalctl -u kube-nginx