公网部署k8s集群(wireguard将公网IP映射为内网IP)

问题：由于公网ip不是绑定在物理机上的，因此通过一般的部署方式总会有问题
解决方案：使用wireguard将公网ip映射成内网ip
注意：主节点配置一定要是最好的，因为上面会运行整个集群的控制面板；

1.通用基础配置（全部节点均需执行）

记得腾讯云控制台那边开放所有端口

1.1.开启IP地址转发

vim /etc/sysctl.conf
# 添加
net.ipv4.ip_forward = 1
net.ipv4.conf.all.proxy_arp = 1

# 更新
sysctl -p /etc/sysctl.conf

#添加iptables规则
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#wg0为wireguard自动生成的虚拟网卡，我们不需要手动设置
iptables -A FORWARD -i wg0 -o wg0 -m conntrack --ctstate NEW -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

1.2.升级linux内核（具体操作可参看官网）

看自己操作系统对应的下载方式
Wireguard官网
以下为CentOS7.6操作
将使用wireguard进行公网的内网映射，需要内核版本为5.15及以上

# 下载内核
# 载入公钥
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

# 升级elrepo
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-5.el7.elrepo.noarch.rpm

# 载入 elrepo-kernel 数据
yum --disablerepo=\* --enablerepo=elrepo-kernel repolist
# 安装最新版本的内核
yum --disablerepo=\* --enablerepo=elrepo-kernel install  kernel-ml.x86_64  -y
# 删除旧版本工具包
yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64  -y

# 修改系统使用的内核
# 查看当前实际启动顺序
grub2-editenv list
# 查看内核插入顺序
grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2

# 设置默认启动内核
grub2-set-default 'CentOS Linux (5.15.2-1.el7.elrepo.x86_64) 7 (Core)'
# 重新创建内核配置
grub2-mkconfig -o /boot/grub2/grub.cfg
# 重启服务器
reboot
# 验证当前内核版本
uname -r 

1.3.使用wireguard

wireguard安装官网很全

# 安装wireguard
sudo yum install yum-utils epel-release
sudo yum install kernel-plus wireguard-tools
sudo yum install elrepo-release epel-release
sudo yum install kmod-wireguard wireguard-tools

# 获取每台机器的公钥私钥，用于通信
# 将其记录下来
wg genkey | tee privatekey | wg pubkey > publickey

1.3.1.在指定位置书写指定的规则

解释：规则其实也是比较好理解，首先，确定本机的内网ip，
设置其他机器的内网ip以及对应的外网ip，这样发送数据包给相应内网ip时网卡就将该包转发给设置的外网ip；
私钥公钥用于数据加密安全；

# master节点上(192.168.1.1)
#　vim /etc/wireguard/wg0.conf
# 之后wireguard会根据该文件名自动生成wg0虚拟网卡
# 以下示例文件为master节点上的的参考；
[Interface]
ListenPort = 54180
PrivateKey = eChFNxb2E7m9a2acpuBFtIEkLReDHVko/RpCtJxoUkA=
Address = 10.0.0.1

[Peer]
PublicKey = BhWyKeMLYFjytq5uCQOb2VEuFVZ6p9vAol5pGg4liDc=
AllowedIPs = 10.0.0.2/32
Endpoint = 81.68.187.197:54180

[Peer]
PublicKey = w7fSqk5CRBlcDWhFJjbzSzOXXMMJ1x1AmTlWmZZhpWM=
AllowedIPs = 10.0.0.3/32
Endpoint = 1.116.38.204:54180

[Peer]
PublicKey = oNcsv4uZ5U4xQZhDCx0QOob9Ao5CikbMM++ktbWvBi0=
AllowedIPs = 10.0.0.4/32
Endpoint = 81.68.248.160:54180

[Peer]
PublicKey = 9RcHYGj4huZAKMpQSkMKn3iIYqKmbiC/lw+dinr03mM=
AllowedIPs = 10.0.0.5/32
Endpoint = 121.5.58.90:54180  

# liyuan-node2节点上(192.168.1.1)
#　vim /etc/wireguard/wg0.conf
# 之后wireguard会根据该文件名自动生成wg0虚拟网卡
#
[Interface]
PrivateKey = OMq+uga9k7XL5a31k6ahzd5SwbKNT/4B9Pqojddwc14=
Address = 192.168.1.2
ListenPort = 5418

[Peer]
PublicKey = 4yOc1xeA8fcP9xfYwpQZ4WGEUmu15vQwKB4laFYUwxg=
EndPoint = 81.68.209.55:5418
AllowedIPs = 192.168.1.1/32

1.3.2.执行这些规则

1.3.3.启动wireguard，自动化以及检查

ip link add wg0 type wireguard
wg-quick up wg0
ip link set wg0 up
systemctl enable wg-quick@wg0
# 配置热重载
wg syncconf wg0 <(wg-quick strip wg0)

可以看到本机10.0.0.1可以ping通过
10.0.0.2
10.0.0.3
10.0.0.4
记住在腾讯云那边要放行相应端口

2.k8s

2.1主节点安装k8s

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 添加转发规则
vim /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward=1

#加载这条规则
sysctl -p /etc/sysctl.d/k8s.conf

#添加docker
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum list docker-ce --showduplicates
# (1)安装docker
yum install docker-ce-3:20.10.12-3.el8
# 设置docker下载源
vim /etc/docker/daemon.json
{
        "registry-mirrors":["https://kn0t2bca.mirror.aliyuncs.com"]
}
# 启动docker服务
systemctl start docker
systemctl enable docker

#(2)添加kubeadm 源
vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

# (3)安装kubeadm
yum install --setopt=obsoletes=0 kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0 -y

#(4)设置开机自启动
systemctl enable kubeadm

#(5)下载k8s所需组件
imageList=(
        kube-apiserver:v1.17.4
        kube-controller-manager:v1.17.4
        kube-scheduler:v1.17.4
        kube-proxy:v1.17.4
        pause:3.1
        etcd:3.4.3-0
        coredns:1.6.5
)

vim install-k8s.sh

for image in ${imageList[@]}
do
        docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$image
        docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$image k8s.gcr.io/$image
        docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$image
done

chmod +x install-k8s.sh
#运行该脚本
./install-k8s.sh

# 初始化kubeadm
kubeadm init --kubernetes-version=v1.17.4 --service-cidr=2.1.0.0/16 --pod-network-cidr=2.244.0.0/16 --ignore-preflight-errors=all --apiserver-advertise-address=10.0.0.1 --v=10 --image-repository="registry.aliyuncs.com/google_containers"

#修改internal-ip为wg0网卡的ip
vim /var/lib/kubelet/kubeadm-flags.env

systemctl daemond-reload
systemctl restart kubelet

2.2.从节点接入主节点k8s

# 添加docker安装源
(1)wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
 
yum list docker-ce

# 安装docker
(2)yum install docker-ce-3:20.10.12-3.el8

# 书写docker配置文件，设置其下载源
(3)vim /etc/docker/daemon.json

{
        "registry-mirrors":["https://kn0t2bca.mirror.aliyuncs.com"]
}

(4)systemctl restart docker
(5)systemctl enable docker

# 添加k8s软件源
(1)vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

(2)yum install kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0 -y
#配置路由转发规则
(3)echo 1 > /proc/sys/net/ipv4/ip_forward
modprobe br_netfilter

kubeadm join 10.0.0.1:6443 --token tkrdse.w6jdluo21cjvag11 \
    --discovery-token-ca-cert-hash sha256:cc7d05228add3cdeea001b57dfdec9abfb556bbab73a2a50404a1eb54a97a246

如果没看到上述的这一段代码，
参考 从节点加入k8s集群

# 将主节点的配置文件拷贝到从节点相同位置
scp [email protected]:/etc/kubernetes/admin.conf /etc/kubernetes/admin.conf

vim /etc/profile
添加 export KUBECONFIG=/etc/kubernetes/admin.conf
source /etc/profile


#修改internal-ip为wg0网卡的ip
vim /var/lib/kubelet/kubeadm-flags.env

添加 --node-ip = 10.0.0.2

kubectl taint node liyuan-master node-role.kubernetes.io/master:NoSchedule-

3.k3s

3.1.主节点接入k3s

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh  -s -  --node-external-ip 主节点公网IP --advertise-address 主节点公网IP --node-ip 主节点内网IP（上述设置的为10.0.0.1） --flannel-iface wg0 (指定网卡)

参数	解释
–node-external-ip	节点外网IP
–advertise-address	节点间通信的ip
–node-ip	节点内网IP
–flannel-iface	指定通信时使用的网卡

3.2从节点接入k3s

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL=https://10.0.0.1:6443（主节点ip） K3S_TOKEN= (见主节点下 /var/lib/rancher/k3s/server/token) sh -s - --node-external-ip 从节点公网IP --node-ip 从节点内网IP，此处为10.0.0.2 --flannel-iface wg0 

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL=https://10.0.0.1:6443 K3S_TOKEN=K10b99b088c197761285a9112f6f9e51faef41e92d3a86d3d668edfbb284bdc71a4::server:b2aee3028f6c8a2077265b773951f968 sh -s - --node-external-ip 1.116.38.204 --node-ip 10.0.0.3 --flannel-iface wg0