公网部署k8s集群(wireguard将公网IP映射为内网IP)

问题:由于公网ip不是绑定在物理机上的,因此通过一般的部署方式总会有问题
解决方案:使用wireguard将公网ip映射成内网ip
注意:主节点配置一定要是最好的,因为上面会运行整个集群的控制面板;

1.通用基础配置(全部节点均需执行)

记得腾讯云控制台那边开放所有端口

1.1.开启IP地址转发

vim /etc/sysctl.conf
# 添加
net.ipv4.ip_forward = 1
net.ipv4.conf.all.proxy_arp = 1

# 更新
sysctl -p /etc/sysctl.conf

#添加iptables规则
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#wg0为wireguard自动生成的虚拟网卡,我们不需要手动设置
iptables -A FORWARD -i wg0 -o wg0 -m conntrack --ctstate NEW -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

1.2.升级linux内核(具体操作可参看官网)

看自己操作系统对应的下载方式
Wireguard官网
以下为CentOS7.6操作
将使用wireguard进行公网的内网映射,需要内核版本为5.15及以上

# 下载内核
# 载入公钥
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

# 升级elrepo
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-5.el7.elrepo.noarch.rpm

# 载入 elrepo-kernel 数据
yum --disablerepo=\* --enablerepo=elrepo-kernel repolist
# 安装最新版本的内核
yum --disablerepo=\* --enablerepo=elrepo-kernel install  kernel-ml.x86_64  -y
# 删除旧版本工具包
yum remove kernel-tools-libs.x86_64 kernel-tools.x86_64  -y

# 修改系统使用的内核
# 查看当前实际启动顺序
grub2-editenv list
# 查看内核插入顺序
grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'" -f2

在这里插入图片描述

# 设置默认启动内核
grub2-set-default 'CentOS Linux (5.15.2-1.el7.elrepo.x86_64) 7 (Core)'
# 重新创建内核配置
grub2-mkconfig -o /boot/grub2/grub.cfg
# 重启服务器
reboot
# 验证当前内核版本
uname -r 

在这里插入图片描述

1.3.使用wireguard

wireguard安装官网很全

# 安装wireguard
sudo yum install yum-utils epel-release
sudo yum install kernel-plus wireguard-tools
sudo yum install elrepo-release epel-release
sudo yum install kmod-wireguard wireguard-tools
# 获取每台机器的公钥私钥,用于通信
# 将其记录下来
wg genkey | tee privatekey | wg pubkey > publickey

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第1张图片

1.3.1.在指定位置书写指定的规则

解释:规则其实也是比较好理解,首先,确定本机的内网ip,
设置其他机器的内网ip以及对应的外网ip,这样发送数据包给相应内网ip时网卡就将该包转发给设置的外网ip;
私钥公钥用于数据加密安全;

# master节点上(192.168.1.1)
# vim /etc/wireguard/wg0.conf
# 之后wireguard会根据该文件名自动生成wg0虚拟网卡
# 以下示例文件为master节点上的的参考;
[Interface]
ListenPort = 54180
PrivateKey = eChFNxb2E7m9a2acpuBFtIEkLReDHVko/RpCtJxoUkA=
Address = 10.0.0.1

[Peer]
PublicKey = BhWyKeMLYFjytq5uCQOb2VEuFVZ6p9vAol5pGg4liDc=
AllowedIPs = 10.0.0.2/32
Endpoint = 81.68.187.197:54180

[Peer]
PublicKey = w7fSqk5CRBlcDWhFJjbzSzOXXMMJ1x1AmTlWmZZhpWM=
AllowedIPs = 10.0.0.3/32
Endpoint = 1.116.38.204:54180

[Peer]
PublicKey = oNcsv4uZ5U4xQZhDCx0QOob9Ao5CikbMM++ktbWvBi0=
AllowedIPs = 10.0.0.4/32
Endpoint = 81.68.248.160:54180

[Peer]
PublicKey = 9RcHYGj4huZAKMpQSkMKn3iIYqKmbiC/lw+dinr03mM=
AllowedIPs = 10.0.0.5/32
Endpoint = 121.5.58.90:54180  
# liyuan-node2节点上(192.168.1.1)
# vim /etc/wireguard/wg0.conf
# 之后wireguard会根据该文件名自动生成wg0虚拟网卡
#
[Interface]
PrivateKey = OMq+uga9k7XL5a31k6ahzd5SwbKNT/4B9Pqojddwc14=
Address = 192.168.1.2
ListenPort = 5418

[Peer]
PublicKey = 4yOc1xeA8fcP9xfYwpQZ4WGEUmu15vQwKB4laFYUwxg=
EndPoint = 81.68.209.55:5418
AllowedIPs = 192.168.1.1/32

1.3.2.执行这些规则

1.3.3.启动wireguard,自动化以及检查

ip link add wg0 type wireguard
wg-quick up wg0
ip link set wg0 up
systemctl enable wg-quick@wg0
# 配置热重载
wg syncconf wg0 <(wg-quick strip wg0)

可以看到本机10.0.0.1可以ping通过
10.0.0.2
10.0.0.3
10.0.0.4
记住在腾讯云那边要放行相应端口

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第2张图片

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第3张图片

2.k8s

2.1主节点安装k8s

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
# 添加转发规则
vim /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward=1

#加载这条规则
sysctl -p /etc/sysctl.d/k8s.conf
#添加docker
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum list docker-ce --showduplicates
# (1)安装docker
yum install docker-ce-3:20.10.12-3.el8
# 设置docker下载源
vim /etc/docker/daemon.json
{
        "registry-mirrors":["https://kn0t2bca.mirror.aliyuncs.com"]
}
# 启动docker服务
systemctl start docker
systemctl enable docker

#(2)添加kubeadm 源
vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

# (3)安装kubeadm
yum install --setopt=obsoletes=0 kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0 -y

#(4)设置开机自启动
systemctl enable kubeadm

#(5)下载k8s所需组件
imageList=(
        kube-apiserver:v1.17.4
        kube-controller-manager:v1.17.4
        kube-scheduler:v1.17.4
        kube-proxy:v1.17.4
        pause:3.1
        etcd:3.4.3-0
        coredns:1.6.5
)

vim install-k8s.sh

for image in ${imageList[@]}
do
        docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$image
        docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$image k8s.gcr.io/$image
        docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$image
done

chmod +x install-k8s.sh
#运行该脚本
./install-k8s.sh

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第4张图片

# 初始化kubeadm
kubeadm init --kubernetes-version=v1.17.4 --service-cidr=2.1.0.0/16 --pod-network-cidr=2.244.0.0/16 --ignore-preflight-errors=all --apiserver-advertise-address=10.0.0.1 --v=10 --image-repository="registry.aliyuncs.com/google_containers"

#修改internal-ip为wg0网卡的ip
vim /var/lib/kubelet/kubeadm-flags.env

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第5张图片

systemctl daemond-reload
systemctl restart kubelet

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第6张图片

2.2.从节点接入主节点k8s

公网部署k8s集群(wireguard将公网IP映射为内网IP)_第7张图片

# 添加docker安装源
(1)wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
 
yum list docker-ce

# 安装docker
(2)yum install docker-ce-3:20.10.12-3.el8

# 书写docker配置文件,设置其下载源
(3)vim /etc/docker/daemon.json

{
        "registry-mirrors":["https://kn0t2bca.mirror.aliyuncs.com"]
}

(4)systemctl restart docker
(5)systemctl enable docker
# 添加k8s软件源
(1)vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

(2)yum install kubeadm-1.17.4-0 kubelet-1.17.4-0 kubectl-1.17.4-0 -y
#配置路由转发规则
(3)echo 1 > /proc/sys/net/ipv4/ip_forward
modprobe br_netfilter
kubeadm join 10.0.0.1:6443 --token tkrdse.w6jdluo21cjvag11 \
    --discovery-token-ca-cert-hash sha256:cc7d05228add3cdeea001b57dfdec9abfb556bbab73a2a50404a1eb54a97a246

如果没看到上述的这一段代码,
参考 从节点加入k8s集群

# 将主节点的配置文件拷贝到从节点相同位置
scp [email protected]:/etc/kubernetes/admin.conf /etc/kubernetes/admin.conf

vim /etc/profile
添加 export KUBECONFIG=/etc/kubernetes/admin.conf
source /etc/profile


#修改internal-ip为wg0网卡的ip
vim /var/lib/kubelet/kubeadm-flags.env

添加 --node-ip = 10.0.0.2
kubectl taint node liyuan-master node-role.kubernetes.io/master:NoSchedule-

3.k3s

3.1.主节点接入k3s

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh  -s -  --node-external-ip 主节点公网IP --advertise-address 主节点公网IP --node-ip 主节点内网IP(上述设置的为10.0.0.1) --flannel-iface wg0 (指定网卡)
参数 解释
–node-external-ip 节点外网IP
–advertise-address 节点间通信的ip
–node-ip 节点内网IP
–flannel-iface 指定通信时使用的网卡

3.2从节点接入k3s

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL=https://10.0.0.1:6443(主节点ip) K3S_TOKEN= (见主节点下 /var/lib/rancher/k3s/server/token) sh -s - --node-external-ip 从节点公网IP --node-ip 从节点内网IP,此处为10.0.0.2 --flannel-iface wg0 

curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_URL=https://10.0.0.1:6443 K3S_TOKEN=K10b99b088c197761285a9112f6f9e51faef41e92d3a86d3d668edfbb284bdc71a4::server:b2aee3028f6c8a2077265b773951f968 sh -s - --node-external-ip 1.116.38.204 --node-ip 10.0.0.3 --flannel-iface wg0

你可能感兴趣的:(linux,运维,网络)