高可用RKE2部署kubernetes（适用于生产环境）

IP	主机名	操作系统	应用	说明
10.10.114.100	k8s-rke2-worker			三台server的vip
10.10.114.101	k8s-rke2-worker01	rocky9.3	keepalive+haproxy,control-plane,etcd,master
10.10.114.102	k8s-rke2-worker02	rocky9.3	keepalive+haproxy,control-plane,etcd,master
10.10.114.103	k8s-rke2-worker03	rocky9.3	keepalive+haproxy,control-plane,etcd,master
10.10.114.104	k8s-rke2-node01	rocky9.3	agent	工作节点

1、环境准备(三台server主机相同)

#配置主机名
hostnamectl set-hostname k8s-rke2-worker01
#配置hosts
cat >> /etc/hosts << EOF
10.10.114.100 k8s-rke2-worker
10.10.114.101 k8s-rke2-worker01
10.10.114.102 k8s-rke2-worker02
10.10.114.103 k8s-rke2-worker03
EOF
#常用软件包安装
dnf -y install epel-release
dnf -y install lrzsz vim gcc glibc openssl openssl-devel net-tools http-tools wget curl  yum-utils telnet
dnf-y update  
关闭swap分区
swapoff -a
修改/etc/fstab配置文件，把swap分区挂载的那一行注释掉
vim /etc/fstab

1.1系统参数设置

参考：https://docs.rancher.cn/docs/rancher2/best-practices/optimize/os/_index

echo "
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
net.ipv4.conf.all.forwarding=1
net.ipv4.neigh.default.gc_thresh1=4096
net.ipv4.neigh.default.gc_thresh2=6144
net.ipv4.neigh.default.gc_thresh3=8192
net.ipv4.neigh.default.gc_interval=60
net.ipv4.neigh.default.gc_stale_time=120

# 参考 https://github.com/prometheus/node_exporter#disabled-by-default
kernel.perf_event_paranoid=-1

#sysctls for k8s node config
net.ipv4.tcp_slow_start_after_idle=0
net.core.rmem_max=16777216
fs.inotify.max_user_watches=524288
kernel.softlockup_all_cpu_backtrace=1

kernel.softlockup_panic=0

kernel.watchdog_thresh=30
fs.file-max=2097152
fs.inotify.max_user_instances=8192
fs.inotify.max_queued_events=16384
vm.max_map_count=262144
fs.may_detach_mounts=1
net.core.netdev_max_backlog=16384
net.ipv4.tcp_wmem=4096 12582912 16777216
net.core.wmem_max=16777216
net.core.somaxconn=32768
net.ipv4.ip_forward=1
net.ipv4.tcp_max_syn_backlog=8096
net.ipv4.tcp_rmem=4096 12582912 16777216

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1

kernel.yama.ptrace_scope=0
vm.swappiness=0

# 可以控制core文件的文件名中是否添加pid作为扩展。
kernel.core_uses_pid=1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route=0
net.ipv4.conf.all.accept_source_route=0

# Promote secondary addresses when the primary address is removed
net.ipv4.conf.default.promote_secondaries=1
net.ipv4.conf.all.promote_secondaries=1

# Enable hard and soft link protection
fs.protected_hardlinks=1
fs.protected_symlinks=1

# 源路由验证
# see details in https://help.aliyun.com/knowledge_detail/39428.html
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2

# see details in https://help.aliyun.com/knowledge_detail/41334.html
net.ipv4.tcp_max_tw_buckets=5000
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_synack_retries=2
kernel.sysrq=1

" >> /etc/sysctl.conf
modprobe br_netfilter
sysctl -p

1.2、需改连接数限制

cat >> /etc/security/limits.conf <<EOF
* soft nofile 65535
* hard nofile 65536
EOF

1.3、NetworkManager管理网络

如果使用NetworkManager管理网络，需要进行如下配置

参考：https://docs.rancher.cn/docs/rke2/known_issues/_index/#networkmanager

systemctl status NetworkManager
cat >> /etc/NetworkManager/conf.d/rke2-canal.conf << EOF
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:flannel*
EOF
systemctl daemon-reload
systemctl restart NetworkManager

1.4、防火墙关闭

#关闭firewalld
systemctl stop firewalld
systemctl disable firewalld
systemctl disable iptables
systemctl stop iptables
#关闭selinux
getenforce
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
getenforce

1.5、设置时间同步

yum -y install chrony
mv /etc/chrony.conf /etc/chrony.conf_bak
cat > /etc/chrony.conf <

以上设置完成后，重启主机，确认所有配置都已经生效。

2、RKE2安装配置

设置一个 HA 集群需要以下步骤：

1. 配置一个固定的注册地址
2. 启动第一个 server 节点
3. 加入其他 server 节点
4. 加入 agent 节点

参考：https://docs.rancher.cn/docs/rke2/install/ha/_index/

注意：由于主机有限，我们就把第一个启动的节点设置为注册地址，下面只进行2、3步骤。

2.1、下载安装

rke2版本信息：https://github.com/rancher/rke2/releases

使用国内镜像地址下载启动脚本

在每个节点都执行下面的脚本
参考：https://docs.rancher.cn/docs/rke2/install/install_options/install_options/_index#%E9%85%8D%E7%BD%AE-linux-%E5%AE%89%E8%A3%85%E8%84%9A%E6%9C%AC

wget https://rancher-mirror.oss-cn-beijing.aliyuncs.com/rke2/install.sh
INSTALL_RKE2_CHANNEL=stable
INSTALL_RKE2_TYPE=server
# INSTALL_RKE2_VERSION=v1.26.10+rke2r2
/bin/bash install.sh

# 执行完之后执行命令，如下
# rke2 --version 
rke2 version v1.26.10+rke2r2 (21e3a8c82da71473f2b846065dcab197a9b2c9d8)
go version go1.20.10 X:boringcrypto
#上面主要是下载一些必要的安装包

2.2、第一个节点启动，初始化

systemctl start rke2-server
systemctl enable rke2-server
# 第一次启动可能会比较慢，因为要初始化、生成各种文件 
#查看第一次启动后生成的文件
ll /var/lib/rancher/rke2/
ll /var/lib/rancher/rke2/bin/
ll /etc/rancher/rke2/