基于 containerd 运行时的 Kubernetes 环境搭建

众说周知，Kubernetes 从 1.24 起就删除了 Dockershim 相关代码，现在整理了一下国内网络环境下使用 containerd 作为运行时的集群搭建步骤

1. 系统准备

  # 允许 iptables 检查桥接流量
sudo tee /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

sudo tee /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
EOF

# 使设置生效
sudo modprobe overlay br_netfilter
sudo sysctl --system

# 禁用虚拟内存
sudo swapoff -a

2. 安装容器运行时

• 安装 containerd

  # Debian/Ubuntu 部分
sudo curl -Lo /etc/apt/trusted.gpg.d/docker-ce.asc http://mirrors.aliyun.com/docker-ce/linux/debian/gpg
echo "deb http://mirrors.aliyun.com/docker-ce/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update
sudo apt install -y containerd.io

# CentOS/RHEL 部分
sudo curl https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo
sed -i 's/download.docker.com/mirrors.aliyun.com\/docker-ce/g' /etc/yum.repos.d/docker-ce.repo
sudo yum install -y containerd.io
sudo systemctl enable containerd

# 配置容器运行时
sudo tee /etc/containerd/config.toml << EOF
version = 2
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
    runtime_type = "io.containerd.runc.v2"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
      SystemdCgroup = true
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://hub-mirror.c.163.com"]
EOF

sudo systemctl restart containerd

• 安装 kubeadm

  # Debian/Ubuntu 部分
sudo curl -Lo /etc/apt/trusted.gpg.d/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg
echo "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubeadm=1.21.10-00 kubectl=1.21.10-00 kubelet=1.21.10-00

# CentOS/RHEL 部分
sudo tee /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=0
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sudo yum install -y --enablerepo=kubernetes kubeadm-1.21.10 kubelet-1.21.10 kubectl-1.21.10
sudo systemctl enable kubelet

# 添加自动填充
echo 'source <(kubectl completion bash)' >>~/.bashrc

# 配置 crictl
sudo tee /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
EOF

3. 使用 Kubeadm 引导集群

编辑如下内容到 /tmp/kubeadm.yaml

  apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    node-labels: "ingress-ready=true"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
ipvs:
  strictARP: true
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  serviceSubnet: "10.96.0.0/16"
  podSubnet: "10.244.0.0/16"
kubernetesVersion: 1.21.10
imageRepository: registry.aliyuncs.com/google_containers
etcd:
  local:
    extraArgs:
      listen-metrics-urls: http://0.0.0.0:2381
controllerManager:
  extraArgs:
    bind-address: 0.0.0.0
scheduler:
  extraArgs:
    bind-address: 0.0.0.0
apiServer:
  certSANs:
  - vcap.me

运行 sudo kubeadm init --config /tmp/kubeadm.yaml 稍等几分钟，K8S 的 Master 节点就启动起来了

• kubernetes 组件安装

  # 链接 kubeconfig
sudo chmod +r /etc/kubernetes/admin.conf
mkdir -p $HOME/.kube && ln -s /etc/kubernetes/admin.conf $HOME/.kube/config

# 安装网络插件
kubectl create -f https://projectcalico.docs.tigera.io/manifests/calico.yaml

# 去除污点
kubectl taint node --all node-role.kubernetes.io/master-

# 安装 Helm
curl -sL https://get.helm.sh/helm-v3.8.1-linux-amd64.tar.gz | sudo tar zxf - --strip-components=1 -C /usr/local/bin

# 安装 Metallb
helm install metallb -n kube-system --repo https://metallb.github.io/metallb metallb --values - << EOF
configInline:
  address-pools:
  - name: default
    protocol: layer2
    addresses:
    - 192.168.4.10-192.168.4.100
EOF

# 安装 Traefik Ingress Controller
helm install traefik -n kube-system --repo https://helm.traefik.io/traefik traefik --values - << EOF
deployment:
  kind: DaemonSet
ingressClass:
  enabled: true
  isDefaultClass: true
EOF

参考资料

1. 弃用 Dockershim 的常见问题
2. 生产环境