ubuntu22.04安装kubernetes1.26 k8s1.26 （使用containerd）
文章目录
- 前言
- 一、基础环境
- 二、开始
-
- 1.关闭防火墙
- 2.禁用SELINUX
- 3禁用所有swap交换分区
- 4设置主机名
- 5创建配置文件
- 6配置服务器支持开启ipvs的前提条件
- 三、使用kubeadm安装k8s 1.26.0
-
- 1、安装containerd
- 2 拉取镜像，测试containerd是否能创建和启动成功
- 四、安装kubeadm、kubelet、kubectl
-
- 更新证书
- 更新源
- yum方式
- apt方式
- 使用kubeadm init初始化集群
- 安装包管理器helm 3
- 部署Pod Network组件Calico
- 验证k8s DNS是否可用
- 添加节点
- 五、闭坑和遇到的问题
-
- 参考
前言
本篇来讲解如何在ubuntu20.4下使用kubeadm安装部署k8s 1.26集群，并使用containerd。
一、基础环境
准备1台虚拟机（别惊慌，后面会教你克隆的）。
| 主机IP | 配置 |
|---|---|
| 192.168.2.5 | master节点，能连外网，ubuntu 20.04版本，至少2核CPU，2G内存 |
安装基础软件例如vim、wget、netstat,curl，方便修改ymal，查看端口，排查错误等等。
设置root用户。
还有就是换源！！！！更换国内镜像源方法
二、开始
1.关闭防火墙
root@master:~# ufw status #ufw查看当前的防火墙状态：inactive状态是防火墙关闭状态 active是开启状态。
root@master:~# ufw enable | disable #启动、关闭防火墙
2.禁用SELINUX
setenforce 0
vim /etc/selinux/config
SELINUX=disabled
3禁用所有swap交换分区
注：最好是安装虚拟机时就不要创建swap交换分区
root@master:~# swapoff -a #禁用所有swap交换分区
root@master:~# free -h
total used free shared buff/cache available
Mem: 1.8G 280M 1.2G 9.6M 286M 1.4G
Swap: 0B 0B 0B
root@master:~# vim /etc/fstab #永久禁用swap,删除或注释掉/etc/fstab里的swap设备的挂载命令即可
#/swap.img none swap sw 0 0
4设置主机名
cat >> /etc/hosts <<EOF
192.168.2.5 K8Smaster
EOF
后续克隆之后在执行
cat >> /etc/hosts <<EOF
192.168.2.6 K8sNode1 #在第二台虚拟机执行
192.168.2.6 K8sNode2 #在第三台虚拟机执行
EOF
注：如果提前配置了3台虚拟机需要同步时间否则会出error
root@master:~# date #查看时区，时间
root@master:~# timedatectl set-timezone Asia/Shanghai #先查看时区是否正常，不正确则替换为上海时区
root@master:~# yum -y install ntp #安装ntp服务
root@master:~# systemctl start ntp #开始ntpd服务,或者做定时任务如：*/5 * * * * /usr/sbin/ntpdate -u 192.168.2.5
root@master:~# systemctl enable ntp
5创建配置文件
创建/etc/modules-load.d/containerd.conf配置文件:
cat << EOF > /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
执行以下命令使配置生效:
modprobe overlay
modprobe br_netfilter
创建/etc/sysctl.d/99-kubernetes-cri.conf配置文件：
cat << EOF > /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
user.max_user_namespaces=28633
EOF
执行以下命令使配置生效:
sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf
6配置服务器支持开启ipvs的前提条件
kube-proxy开启ipvs的前提需要加载以下的内核模块:
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
那么执行脚本
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。
接下来还需要确保各个节点上已经安装了ipset软件包，为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm。
如果不满足以上前提条件，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。
三、使用kubeadm安装k8s 1.26.0
kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具，这个工具能通过两条指令完成一个kubernetes集群的部署。
1、创建一个master节点，kubeadm init。
2、将node节点加入kubernetes集群，kubeadm join <master_IP:port >。
我当然是用yaml啦！kubeadm init --config kubeadm.yaml 一键三连！
1、安装containerd
（在所有节点服务器上都执行，因为k8s 1.24版本默认CRI为containerd，cri称之为容器运行时插件）
containerd的官网
containerd官网安装教程，官网安装文档提供了源码包安装和普通的yum、apt-get安装，这里使用源码包安装。
下载Containerd的二进制包:
#安装containerd
wget https://github.com/containerd/containerd/releases/download/v1.6.14/containerd-1.6.14-linux-amd64.tar.gz #巨慢！建议浏览器下载，然后CP到/usr/local 然后执行解压
tar Cxzvf /usr/local containerd-1.6.8-linux-amd64.tar.gz #解压到/usr/local/bin目录下了
bin/ #解压到/usr/local/bin目录下了
bin/containerd-shim-runc-v2 #这6个可执行文件就是解压出来的containerd相关命令
bin/containerd-shim
bin/ctr
bin/containerd-shim-runc-v1
bin/containerd
bin/containerd-stress
#使用systemcd来管理containerd
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
mv containerd.service /usr/lib/systemd/system/
systemctl daemon-reload && systemctl enable --now containerd
systemctl status containerd #containerd已经是running状态了
#安装runc
#runc是容器运行时，runc实现了容器的init，run，create，ps...我们在运行容器所需要的cmd：
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64 && \
install -m 755 runc.amd64 /usr/local/sbin/runc
修改containerd的配置，因为containerd默认从k8s官网拉取镜像
mkdir -p /etc/containerd #创建一个目录用于存放containerd的配置文件
containerd config default | sudo tee /etc/containerd/config.toml #把containerd配置导出到文件
vim /etc/containerd/config.toml #修改配置文件
[plugins."io.containerd.grpc.v1.cri"]
..................................
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9" #搜索sandbox_image，把原来的k8s.gcr.io/pause:3.6改为"registry.aliyuncs.com/google_containers/pause:3.9"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
..........................
SystemdCgroup = true #搜索SystemdCgroup，把这个false改为true
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d" #搜索config_path，配置镜像加速地址（这是一个目录下面创建）
#创建镜像加速的目录
mkdir /etc/containerd/certs.d/docker.io -pv
#配置加速
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://b9pmyelo.mirror.aliyuncs.com"]
capabilities = ["pull", "resolve"]
EOF
#加载containerd的内核模块
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
#重启containerd
systemctl restart containerd
systemctl status containerd
2 拉取镜像，测试containerd是否能创建和启动成功
ctr i pull docker.io/library/nginx:alpine #能正常拉取镜像说明没啥问题
ctr images ls #查看镜像
ctr c create --net-host docker.io/library/nginx:alpine nginx #创建容器
ctr task start -d nginx #启动容器，正常说明containerd没啥问题
ctr containers ls #查看容器
ctr tasks kill -s SIGKILL nginx #终止容器
ctr containers rm nginx #删除容器
四、安装kubeadm、kubelet、kubectl
更新证书
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add
更新源
apt-get update
yum方式
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
apt方式
sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
#apt-mark 用于将软件包标记/取消标记为自动安装。 hold 选项用于将软件包标记为保留，以防止软件包被自动安装、升级或删除。
使用kubeadm init初始化集群
systemctl enable kubelet.service
使用kubeadm config print init-defaults --component-configs KubeletConfiguration可以打印集群初始化默认的使用的配置：
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 1.2.3.4 #需要替换master节点IP
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
name: node
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.k8s.io #这个源要换成阿里的！registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.26.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
从默认的配置中可以看到，可以使用imageRepository定制在集群初始化时拉取k8s所需镜像的地址。基于默认配置定制出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml：
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.2.5 #master节点IP
bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
taints:
- effect: PreferNoSchedule
key: node-role.kubernetes.io/master
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: 1.26.0
imageRepository: registry.aliyuncs.com/google_containers #阿里的源
networking:
podSubnet: 10.244.0.0/16
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
failSwapOn: false
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
这里定制了imageRepository为阿里云的registry，避免因gcr被墙，无法直接拉取镜像。criSocket设置了容器运行时为containerd。 同时设置kubelet的cgroupDriver为systemd，设置kube-proxy代理模式为ipvs。
在开始初始化集群之前可以使用kubeadm config images pull --config kubeadm.yaml预先在各个服务器节点上拉取所k8s需要的容器镜像。
kubeadm config images pull --config kubeadm.yaml #提前拉取镜像
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.26.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.6-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.9.3
接下来使用kubeadm初始化集群执行下面的命令：
kubeadm init --config kubeadm.yaml
W0104 01:55:08.531622 29428 common.go:84] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta2". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
[init] Using Kubernetes version: v1.26.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.503362 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:PreferNoSchedule]
[bootstrap-token] Using token: o4pn35.3zjs1udp6uw1eg8w
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
#注意：这段kubeamd join命令的token只有24h，24h就过期，需要执行kubeadm token create --print-join-command 重新生成。
kubeadm join 192.168.2.5:6443 --token shrl66.4tkloy7gnukj03l4 \
--discovery-token-ca-cert-hash sha256:f82eb61a39fbc0164f0b91e2d2619fc3d8f9b2d1b235ed3a60c4cc4e97b68dc9
上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。 其中有以下关键内容：
[certs]生成相关的各种证书
[kubeconfig]生成相关的kubeconfig文件
[kubelet-start] 生成kubelet的配置文件"/var/lib/kubelet/config.yaml"
[control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod
[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
[addons]安装基本插件:CoreDNS, kube-proxy
下面的命令是配置常规用户如何使用kubectl访问集群：
#我们根据输入的提示信息复制粘贴照着做即可
root@master:~# mkdir -p $HOME/.kube #复制上面提示照着做即可
root@master:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config #复制上面提示照着做即可
root@master:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config #复制上面提示照着做即可
root@master:~# export KUBECONFIG=/etc/kubernetes/admin.conf
查看一下集群状态，确认个组件都处于healthy状态
kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true","reason":""}
如果出现问题，就执行kubeadm reset命令进行清理，然后执行执行rm -rf $HOME/.kube，再init。（坑后面会补充）
安装包管理器helm 3
Helm是Kubernetes的包管理器，后续流程也将使用Helm安装Kubernetes的常用组件。 这里先在master节点node1上安装helm。
wget https://get.helm.sh/helm-v3.10.3-linux-amd64.tar.gz
tar -zxvf helm-v3.10.3-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/
执行helm list确认没有错误输出。
部署Pod Network组件Calico
选择calico作为k8s的Pod网络组件，下面使用helm在k8s集群中安装calico。
下载tigera-operator的helm chart:
wget https://github.com/projectcalico/calico/releases/download/v3.24.5/tigera-operator-v3.24.5.tgz
查看这个chart的中可定制的配置:
helm show values tigera-operator-v3.24.5.tgz
imagePullSecrets: {}
installation:
enabled: true
kubernetesProvider: ""
apiServer:
enabled: true
certs:
node:
key:
cert:
commonName:
typha:
key:
cert:
commonName:
caBundle:
#Resource requests and limits for the tigera/operator pod.
resources: {}
# Tolerations for the tigera/operator pod.
tolerations:
- effect: NoExecute
operator: Exists
- effect: NoSchedule
operator: Exists
# NodeSelector for the tigera/operator pod.
nodeSelector:
kubernetes.io/os: linux
# Custom annotations for the tigera/operator pod.
podAnnotations: {}
# Custom labels for the tigera/operator pod.
podLabels: {}
# Image and registry configuration for the tigera/operator pod.
tigeraOperator:
image: tigera/operator
version: v1.28.5
registry: quay.io
calicoctl:
image: docker.io/calico/ctl
tag: v3.24.5
定制的values.yaml如下:
# 可针对上面的配置进行定制,例如calico的镜像改成从私有库拉取。
# 这里只是个人本地环境测试k8s新版本，这里只有下面几行配置
apiServer:
enabled: false
使用helm安装calico：
helm install calico tigera-operator-v3.24.5.tgz -n kube-system --create-namespace -f values.yaml
等待并确认所有pod处于Running状态:
kubectl get pod -n kube-system | grep tigera-operator
tigera-operator-5fb55776df-wxbph 1/1 Running 0 5m10s
kubectl get pods -n calico-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-68884f975d-5d7p9 1/1 Running 0 5m24s
calico-node-twbdh 1/1 Running 0 5m24s
calico-typha-7b4bdd99c5-ssdn2 1/1 Running 0 5m24s
查看一下calico向k8s中添加的api资源:
kubectl api-resources | grep calico
bgpconfigurations crd.projectcalico.org/v1 false BGPConfiguration
bgppeers crd.projectcalico.org/v1 false BGPPeer
blockaffinities crd.projectcalico.org/v1 false BlockAffinity
caliconodestatuses crd.projectcalico.org/v1 false CalicoNodeStatus
clusterinformations crd.projectcalico.org/v1 false ClusterInformation
felixconfigurations crd.projectcalico.org/v1 false FelixConfiguration
globalnetworkpolicies crd.projectcalico.org/v1 false GlobalNetworkPolicy
globalnetworksets crd.projectcalico.org/v1 false GlobalNetworkSet
hostendpoints crd.projectcalico.org/v1 false HostEndpoint
ipamblocks crd.projectcalico.org/v1 false IPAMBlock
ipamconfigs crd.projectcalico.org/v1 false IPAMConfig
ipamhandles crd.projectcalico.org/v1 false IPAMHandle
ippools crd.projectcalico.org/v1 false IPPool
ipreservations crd.projectcalico.org/v1 false IPReservation
kubecontrollersconfigurations crd.projectcalico.org/v1 false KubeControllersConfiguration
networkpolicies crd.projectcalico.org/v1 true NetworkPolicy
networksets crd.projectcalico.org/v1 true NetworkSet
这些api资源是属于calico的，因此不建议使用kubectl来管理，推荐按照calicoctl来管理这些api资源。 将calicoctl安装为kubectl的插件:
cd /usr/local/bin
curl -o kubectl-calico -O -L "https://github.com/projectcalico/calicoctl/releases/download/v3.21.5/calicoctl-linux-amd64"
chmod +x kubectl-calico
也可以自行下来，然后cp到/usr/local/bin，改名字，改成可执行。
验证插件正常工作:
kubectl calico -h
验证k8s DNS是否可用
kubectl run curl --image=radial/busyboxplus:curl -it
If you don't see a command prompt, try pressing enter.
[ root@curl:/ ]$
进入后执行nslookup kubernetes.default确认解析正常:
nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
添加节点
克隆出2台K8sNode1, K8sNode1添加到Kubernetes集群中，分别在node2, node3上执行:
kubeadm join 192.168.2.5:6443 --token shrl66.4tkloy7gnukj03l4 \
--discovery-token-ca-cert-hash sha256:f82eb61a39fbc0164f0b91e2d2619fc3d8f9b2d1b235ed3a60c4cc4e97b68dc9
node2和node3加入集群很是顺利，在master节点上执行命令查看集群中的节点：
kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane 47m v1.26.0
K8sNode1 Ready <none> 31s v1.26.0
K8sNode1 Ready <none> 19s v1.26.0
五、闭坑和遇到的问题
在安装过程中遇到的一些问题和操作时的流程注意事项，也欢迎各位总结汇总，我也会定期更新
1、在kubeadm init成功之后，如果想kubeadm reset ，那么需要在执行kubeadm reset之前手动删除 $HOME/.kube。否则会报：
Unable to connect to the server: x509: certificate signed by unknown authority
不要犹豫直接删rm -rf $HOME/.kube，可以解决。
2、kubeadm init 时出现卡在
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
然后通过journalctl -f -u containerd看容器引擎的日志
failed, error" error="failed to get sandbox image \"k8s.gcr.io/pause:3.6\": failed to pull image \"k8s.gcr.io/pause:3.6\": failed to pull and unpack image \"k8s.gcr.io/pause:3.6\": failed to resolve reference \"k8s.gcr.io/pause:3.6\": failed to do request: Head \"https://k8s.gcr.io/v2/pause/manifests/3.6\": dial tcp 108.177.125.82:443: i/o timeout"
Jul 05 19:08:30 k8s-testing01-190 containerd[13788]: time="2023-3-01T19:08:30.696324518+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://k8s.gcr.io/v2/pause/manifests/3.6\": dial tcp 108.177.125.82:443: i/o timeout" host=k8s.gcr.io
因为containerd默认从k8s官网拉取镜像，其次默认配置中是pause:3.6版本，而实际我们需要的3.9版本。
failed to pull and unpack image \"k8s.gcr.io/pause:3.6\
所以我们安装完containerd一定修改containerd的配置，改源，改pause版本号。
参考
参考