kubernetes-v1.22.13集群搭建

kubernetes-v1.22.13集群搭建

本文采用 kubeadm 的方式搭建 kubernetes-v1.22.13 集群。

1、环境准备

服务器规划(本实验采用虚拟机)：

ip	hostname
192.168.43.211	master
192.168.43.212	slave1
192.168.43.213	slave2
192.168.43.214	master2

2、系统初始化(all node)

2.1 关闭防火墙

# 第1步
# 临时关闭
systemctl stop firewalld
# 永久关闭
systemctl disable firewalld

2.2 关闭 selinux

# 第2步
# 临时关闭
setenforce 0
# 永久关闭
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config

2.3 关闭 swap

# 第3步
# 临时关闭
swapoff -a
# 永久关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab

2.4 设置主机名称

使用命令 hostnamectl set-hostname hostname 设置主机名称，如下四台主机分别设置为：

# 第4步
# 设置
hostnamectl set-hostname master
hostnamectl set-hostname slave1
hostnamectl set-hostname slave2
hostnamectl set-hostname master2
# 查看当前主机名称
hostname

2.5 添加hosts

在每个节点中添加 hosts，即节点IP地址+节点名称。

# 第5步
cat >> /etc/hosts << EOF
192.168.43.211 master
192.168.43.211 cluster-endpoint
192.168.43.212 slave1
192.168.43.213 slave2
192.168.43.214 master2
EOF

2.6 将桥接的IPv4流量传递到iptables的链

# 第6步
# 设置
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# 使其生效
sysctl --system

2.7 时间同步

让各个节点(虚拟机)中的时间与本机时间保持一致。

# 第7步
yum install ntpdate -y
ntpdate time.windows.com

注意：虚拟机不管关机还是挂起，每次重新操作都需要更新时间进行同步。

3、Docker的安装(all node)

3.1 卸载旧版本

# 第8步
yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

3.2 设置镜像仓库

# 第9步
# 默认是国外的,这里使用阿里云的镜像
yum-config-manager \
    --add-repo \
    http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3.3 安装需要的插件

# 第10步
yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

3.4 更新yum软件包索引

# 第11步
# 更新yum软件包索引
yum makecache fast

3.5 安装docker引擎

# 第12步
# 安装特定版本 
# 查看有哪些版本
yum list docker-ce --showduplicates | sort -r
yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io
# 这里安装
yum install docker-ce-20.10.21 docker-ce-cli-20.10.21 containerd.io
# 安装最新版本
yum install docker-ce docker-ce-cli containerd.io

3.6 启动Docker

# 第13步
systemctl enable docker && systemctl start docker

3.7 配置Docker镜像加速

# 第14步
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

# 重启
systemctl restart docker

3.8 查看加速是否生效

# 第15步
docker info

3.9 验证Docker信息

# 第16步
docker -v

3.10 其它Docker命令

# 停止docker
systemctl stop docker

# 查看docker状态
systemctl status docker

3.11 卸载Docker的命令

yum remove docker-ce-20.10.21 docker-ce-cli-20.10.21 containerd.io
rm -rf /var/lib/docker
rm -rf /var/lib/containerd

4、添加阿里云yum源(all node)

# 第17步
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[Kubernetes]
name=kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

5、kubeadm、kubelet、kubectl的安装(all node)

# 第18步
yum install -y kubelet-1.22.13 kubeadm-1.22.13 kubectl-1.22.13 --disableexcludes=kubernetes

6、启动kubelet服务(all node)

# 第19步
systemctl enable kubelet && systemctl start kubelet

7、kubeadm config(all node)

kubeadm 已经进入 GA 阶段，其控制面初始化和加入节点步骤都支持大量的可定制内容，因此 kubeadm 还提供

了配置文件功能用于复杂定制。同时，kubeadm 将配置文件以 ConfigMap 的形式保存到集群之中，便于后续的

查询和升级工作。kubeadm config 子命令提供了对这一组功能的支持：

• kubeadm config upload from-file：由配置文件上传到集群中生成ConfigMap。

• kubeadm config upload from-flags：由配置参数生成ConfigMap。

• kubeadm config view：查看当前集群中的配置值。

• kubeadm config print init-defaults：输出kubeadm init默认参数文件的内容。

• kubeadm config print join-defaults：输出kubeadm join默认参数文件的内容。

• kubeadm config migrate：在新旧版本之间进行配置转换。

• kubeadm config images list：列出所需的镜像列表。

• kubeadm config images pull：拉取镜像到本地。

# 第20步
# 查看安装需要的镜像
[root@master ~]# kubeadm config images list
I0621 15:44:56.362196   11697 version.go:255] remote version is much newer: v1.27.3; falling back to: stable-1.22
k8s.gcr.io/kube-apiserver:v1.22.17
k8s.gcr.io/kube-controller-manager:v1.22.17
k8s.gcr.io/kube-scheduler:v1.22.17
k8s.gcr.io/kube-proxy:v1.22.17
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4

8、生成和修改初始化文件(master node)

# 第21步
# 生成初始化文件
[root@master ~]# kubeadm config print init-defaults > kubeadm-config.yaml
[root@master ~]# cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: 1.22.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
scheduler: {}

关于 kubeadm-config.yaml 更多配置语法参考：

https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2

使用 kubeadm-config.yaml 配置主节点：

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/

kube-proxy 开启 ipvs 参考：

https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md

kubelet的配置示例参考：

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#configure-kubelets-using-kubeadm

其它：

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network

修改文件的内容：

# 修改初始化文件
# 第22步
[root@master ~]# vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.43.211 # 修改apiserver-advertise-address为你自己的ip
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: master # 修改为master节点的主机名
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
# 修改镜像地址
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.22.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12 # 修改service网段
  podSubnet: 10.244.0.0/16 # 修改pod网段
# 修改kubernetes版本
kubernetesVersion: v1.22.13
# 修改control-plane-endpoint
controlPlaneEndpoint: cluster-endpoint
scheduler: {}

我们这里使用的是阿里云的镜像，也可以不进行修改任然使用 k8s.gcr.io 的配置，需要将阿里云的镜像替换

k8s.gcr.io 的镜像：

#!/bin/bash

url=registry.cn-hangzhou.aliyuncs.com/google_containers
version=v1.17.4
images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
for imagename in ${images[@]} ; do
  docker pull $url/$imagename
  docker tag $url/$imagename k8s.gcr.io/$imagename
  docker rmi -f $url/$imagename
done

我们这里可以提前下载好阿里云的镜像：

# 下载镜像
# 第23步
#!/bin/bash

images=`kubeadm config images list --config kubeadm-config.yaml`
if [[ -n ${images} ]]
then
  echo "开始拉取镜像"
  for i in ${images};
    do 
      echo $i
      docker pull $i;
  done
else
 echo "没有可拉取的镜像"
fi

# 查看镜像
# 第24步
[root@master ~]# docker images
REPOSITORY                                                        TAG        IMAGE ID       CREATED         SIZE
registry.aliyuncs.com/google_containers/kube-apiserver            v1.22.13   621a47290365   10 months ago   128MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.22.13   e502e6e4602a   10 months ago   122MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.22.13   cdb2e5a20b69   10 months ago   52.7MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.22.13   b03c3a7e321f   10 months ago   104MB
registry.aliyuncs.com/google_containers/etcd                      3.5.0-0    004811815584   2 years ago     295MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.4     8d147537fb7d   2 years ago     47.6MB
registry.aliyuncs.com/google_containers/pause                     3.5        ed210e3e4a5b   2 years ago     683kB

9、kubeadm初始化(master node)

# 初始化
# 第25步
[root@master ~]# kubeadm init --config=kubeadm-config.yaml
W0621 16:54:30.759263   48808 strict.go:55] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: error converting YAML to JSON: yaml: unmarshal errors:
  line 20: key "kubernetesVersion" already set in map
[init] Using Kubernetes version: v1.22.13
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cluster-endpoint kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local node] and IPs [10.96.0.1 192.168.43.211]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost node] and IPs [192.168.43.211 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost node] and IPs [192.168.43.211 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.004414 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node node as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:624a6f266ee4be08e6d161d6f4e57e73dc44b14da777f7bb11495c6e5a70f965 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:624a6f266ee4be08e6d161d6f4e57e73dc44b14da777f7bb11495c6e5a70f965

查看命令执行后的提示信息，看到 Your Kubernetes control-plane has initialized successfully! 说

明我们 master 节点上的 k8s 集群已经搭建成功。

10、开启kubectl工具的使用(master node)

# 第26步
[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

11、查看安装情况(master node)

# 第27步
[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES                  AGE   VERSION
master   NotReady   control-plane,master   12m   v1.22.13

[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-7f6cbbb7b8-7s96l       0/1     Pending   0          14m
kube-system   coredns-7f6cbbb7b8-vwhvr       0/1     Pending   0          14m
kube-system   etcd-node                      1/1     Running   0          14m
kube-system   kube-apiserver-node            1/1     Running   0          14m
kube-system   kube-controller-manager-node   1/1     Running   0          14m
kube-system   kube-proxy-fhttf               1/1     Running   0          14m
kube-system   kube-scheduler-node            1/1     Running   0          14m

12、slave节点加入(slave node)

# 第28步
[root@slave1 ~]# kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
>         --discovery-token-ca-cert-hash sha256:624a6f266ee4be08e6d161d6f4e57e73dc44b14da777f7bb11495c6e5a70f965
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@slave2 ~]# kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
>         --discovery-token-ca-cert-hash sha256:624a6f266ee4be08e6d161d6f4e57e73dc44b14da777f7bb11495c6e5a70f965
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

查看安装情况：

# 第29步
[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES                  AGE    VERSION
master   NotReady   control-plane,master   20m    v1.22.13
slave1   NotReady   <none>                 2m8s   v1.22.13
slave2   NotReady   <none>                 2m5s   v1.22.13

[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-7f6cbbb7b8-7s96l       0/1     Pending   0          20m
kube-system   coredns-7f6cbbb7b8-vwhvr       0/1     Pending   0          20m
kube-system   etcd-node                      1/1     Running   0          20m
kube-system   kube-apiserver-node            1/1     Running   0          20m
kube-system   kube-controller-manager-node   1/1     Running   0          20m
kube-system   kube-proxy-cjlrb               1/1     Running   0          2m9s
kube-system   kube-proxy-fhttf               1/1     Running   0          20m
kube-system   kube-proxy-s9bkk               1/1     Running   0          2m6s
kube-system   kube-scheduler-node            1/1     Running   0          20m

13、master2节点加入集群(master2 node)

证书拷贝：

# 第30步
# 创建目录
[root@master2 ~]# mkdir -p /etc/kubernetes/pki/etcd

# 第31步
# 将master节点上的证书拷贝到master2节点上
[root@master ~]# scp -rp /etc/kubernetes/pki/ca.* master2:/etc/kubernetes/pki
[root@master ~]# scp -rp /etc/kubernetes/pki/sa.* master2:/etc/kubernetes/pki
[root@master ~]# scp -rp /etc/kubernetes/pki/front-proxy-ca.* master2:/etc/kubernetes/pki
[root@master ~]# scp -rp /etc/kubernetes/pki/etcd/ca.* master2:/etc/kubernetes/pki/etcd
[root@master ~]# scp -rp /etc/kubernetes/admin.conf master2:/etc/kubernetes
[root@master ~]# scp kubeadm-config.yaml root@master2:~

提前下载好镜像：

# 下载镜像
# 第32步
#!/bin/bash

images=`kubeadm config images list --config kubeadm-config.yaml`
if [[ -n ${images} ]]
then
  echo "开始拉取镜像"
  for i in ${images};
    do 
      echo $i
      docker pull $i;
  done
else
 echo "没有可拉取的镜像"
fi

加入集群：

# 第33步
[root@master2 ~]# kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
>         --discovery-token-ca-cert-hash sha256:624a6f266ee4be08e6d161d6f4e57e73dc44b14da777f7bb11495c6e5a70f965 \
>         --control-plane
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cluster-endpoint kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master2] and IPs [10.96.0.1 192.168.43.214]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master2] and IPs [192.168.43.214 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master2] and IPs [192.168.43.214 127.0.0.1 ::1]
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node master2 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master2 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

# 第34步
[root@master2 ~]# mkdir -p $HOME/.kube
[root@master2 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master2 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看节点：

# 第35步
[root@master2 ~]# kubectl get nodes
NAME      STATUS     ROLES                  AGE     VERSION
master    NotReady   control-plane,master   35m     v1.22.13
master2   NotReady   control-plane,master   2m34s   v1.22.13
slave1    NotReady   <none>                 16m     v1.22.13
slave2    NotReady   <none>                 16m     v1.22.13

[root@master2 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                              READY   STATUS    RESTARTS        AGE
kube-system   coredns-7f6cbbb7b8-7s96l          0/1     Pending   0               34m
kube-system   coredns-7f6cbbb7b8-vwhvr          0/1     Pending   0               34m
kube-system   etcd-master2                      1/1     Running   0               2m31s
kube-system   etcd-node                         1/1     Running   0               35m
kube-system   kube-apiserver-master2            1/1     Running   0               2m36s
kube-system   kube-apiserver-node               1/1     Running   0               35m
kube-system   kube-controller-manager-master2   1/1     Running   0               2m36s
kube-system   kube-controller-manager-node      1/1     Running   1 (2m20s ago)   35m
kube-system   kube-proxy-cjlrb                  1/1     Running   0               16m
kube-system   kube-proxy-fhttf                  1/1     Running   0               34m
kube-system   kube-proxy-s44k8                  1/1     Running   0               2m37s
kube-system   kube-proxy-s9bkk                  1/1     Running   0               16m
kube-system   kube-scheduler-master2            1/1     Running   0               2m36s
kube-system   kube-scheduler-node               1/1     Running   1 (2m20s ago)   35m

注：由于网络插件还没有部署，所有节点还没有准备就绪，状态为 NotReady，下面安装网络插件。

14、安装网络插件fannel(master node)

# 第36步
# 获取fannel的配置文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# 如果出现无法访问的情况,可以直接用下面的flannel网络的官方github地址
wget https://github.com/flannel-io/flannel/tree/master/Documentation/kube-flannel.yml

# 第37步
# 修改文件内容
net-conf.json: |
    {
      "Network": "10.244.0.0/16", #这里的网段地址需要与master初始化的必须保持一致
      "Backend": {
        "Type": "vxlan"
      }
    }

# 第38步
[root@master ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

查看节点情况：

# 第39步
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE      NAME                              READY   STATUS                  RESTARTS      AGE
kube-flannel   kube-flannel-ds-jvwpw             0/1     Init:ImagePullBackOff   0             5m5s
kube-flannel   kube-flannel-ds-k2qsz             0/1     Init:ImagePullBackOff   0             5m5s
kube-flannel   kube-flannel-ds-nkxth             0/1     Init:ImagePullBackOff   0             5m5s
kube-flannel   kube-flannel-ds-xnr9q             0/1     Init:ImagePullBackOff   0             5m5s
kube-system    coredns-7f6cbbb7b8-7s96l          0/1     Pending                 0             45m
kube-system    coredns-7f6cbbb7b8-vwhvr          0/1     Pending                 0             45m
kube-system    etcd-master2                      1/1     Running                 0             12m
kube-system    etcd-node                         1/1     Running                 0             45m
kube-system    kube-apiserver-master2            1/1     Running                 0             12m
kube-system    kube-apiserver-node               1/1     Running                 0             45m
kube-system    kube-controller-manager-master2   1/1     Running                 0             12m
kube-system    kube-controller-manager-node      1/1     Running                 1 (12m ago)   45m
kube-system    kube-proxy-cjlrb                  1/1     Running                 0             26m
kube-system    kube-proxy-fhttf                  1/1     Running                 0             45m
kube-system    kube-proxy-s44k8                  1/1     Running                 0             12m
kube-system    kube-proxy-s9bkk                  1/1     Running                 0             26m
kube-system    kube-scheduler-master2            1/1     Running                 0             12m
kube-system    kube-scheduler-node               1/1     Running                 1 (12m ago)   45m

如果出现 Init:ImagePullBackOff 错误，请提前准备好镜像进行导入即可。

# 第40步
# master节点执行
[root@master ~]# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
master    Ready    control-plane,master   52m   v1.22.13
master2   Ready    control-plane,master   19m   v1.22.13
slave1    Ready    <none>                 34m   v1.22.13
slave2    Ready    <none>                 34m   v1.22.13

# 第41步
# master2节点执行
[root@master2 ~]# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
master    Ready    control-plane,master   52m   v1.22.13
master2   Ready    control-plane,master   19m   v1.22.13
slave1    Ready    <none>                 34m   v1.22.13
slave2    Ready    <none>                 34m   v1.22.13

查看 pod 情况：

# 第42步
# master节点执行
[root@master ~]# kubectl get pods --all-namespaces
NAMESPACE      NAME                              READY   STATUS    RESTARTS      AGE
kube-flannel   kube-flannel-ds-jvwpw             1/1     Running   0             13m
kube-flannel   kube-flannel-ds-k2qsz             1/1     Running   0             13m
kube-flannel   kube-flannel-ds-nkxth             1/1     Running   0             13m
kube-flannel   kube-flannel-ds-xnr9q             1/1     Running   0             13m
kube-system    coredns-7f6cbbb7b8-7s96l          1/1     Running   0             53m
kube-system    coredns-7f6cbbb7b8-vwhvr          1/1     Running   0             53m
kube-system    etcd-master2                      1/1     Running   0             21m
kube-system    etcd-node                         1/1     Running   0             53m
kube-system    kube-apiserver-master2            1/1     Running   0             21m
kube-system    kube-apiserver-node               1/1     Running   0             53m
kube-system    kube-controller-manager-master2   1/1     Running   0             21m
kube-system    kube-controller-manager-node      1/1     Running   1 (20m ago)   53m
kube-system    kube-proxy-cjlrb                  1/1     Running   0             35m
kube-system    kube-proxy-fhttf                  1/1     Running   0             53m
kube-system    kube-proxy-s44k8                  1/1     Running   0             21m
kube-system    kube-proxy-s9bkk                  1/1     Running   0             35m
kube-system    kube-scheduler-master2            1/1     Running   0             21m
kube-system    kube-scheduler-node               1/1     Running   1 (20m ago)   53m

# 第43步
# master2节点执行
[root@master2 ~]# kubectl get pods --all-namespaces
NAMESPACE      NAME                              READY   STATUS    RESTARTS      AGE
kube-flannel   kube-flannel-ds-jvwpw             1/1     Running   0             13m
kube-flannel   kube-flannel-ds-k2qsz             1/1     Running   0             13m
kube-flannel   kube-flannel-ds-nkxth             1/1     Running   0             13m
kube-flannel   kube-flannel-ds-xnr9q             1/1     Running   0             13m
kube-system    coredns-7f6cbbb7b8-7s96l          1/1     Running   0             53m
kube-system    coredns-7f6cbbb7b8-vwhvr          1/1     Running   0             53m
kube-system    etcd-master2                      1/1     Running   0             21m
kube-system    etcd-node                         1/1     Running   0             54m
kube-system    kube-apiserver-master2            1/1     Running   0             21m
kube-system    kube-apiserver-node               1/1     Running   0             54m
kube-system    kube-controller-manager-master2   1/1     Running   0             21m
kube-system    kube-controller-manager-node      1/1     Running   1 (21m ago)   54m
kube-system    kube-proxy-cjlrb                  1/1     Running   0             35m
kube-system    kube-proxy-fhttf                  1/1     Running   0             53m
kube-system    kube-proxy-s44k8                  1/1     Running   0             21m
kube-system    kube-proxy-s9bkk                  1/1     Running   0             35m
kube-system    kube-scheduler-master2            1/1     Running   0             21m
kube-system    kube-scheduler-node               1/1     Running   1 (21m ago)   54m

至此，通过 kubeadm 工具就实现了 Kubernetes 集群的快速搭建。如果安装失败，则可以执行 kubeadm reset

命令将主机恢复原状，重新执行 kubeadm init 命令，再次进行安装。

Kubernetes 集群安装目录：/etc/kubernetes/

Kubernetes 集群组件配置文件目录：/etc/kubernetes/manifests/

注：以后所有 yaml 文件都只在 master 节点执行。

15、使用另一种方式让slave加入集群

前面我们使用生成的命令让slave加入集群：

kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:624a6f266ee4be08e6d161d6f4e57e73dc44b14da777f7bb11495c6e5a70f965

下面我们使用 kubeadm config print join-defaults 生成配置文件让slave加入集群。

重新初始化集群：

[root@master ~]# kubeadm reset -f
[root@slave1 ~]# kubeadm reset -f
[root@slave1 ~]# kubeadm reset -f
[root@master2 ~]# kubeadm reset -f

[root@master ~]# kubeadm init --config=kubeadm-config.yaml
W0621 20:34:09.948042   89489 strict.go:55] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: error converting YAML to JSON: yaml: unmarshal errors:
  line 20: key "kubernetesVersion" already set in map
[init] Using Kubernetes version: v1.22.13
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [cluster-endpoint kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.43.211]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.43.211 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.43.211 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 5.504992 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:7c250d4b0a4fccc9cc9a45c35a232533c4c5b0410a18a0b23f67dc13467b6576 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join cluster-endpoint:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:7c250d4b0a4fccc9cc9a45c35a232533c4c5b0410a18a0b23f67dc13467b6576

kubeadm 命令生成配置文件：

[root@slave1 ~]# kubeadm config print join-defaults > join-config.yaml
[root@slave2 ~]# kubeadm config print join-defaults > join-config.yaml

[root@slave1 ~]# cat join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: kube-apiserver:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: slave1
  taints: null
  
[root@slave2 ~]# cat join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: kube-apiserver:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: slave2
  taints: null

修改 join-config.yaml 文件：

[root@slave1 ~]# cat join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: cluster-endpoint:6443 # master节点的ip地址和端口
    token: abcdef.0123456789abcdef # 刚刚在master上获取的token
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef # 刚刚在master上获取的token
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: slave1
  taints: null
  
[root@slave2 ~]# cat join-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
  bootstrapToken:
    apiServerEndpoint: cluster-endpoint:6443 # master节点的ip地址和端口
    token: abcdef.0123456789abcdef # 刚刚在master上获取的token
    unsafeSkipCAVerification: true
  timeout: 5m0s
  tlsBootstrapToken: abcdef.0123456789abcdef # 刚刚在master上获取的token
kind: JoinConfiguration
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  imagePullPolicy: IfNotPresent
  name: slave2
  taints: null

加入：

[root@slave1 ~]# kubeadm join --config=join-config.yaml
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@slave2 ~]# kubeadm join --config=join-config.yaml
[discovery.bootstrapToken.token: Invalid value: "abcdef.0123456789abcdef1": the bootstrap token is invalid, discovery.tlsBootstrapToken: Invalid value: "abcdef.0123456789abcdef1": the bootstrap token is invalid]
To see the stack trace of this error execute with --v=5 or higher
[root@slave2 ~]# kubeadm join --config=join-config.yaml
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

查看节点信息：

[root@master ~]# kubectl get nodes
NAME     STATUS      ROLES                  AGE    VERSION
master   NotReady    control-plane,master   11m    v1.22.13
slave1   NotReady    <none>                 102s   v1.22.13
slave2   NotReady    <none>                 41s    v1.22.13  

16、卸载k8s环境

yum -y remove kubelet kubeadm kubectl
sudo kubeadm reset -f
sudo rm -rvf $HOME/.kube
sudo rm -rvf ~/.kube/
sudo rm -rvf /etc/kubernetes/
sudo rm -rvf /etc/systemd/system/kubelet.service.d
sudo rm -rvf /etc/systemd/system/kubelet.service
sudo rm -rvf /usr/bin/kube*
sudo rm -rvf /etc/cni
sudo rm -rvf /opt/cni
sudo rm -rvf /var/lib/etcd
sudo rm -rvf /var/etcd

至此，kubernetes-v1.22.13 集群就部署完成了。