K8s：先跑起来，部署Kubernetes Cluster集群

更新中

本文为Kubernetes系列的第一篇文章。

越是门槛高的知识，就越需要一个最小化可用的系统。如果直接上来就只学习理论知识和概念，就很容易造成：从入门到放弃。
按照一贯的学习思路在短时间内搭建出来一个可用的系统。这样可以尽快建立起对学习对象的感性认识。先用起来，然后快速了解基本概念、功能和使用场景，进而不断加深研究与开发。

参考链接
使用kubeadm创建Kubernetes集群

0x00环境准备

本文使用主机的系统皆为CentOS 7，所有机器之间网络连接正常。
确保3台主机的 /etc/hostname 和/etc/hosts文件中已经修改为正确的主机名。以及相应的主机名称解析，修改后，请重启系统。
注意：请确保CPU至少2核，内存2G
所有命令皆在root用户下运行

Master：k8s-master
Node：k8s-node1，k8s-node2

主机名	IP地址	作用	配置
k8s-master	192.168.2.103	主节点Master	2U4G64G
k8s-node1	192.168.2.123	从节点Node	2U4G64G
k8s-node2	192.168.2.130	从节点Node	2U4G64G

1.host相关配置

（1）文件修改

配置文件/etc/hosts

Master节点

hostnamectl set-hostname k8s-master
hostname
hostnamectl
cat /etc/hosts

echo 192.168.2.103 k8s-master >> /etc/hosts
echo 192.168.2.123 k8s-node1 >> /etc/hosts
echo 192.168.2.130 k8s-node2 >> /etc/hosts
cat /etc/hosts

Node1节点

hostnamectl set-hostname k8s-node1
hostname
hostnamectl
cat /etc/hosts

echo 192.168.2.103 k8s-master >> /etc/hosts
echo 192.168.2.123 k8s-node1 >> /etc/hosts
echo 192.168.2.130 k8s-node2 >> /etc/hosts
cat /etc/hosts

Node2节点

hostnamectl set-hostname k8s-node2
hostname
hostnamectl
cat /etc/hosts

echo 192.168.2.103 k8s-master >> /etc/hosts
echo 192.168.2.104 k8s-node1 >> /etc/hosts
echo 192.168.2.125 k8s-node2 >> /etc/hosts
cat /etc/hosts

注：
①某些发行版本在/etc/hosts 文件中添加了多余的条目，该条目将实际的主机名解析为另一个回送IP地址，例如127.0.1.1。如果有的话，则必须将其注释或删除，以防止名称解析问题。
②不要删除127.0.0.1条目。
③每一个节点都需要对其他节点进行主机名的解析配置。
④主机名修改完成后，退出当前终端，重新登录服务器。

（2）验证连通性

在每一个节点执行

ping -c 4 k8s-master
ping -c 4 k8s-node1
ping -c 4 k8s-node2
ping -c 4 qq.com

2.防火墙配置

在实验环境中可以选择直接禁用防火墙，但是在生产环境中根据实际的需求进行规则的配置。
禁用防火墙

（1）关闭firewalld

在CentOS7上面防火墙是firewalld服务，停止并且禁止开机启动firewalld.service

systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service

（2）关闭 selinux

setenforce 0
getenforce
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
grep SELINUX=disabled /etc/sysconfig/selinux
grep SELINUX=disabled /etc/selinux/config

3.配置k8s.conf

参考链接

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

执行如下命令使修改生效：

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

4.swap配置

禁用swap
Linux swapoff命令用于关闭系统交换区(swap area)。swapoff实际上为swapon的符号连接，可用来关闭系统的交换区。

sed -i '/swap/ s/^/#/' /etc/fstab
swapoff -a
free -m

-a 将/etc/fstab文件中所有设置为swap的设备关闭
-h 帮助信息
-V 版本信息

5.通用YUM源配置

注：博主使用阿里云的ISO镜像，已经默认为阿里云的镜像源，故跳过此步骤。
（1）备份域名原repo文件，在CentOS中配置使用网易和阿里的开源镜像。

cd
mkdir ori_repo-config
mv /etc/yum.repos.d/* ./ori_repo-config/
cd /etc/yum.repos.d/
wget http://mirrors.aliyun.com/repo/Centos-7.repo
wget http://mirrors.163.com/.help/CentOS7-Base-163.repo

（2）更新源

yum clean all
yum makecache
yum repolist
cd

6.基础安装

（1）Linux常用工具

yum install -y lsof vim net-tools wget git curl

yum install -y yum-utils device-mapper-persistent-data lvm2

（2）NTP时间同步

使用下面的命令一次性完成配置，详细配置附后。

yum  -y install chrony
sed -i '3s/0.centos.pool.ntp.org/ntp1.aliyun.com/g' /etc/chrony.conf
sed -i '4s/1.centos.pool.ntp.org/ntp2.aliyun.com/g' /etc/chrony.conf
sed -i '5,6s/^/#/g' /etc/chrony.conf
sed -i "s|#allow 192.168.0.0/16|allow 192.168.2.0/24|g" /etc/chrony.conf
systemctl restart chronyd.service
systemctl status chronyd.service
systemctl enable chronyd.service
systemctl list-unit-files |grep chronyd.service
timedatectl set-timezone Asia/Shanghai
chronyc sources
timedatectl status

（1）编辑chrony.conf文件，使用下面的命令完成添加。

yum  -y install chrony
vim /etc/chrony.conf

# add follow
server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
allow 192.168.232.2/24

注意：
①替换NTP_SERVER为合适的更准确（较低层）NTP服务器的主机名或IP地址。
②该配置支持多个server键。默认情况下，控制器节点通过公共服务器池同步时间。但是也可以选择配置备用服务器，例如组织提供的服务器。
③要使其他节点能够连接到控制器节点上的chrony守护程序，请将此密钥添加到上述相同的chrony.conf文件中：allow 10.0.0.0/24，如有必要，请替换10.0.0.0/24为您的子网描述。

（2）重新启动NTP服务，并配置开机启动：

systemctl restart chronyd.service
systemctl status chronyd.service
systemctl enable chronyd.service
systemctl list-unit-files |grep chronyd.service

（3）设置时区，同步时间

timedatectl set-timezone Asia/Shanghai
chronyc sources
timedatectl status

0x01安装步骤

1.安装docker环境

所有节点都需要安装docker，
docker官方安装指南
菜鸟教程参考

阿里教程：
https://developer.aliyun.com/article/110806
https://yq.aliyun.com/articles/626118

（1）卸载旧的版本

yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

（2）官方脚本自动安装

手动安装，请参考菜鸟教程的安装指南
通过阿里云安装。

curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun

docker version

systemctl start docker
systemctl status docker
systemctl enable docker
systemctl list-unit-files |grep chronyd.service

（3）配置Docker加速

快速配置

cat > /etc/docker/daemon.json <<EOF
{
    "registry-mirrors":["https://docker.mirrors.ustc.edu.cn/"]
}
EOF

详细情况如下

touch /etc/docker/daemon.json
vi /etc/docker/daemon.json

{
    "registry-mirrors":["https://docker.mirrors.ustc.edu.cn/"]
}

重新启动Docker并查看Docker信息

systemctl daemon-reload
docker info
systemctl restart docker
systemctl status docker

至此，Docker在所有节点安装完成。

2.安装Kubernetes工具

在所有节点安装kubectl、kubelet、kubeadm

（1）端口开放说明

Master主节点

端口范围	用途
6443 *	Kubernetes API server
2379-2380	etcd server client API
10250	Kubelet API
10251	kube-scheduler
10252	kube-controller-manager
10255	Read-only Kubelet API (Heapster)

Node从节点

端口范围	用途
10250	Kubelet API
10255	Read-only Kubelet API (Heapster)
30000-32767	NodePort Services默认端口范围。

参考链接
http://docs.kubernetes.org.cn/457.html
https://www.kubernetes.org.cn/installkubectl
ubuntu：
https://www.cnblogs.com/mouseleo/p/11930784.html
https://blog.csdn.net/shykevin/article/details/98811021

centos：
https://yq.aliyun.com/articles/626118

• kubectl
  kubectl是Kubernetes命令行工具。通过运行kubectl可以部署和管理应用，查看各类资源，创建、删除和更新各种组件。
• kublet
  kublet运行在Cluster的所有节点上，负责启动Pod和容器。
• kubeadm
  kubeadm用于初始化Cluster。

（2）配置Kubernetes的阿里云YUM

参考链接

https://yq.aliyun.com/articles/626118

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

（3）安装kubectl、kubelet、kubeadm

参考指南
https://yq.aliyun.com/articles/626118

使用Kubernetes命令行工具kubectl在Kubernetes上部署和管理应用程序。使用kubectl，可以检查集群资源; 创建，删除和更新组件。
kubectl可作为快速应用程序使用。

yum clean all
yum makecache fast -y
yum install -y kubelet-1.19.3 kubeadm-1.19.3 kubectl-1.19.3

启动kubelet服务

systemctl start kubelet
systemctl status kubelet
systemctl enable kubelet

此时status应该是无法正常启动的，可以忽略，具体的解决措施请继续下一节的部署操作。

systemctl status kubelet

问题解释：kubelet服务启动失败，错误代码255

至此，所有节点，Kubernetes工具安装完成。

3.安装Kubernetes集群

（1）配置 kubelet启动参数

解决上一节的kubulet启动问题

https://www.kubernetes.org.cn/installkubectl

CentOS7安装kubernetes1.11.2图文完整版

用 kubeadm 搭建集群环境

安装完成后，我们还需要对kubelet进行配置，因为用yum源的方式安装的kubelet生成的配置文件将参数–cgroup-driver改成了systemd，而 docker 的cgroup-driver是cgroupfs，这二者必须一致才行，我们可以通过docker info命令查看：

docker info |grep Cgroup

修改文件 kubelet 的配置文件，将其中的KUBELET_CGROUP_ARGS参数更改成cgroupfs：

以前的版本的路径：/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，

操作：

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

当前版本的路径为：/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 

在[service]添加一行Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"

sed -i 'N;4iEnvironment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

修改完成后，重新加载我们的配置文件即可：

systemctl daemon-reload
systemctl restart kubelet
systemctl status kubelet

此时status应该是无法正常启动的，可以忽略，具体的解决措施请继续下一节的部署操作。

systemctl status kubelet

问题解释：kubelet服务启动失败，错误代码255

看Kubernets官方文档，其实里面已经写了很清楚:
The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally.

简单地说就是在kubeadm init 之前kubelet会不断重启。

（2）准备镜像

K8s每个版本需要的镜像版本号在kubeadm Setup Tool Reference Guide这个文档的的Running kubeadm without an internet connection一节里有写。所以可以根据安装的实际版本来调整这个脚本的参数。可以使用命令获取需要的镜像版本如下:
国内拉取google kubernetes镜像

kubeadm config images list

[root@k8s-master ~]# kubeadm config images list
W1106 15:04:10.306351    2206 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.19.3
k8s.gcr.io/kube-controller-manager:v1.19.3
k8s.gcr.io/kube-scheduler:v1.19.3
k8s.gcr.io/kube-proxy:v1.19.3
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0
[root@k8s-master ~]# 

根据镜像仓库获取地址调整脚本，如下:

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.19.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0

sh push.sh

docker images

准备好的镜像如下,与kubeadm config images list时输出的镜像名称版本一致。
我们这里是k8s.gcr.io所以 脚本中pull 时重命名使用的是

docker tag ${username}/${image} k8s.gcr.io/${image}

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.3 k8s.gcr.io/kube-apiserver:v1.19.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.19.3 k8s.gcr.io/kube-controller-manager:v1.19.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.19.3 k8s.gcr.io/kube-scheduler:v1.19.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.19.3 k8s.gcr.io/kube-proxy:v1.19.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0 k8s.gcr.io/etcd:3.4.13-0
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.7.0 k8s.gcr.io/coredns:1.7.0

kubeadm config images list
docker images

（3）初始化Master

到这里我们的准备工作就完成了，接下来我们就可以在master节点上用kubeadm命令来初始化我们的集群了：
master 是控制组件运行的机器，包括etcd和API Server等（使用kubectl CLI与之通信启动）。

参考链接

方法一、使用如下命令

注意：多次运行kubeadm init命令，需要reset

kubeadm reset

kubeadm init --kubernetes-version=v1.19.3 --pod-network-cidr=192.169.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.2.103

参数解释：

– image-repository 指定镜像源，指定为阿里云的源，这样就会避免在拉取镜像超时，如果没问题，过几分钟就能看到成功的日志输入
–kubernetes-version 指定版本
–pod-network-cidr 指定pod网络地址。设置为内网网段！

命令非常简单，就是kubeadm init，后面的参数是需要安装的集群版本，因为我们这里选择flannel作为 Pod 的网络插件，所以需要指定–pod-network-cidr=10.244.0.0/16，然后是 apiserver 的通信地址，这里就是我们 master 节点的 IP 地址。执行上面的命令，如果出现running with swap on is not supported. Please disable swap之类的错误，则我们还需要增加一个参数–ignore-preflight-errors=Swap来忽略 swap 的错误提示信息：

执行情况如下

[root@k8s-master ~]# kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W1106 15:56:46.414897   14265 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.

[root@k8s-master ~]# 
[root@k8s-master ~]# kubeadm init --kubernetes-version=v1.19.3 --pod-network-cidr=192.169.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.2.103
W1106 15:56:52.839787   14278 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.3
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.2.103]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.2.103 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.2.103 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 10.002039 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: do7j65.ku7srin3exqu68p5
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
    --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed 
[root@k8s-master ~]# 

要注意将上面的加入集群的命令保存下面，如果忘记保存上面的 token 和 sha256 值的话也不用担心，我们可以使用下面的命令来查找：

[root@k8s-master ~]# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
do7j65.ku7srin3exqu68p5   23h         2020-11-07T15:57:08+08:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
[root@k8s-master ~]# 

[root@k8s-master ~]# kubeadm init
W1106 15:45:44.005765   11612 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.3
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR Port-6443]: Port 6443 is in use
	[ERROR Port-10259]: Port 10259 is in use
	[ERROR Port-10257]: Port 10257 is in use
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
	[ERROR Port-10250]: Port 10250 is in use
	[ERROR Port-2379]: Port 2379 is in use
	[ERROR Port-2380]: Port 2380 is in use
	[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[root@k8s-master ~]# 

方法二、配置初始化文件kubeadm-init.yaml

查看默认配置文件的编写

kubeadm config print init-defaults > init-defaults.yaml

自行编写初始化配置文件kubeadm-init.yaml
文件内容如下

apiVersion: kubeadm.k8s.io/v1beta2
kind: MasterConfiguration
kubernetesVersion: v1.19.3    # kubernetes的版本
api:
  advertiseAddress: 192.168.2.103 # Master的IP地址
networking:
  podSubnet: 192.168.0.0/16    # pod网络的网段
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers # image的仓库源

编写配置命令

cat <<EOF > /root/kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: MasterConfiguration
kubernetesVersion: v1.19.3
api:
  advertiseAddress: 192.168.2.103
networking:
  podSubnet: 192.168.0.0/16
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
EOF

开始初始化，仅master节点运行

kubeadm init --config kubeadm-init.yaml

（4）使kubectl正常工作

上面的信息记录了 kubeadm 初始化整个集群的过程，生成相关的各种证书、kubeconfig 文件、bootstraptoken 等等，
后边是使用kubeadm join往集群中添加节点时用到的命令.
在初始化的操作中，我们看到Linux推荐使用普通用户执行kubectl命令，因为使用root用户执行会有一些问题。

To start using your cluster, you need to run (as a regular user):

非root用户

su centos	#切换至普通用户

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

执行情况如下：

[root@k8s-master ~]# su centos
[centos@k8s-master root]$ cd
[centos@k8s-master ~]$ pwd
/home/centos
[centos@k8s-master ~]$ mkdir -p $HOME/.kube
[centos@k8s-master ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for centos:
[centos@k8s-master ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
[centos@k8s-master ~]$

root用户

export KUBECONFIG=/etc/kubernetes/admin.conf

执行情况如下：

[centos@k8s-master ~]$ exit
exit
[root@k8s-master ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
[root@k8s-master ~]#

为了使用的便捷，启用kubectl的自动补全功能。
参考：https://blog.csdn.net/shykevin/article/details/98811021

yum install -y bash-completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
source  ~/.bashrc

最后给出了将节点加入集群的命令：
暂时不需要执行

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.103:6443 --token eiocuu.39w009hfxddoktvh \
    --discovery-token-ca-cert-hash sha256:7910428d0ce7ec23d28c05dc9e4a4703da30eece45119d0e4aedd9f71ab6f4d0
[root@k8s-master ~]#

我们根据上面的提示配置好 kubectl 后，就可以使用 kubectl 来查看集群的信息了：

kubectl get cs
kubectl get csr
kubectl get nodes

遇到问题解决记录
https://blog.csdn.net/BigData_Mining/article/details/87887860
https://www.cnblogs.com/potato-chip/p/13973760.html

执行情况如下：

[root@k8s-master ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS      MESSAGE                                                                                       ERROR
scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
etcd-0               Healthy     {"health":"true"}                                                                             
[root@k8s-master ~]# kubectl get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                CONDITION
csr-qlmh7   8m28s   kubernetes.io/kube-apiserver-client-kubelet   system:node:k8s-master   Approved,Issued
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
k8s-master   NotReady   master   8m28s   v1.19.3
[root@k8s-master ~]# 

（5）安装Pod网络

POD网络有多种实现方式
http://docs.kubernetes.org.cn/459.html#34pod

结合初始化《每天五分钟玩转kubernetes》,第3.3.1节

在Master节点

接下来我们来安装flannel网络插件，很简单，和安装普通的 POD 没什么两样：
参考链接

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl apply -f  kube-flannel.yml

或者

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

执行情况如下

[root@k8s-master system]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
--2020-10-16 16:21:02--  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.108.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4813 (4.7K) [text/plain]
Saving to: 'kube-flannel.yml'

100%[================================================================================================>] 4,813       --.-K/s   in 0s

2020-10-16 16:21:03 (15.6 MB/s) - 'kube-flannel.yml' saved [4813/4813]

[root@k8s-master system]# kubectl apply -f  kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@k8s-master system]#

另外需要注意的是如果你的节点有多个网卡的话，需要在 kube-flannel.yml 中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现 dns 无法解析。flanneld 启动参数加上--iface=

args:
- --ip-masq
- --kube-subnet-mgr
- --iface=eth0

安装完成后使用 kubectl get pods 命令可以查看到我们集群中的组件运行状态，如果都是Running 状态的话，那么恭喜你，你的 master 节点安装成功了。

kubectl get pods --all-namespaces

[root@k8s-master system]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-gp6fz             1/1     Running   0          44m
kube-system   coredns-6d56c8448f-rcxf9             1/1     Running   0          44m
kube-system   etcd-k8s-master                      1/1     Running   0          45m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          45m
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          45m
kube-system   kube-flannel-ds-qvgmn                1/1     Running   0          106s
kube-system   kube-proxy-5nbfw                     1/1     Running   0          44m
kube-system   kube-scheduler-k8s-master            1/1     Running   0          45m
[root@k8s-master system]#

（6）添加Node节点

同样的上面的环境配置、docker 安装、kubeadmin、kubelet、kubectl 这些都在Node(10.151.30.62)节点安装配置好过后，我们就可以直接在 Node 节点上执行kubeadm join命令了（上面初始化的时候有），同样加上参数--ignore-preflight-errors=Swap

[root@k8s-master ~]# kubeadm init --kubernetes-version=v1.19.3 --pod-network-cidr=192.169.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=192.168.2.103

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a
regular user:

mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf
$HOME/.kube/config sudo chown $(id-u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster. Run “kubectl apply
-f [podnetwork].yaml” with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following
on each as root:

kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5
–discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed
[root@k8s-master ~]#

在k8s-node1、k8s-node2中分别执行

kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
    --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed 

[root@k8s-node1 ~]# kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
>     --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-node1 ~]# 

[root@k8s-node2 ~]# kubeadm join 192.168.2.103:6443 --token do7j65.ku7srin3exqu68p5 \
>     --discovery-token-ca-cert-hash sha256:f9848158920157dbf87f58383c0b7c28f195d1a4fcad5f205bcb68564f77f4ed
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING Hostname]: hostname "k8s-node2" could not be reached
	[WARNING Hostname]: hostname "k8s-node2": lookup k8s-node2 on 192.168.2.1:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s-node2 ~]# 

Kubernetes-kubectl命令出现错误
【The connection to the server localhost:8080 was refused - did you specify the right host or port?】

ll /etc/kubernetes/kubelet.conf
echo export KUBECONFIG=/etc/kubernetes/kubelet.conf >> ~/.bash_profile
source ~/.bash_profile

4.验证Kubernetes集群

（1）节点校验

kubectl get nodes
kubectl get pods --all-namespaces

[root@k8s-node1 ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   23m     v1.19.3
k8s-node1    Ready    <none>   2m21s   v1.19.3
k8s-node2    Ready    <none>   2m10s   v1.19.3
[root@k8s-node1 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-79ml8             1/1     Running   0          23m
kube-system   coredns-6d56c8448f-cnrjg             1/1     Running   0          23m
kube-system   etcd-k8s-master                      1/1     Running   0          23m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          23m
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          23m
kube-system   kube-flannel-ds-7sgnw                1/1     Running   0          10m
kube-system   kube-flannel-ds-gqn7b                1/1     Running   0          2m21s
kube-system   kube-flannel-ds-krpqf                1/1     Running   0          2m10s
kube-system   kube-proxy-ljc6m                     1/1     Running   0          23m
kube-system   kube-proxy-vbcm2                     1/1     Running   0          2m10s
kube-system   kube-proxy-ws7tf                     1/1     Running   0          2m21s
kube-system   kube-scheduler-k8s-master            1/1     Running   0          23m
[root@k8s-node1 ~]# 

[root@k8s-master ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-79ml8             1/1     Running   0          23m
kube-system   coredns-6d56c8448f-cnrjg             1/1     Running   0          23m
kube-system   etcd-k8s-master                      1/1     Running   0          23m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          23m
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          23m
kube-system   kube-flannel-ds-7sgnw                1/1     Running   0          10m
kube-system   kube-flannel-ds-gqn7b                1/1     Running   0          2m19s
kube-system   kube-flannel-ds-krpqf                1/1     Running   0          2m8s
kube-system   kube-proxy-ljc6m                     1/1     Running   0          23m
kube-system   kube-proxy-vbcm2                     1/1     Running   0          2m8s
kube-system   kube-proxy-ws7tf                     1/1     Running   0          2m19s
kube-system   kube-scheduler-k8s-master            1/1     Running   0          23m
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   24m     v1.19.3
k8s-node1    Ready    <none>   3m9s    v1.19.3
k8s-node2    Ready    <none>   2m58s   v1.19.3
[root@k8s-master ~]# 

[root@k8s-node2 ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   23m     v1.19.3
k8s-node1    Ready    <none>   2m23s   v1.19.3
k8s-node2    Ready    <none>   2m12s   v1.19.3
[root@k8s-node2 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-79ml8             1/1     Running   0          23m
kube-system   coredns-6d56c8448f-cnrjg             1/1     Running   0          23m
kube-system   etcd-k8s-master                      1/1     Running   0          23m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          23m
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          23m
kube-system   kube-flannel-ds-7sgnw                1/1     Running   0          10m
kube-system   kube-flannel-ds-gqn7b                1/1     Running   0          2m23s
kube-system   kube-flannel-ds-krpqf                1/1     Running   0          2m12s
kube-system   kube-proxy-ljc6m                     1/1     Running   0          23m
kube-system   kube-proxy-vbcm2                     1/1     Running   0          2m12s
kube-system   kube-proxy-ws7tf                     1/1     Running   0          2m23s
kube-system   kube-sched

（2）服务校验

kubectl version
systemctl status kubelet

5.常见问题处理

运行kubectl version以验证您安装的版本是否已经是最新。

kubectl version

“The connection to the server localhost:8080 was refused - did you specify the right host or port?”

ll /etc/kubernetes/kubelet.conf 
echo export KUBECONFIG=/etc/kubernetes/kubelet.conf  ~/.bash_profile
source ~/.bash_profile 

阻止自动更新
apt-mark hold kubelet=1.15.2-00 kubeadm=1.15.2-00 kubectl=1.15.2-00`

hold – 标记指定软件包为保留(held back)，阻止软件自动更新

0x02部署应用

0x03Kubernetes Dashboard

1.安装dashboard可视化插件

参考链接：
kubernetes部署dashboard可视化插件
官方Kubernetes Web UI
github项目地址

当前部署dashboard版本：v2.0.4，注意检查dashboard版本与kubernetes版本兼容性：
v2.0.4

Kubernetes version	1.16	1.17	1.18	1.19
Compatibility	?	?	?	✓

• ✓ Fully supported version range.
• ? Due to breaking changes between Kubernetes API versions, some features might not work correctly in the Dashboard.

执行yaml文件直接部署：

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml

[root@k8s-master ~]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.4/aio/deploy/recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[root@k8s-master ~]# 

查看dashboard运行状态，以deployment方式部署，运行2个pod及2个service：

kubectl -n kubernetes-dashboard get pods
kubectl -n kubernetes-dashboard get svc 

[root@k8s-master ~]# kubectl -n kubernetes-dashboard get pods
NAME                                         READY   STATUS              RESTARTS   AGE
dashboard-metrics-scraper-7b59f7d4df-fddnw   1/1     Running             0          22s
kubernetes-dashboard-665f4c5ff-gvsw4         0/1     ContainerCreating   0          22s
[root@k8s-master ~]# kubectl -n kubernetes-dashboard get svc 
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dashboard-metrics-scraper   ClusterIP   10.98.228.65    <none>        8000/TCP   2m45s
kubernetes-dashboard        ClusterIP   10.104.230.65   <none>        443/TCP    2m45s
[root@k8s-master ~]# 

2.配置访问dashboard端口

这里作为演示，使用nodeport方式将dashboard服务暴露在集群外，指定使用30443端口，可自定义：

kubectl  patch svc kubernetes-dashboard -n kubernetes-dashboard \
-p '{"spec":{"type":"NodePort","ports":[{"port":443,"targetPort":8443,"nodePort":30443}]}}'

[root@k8s-master ~]# kubectl  patch svc kubernetes-dashboard -n kubernetes-dashboard \
> -p '{"spec":{"type":"NodePort","ports":[{"port":443,"targetPort":8443,"nodePort":30443}]}}'
service/kubernetes-dashboard patched
[root@k8s-master ~]# 

查看暴露的service,已修改为nodeport类型：

kubectl -n kubernetes-dashboard get svc

[root@k8s-master ~]# kubectl -n kubernetes-dashboard get svc
NAME                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.98.228.65    <none>        8000/TCP        4m37s
kubernetes-dashboard        NodePort    10.104.230.65   <none>        443:30443/TCP   4m37s
[root@k8s-master ~]# 

或者下载yaml文件手动修改service部分，详细请参考的修改service部分。

3.登录dashboard

https://github.com/kubernetes/dashboard/blob/master/docs/user/accessing-dashboard/README.md

https://<dashboard_ip>:30443