CentOS7虚拟机搭建Kubernetes集群（使用私有镜像仓库）

虚拟机安装和基础配置

VisualBox安装三台虚拟机(CentOS7)，一台Mater节点，两台worker节点。
网络设置为双网卡：
网卡1配置成桥接网卡，用来连接外网；
网卡2配置成Host-only网络或内部网络，用来做集群内网。
k8s在创建集群时还需要创建Pod网络，本文使用Calico插件自建网络。
Mater节点CPU至少使用2核，单核会导致k8s报错。
Mater节点修改主机名
hostnamectl --static set-hostname k8s.master
设置网卡2为固定ip（host-only网络默认192.168.56.x段，私有镜像仓库机的网卡2也是该网段）
ip addr或ifconfig查看网卡设备名
（网卡1设备名一般为enp0s3，网卡2设备名一般为enp0s8，请根据实际情况配置）
新增/etc/sysconfig/network-scripts/ifcfg-enp0s8文件，其中：
BOOTPROTO=static # 自动获取ip的话设置为dhcp，固定ip设置为static
以下信息按实际填写，uuid可以通过nmcli connection show指令获得，MAC地址可以从ip addr或ifconfig指令获得。
NAME=k8s_internal
UUID=3b32f83f-abe0-3337-934b-15817cd31acb
DEVICE=enp0s8
ONBOOT=yes # 一定要配成yes，开机启动
IPADDR=192.168.56.111
NETMASK=255.255.255.0
GATEWAY=192.168.56.1
HWADDR=08:00:27:67:9f:0c

[root@k8s_master ~]# cat /etc/sysconfig/network-scripts/ifcfg-enp0s8
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=k8s_internal
UUID=3b32f83f-abe0-3337-934b-15817cd31acb
DEVICE=enp0s8
ONBOOT=yes
IPADDR=192.168.56.111
NETMASK=255.255.255.0
GATEWAY=192.168.56.1
HWADDR=08:00:27:67:9f:0c

重启network或是虚拟机，就ok了。
（按照上面的步骤完成worker节点配置）

yum安装docker-ce，kubeadm，kubelet，kubectl

由于众所周知的原因，必须使用国内源才能保证下载安装顺利......
更新Docker源
在master和worker上执行：

cd /etc/yum.repos.d/
wget -c https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

创建Kubernetes源，kubernetes.repo，输入以下内容保存：

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

安装、激活docker和kubelet
（注：
如果你的仓库里有k8s的旧版本镜像并且要使用，那么安装kubeadm kubelet kubectrl时必须指定跟旧版本镜像版本号一致的版本进行安装。
否则镜像与kube程序版本不一致，无法初始化集群）

yum install docker-ce
yum install kubeadm kubelet kubectl
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet

修改系统参数。

#SELINUX
在文件中/etc/selinux/config 修改 SELINUX=permissive
#Firewall
systemctl disable firewalld && systemctl stop firewalld
#网络参数设置
sysctl -w net.bridge.bridge-nf-call-iptables=1
在文件中/etc/sysctl.d/k8s.conf 添加一行 net.bridge.bridge-nf-call-iptables=1
#关闭SWAP
swapoff -a 
在文件中/etc/fstab中将swap所在行注释掉

修改/etc/hosts，将ip-域名对应关系添加进去。
192.168.56.111 k8s.master
192.168.56.112 k8s.nodea
192.168.56.113 k8s.nodeb
192.168.56.110 registry.phoenix.com
 

配置k8s Master节点

在master节点执行kubeadm config images list，然后在镜像仓库机中下载输出的镜像:版本。
(再次注意kube-xxx的版本，所使用的image必须与所安装的kubeadm版本（kubeadm verison）一致，否则init失败(因为版本对不上，后面见log)）：

[root@k8s ~]# kubeadm config images list
W0628 20:05:55.532126    4127 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0628 20:05:55.532420    4127 version.go:99] falling back to the local client version: v1.15.0
k8s.gcr.io/kube-apiserver:v1.15.0
k8s.gcr.io/kube-controller-manager:v1.15.0
k8s.gcr.io/kube-scheduler:v1.15.0
k8s.gcr.io/kube-proxy:v1.15.0
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

在镜像仓库机准备如下shell脚本文件并运行，到阿里云的镜像库pull所需镜像。
之后使用docker image ls可以看到下载好的镜像。

#! /bin/bash
images=(
  kube-apiserver:v1.15.0
  kube-controller-manager:v1.15.0
  kube-scheduler:v1.15.0
  kube-proxy:v1.15.0
  pause:3.1
  etcd:3.3.10
  coredns:1.3.1
)

for imageName in ${images[@]} ; do
  docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done

确认image已下载。
docker image ls

[root@webserver ~]# docker images
REPOSITORY                                                                    TAG                 IMAGE ID            CREATED             SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.15.0             d235b23c3570        8 days ago          82.4MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.15.0             201c7a840312        8 days ago          207MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.15.0             2d3813851e87        8 days ago          81.1MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.15.0             8328bb49b652        8 days ago          159MB
registry                                                                      latest              f32a97de94e1        3 months ago        25.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   1.3.1               eb516548c180        5 months ago        40.3MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.3.10              2c4adeb21b4f        6 months ago        258MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.1                 da86e6ba6ca1        18 months ago       742kB

（将下好的镜像push到配置好的私有镜像仓库中）
在镜像仓库机上运行：

#! /bin/bash
images=(
  kube-proxy:v1.15.0
  kube-apiserver:v1.15.0
  kube-controller-manager:v1.15.0
  kube-scheduler:v1.15.0
  coredns:1.3.1
  etcd:3.3.10
  pause:3.1
)

for imageName in ${images[@]} ; do
  docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName registry.phoenix.com:5000/k8s.gcr.io/$imageName
  docker push registry.phoenix.com:5000/k8s.gcr.io/$imageName
done

[root@webserver ~]# ./k8s_images_to_private_registry.sh
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/kube-proxy]
2f35b77d8045: Pushed
15c9248be8a9: Pushed
fe9a8b4f1dcc: Pushed
v1.15.0: digest: sha256:7b94921f1c64876d3663698ade724fce79b417b32f0e1053976ca68a18fc0cba size: 951
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/kube-apiserver]
55ea3e06e6da: Pushed
fe9a8b4f1dcc: Layer already exists
v1.15.0: digest: sha256:3f74f63d11e1643818ca0dd835a649fbe78883b402f0b7cf4a9138918edc9bac size: 741
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/kube-controller-manager]
50c95cd2e602: Pushed
fe9a8b4f1dcc: Layer already exists
v1.15.0: digest: sha256:ddee337acd50f05781dcb0cb1f67a7695da33580c7d7a86f22ecdea623c11c46 size: 741
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/kube-scheduler]
4c6646e96ecc: Pushed
fe9a8b4f1dcc: Layer already exists
v1.15.0: digest: sha256:caeaa9ad9caed1f255ad30773a2b14d2bce9c821a473de03fe9bed8f9a7e696e size: 741
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/coredns]
c6a5fc8a3f01: Layer already exists
fb61a074724d: Layer already exists
1.3.1: digest: sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b size: 739
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/etcd]
6fbfb277289f: Layer already exists
30796113fb51: Layer already exists
8a788232037e: Layer already exists
3.3.10: digest: sha256:240bd81c2f54873804363665c5d1a9b8e06ec5c63cfc181e026ddec1d81585bb size: 950
The push refers to repository [registry.phoenix.com:5000/k8s.gcr.io/pause]
e17133b79956: Layer already exists
3.1: digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d size: 527

用py脚本查看私有仓库镜像：

[root@webserver ~]# python list_private_images.py registry.phoenix.com 5000
registry.phoenix.com:5000/hello-world:latest
registry.phoenix.com:5000/k8s.gcr.io/coredns:1.3.1
registry.phoenix.com:5000/k8s.gcr.io/etcd:3.3.10
registry.phoenix.com:5000/k8s.gcr.io/kube-apiserver:v1.15.0
registry.phoenix.com:5000/k8s.gcr.io/kube-controller-manager:v1.15.0
registry.phoenix.com:5000/k8s.gcr.io/kube-proxy:v1.15.0
registry.phoenix.com:5000/k8s.gcr.io/kube-scheduler:v1.15.0
registry.phoenix.com:5000/k8s.gcr.io/pause:3.1

master主机从私有仓库pull镜像到本地：
（
如果私有镜像仓库使用的是http方式，则需要修改master、node机的/usr/lib/systemd/system/docker.service，将：
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
改为：
ExecStart=/usr/bin/dockerd --insecure-registry registry.phoenix.com:5000
然后令配置生效：
systemctl daemon-reload && systemctl restart docker
如果私有镜像仓库使用的是https方式，则需要master、node机安装镜像仓库的自签发ca证书，然后docker login登录私有镜像仓库。
）
master主机准备以下脚本：

#! /bin/bash
images=(
  k8s.gcr.io/kube-proxy:v1.15.0
  k8s.gcr.io/kube-apiserver:v1.15.0
  k8s.gcr.io/kube-controller-manager:v1.15.0
  k8s.gcr.io/kube-scheduler:v1.15.0
  k8s.gcr.io/coredns:1.3.1
  k8s.gcr.io/etcd:3.3.10
  k8s.gcr.io/pause:3.1
)

for imageName in ${images[@]} ; do
  docker pull registry.phoenix.com:5000/$imageName
  docker tag registry.phoenix.com:5000/$imageName $imageName
done

运行k8s时，镜像名必须为k8s.gcr.io开头，所以要再tag一遍。
运行脚本：

[root@k8s ~]# ./k8s_pull_images_from_private_registry.sh
v1.15.0: Pulling from k8s.gcr.io/kube-proxy
39fafc05754f: Already exists
db3f71d0eb90: Already exists
b593bfa65f6f: Pull complete
Digest: sha256:7b94921f1c64876d3663698ade724fce79b417b32f0e1053976ca68a18fc0cba
Status: Downloaded newer image for registry.phoenix.com:5000/k8s.gcr.io/kube-proxy:v1.15.0
v1.15.0: Pulling from k8s.gcr.io/kube-apiserver
39fafc05754f: Already exists
b9be38fe6288: Pull complete
Digest: sha256:3f74f63d11e1643818ca0dd835a649fbe78883b402f0b7cf4a9138918edc9bac
Status: Downloaded newer image for registry.phoenix.com:5000/k8s.gcr.io/kube-apiserver:v1.15.0
v1.15.0: Pulling from k8s.gcr.io/kube-controller-manager
39fafc05754f: Already exists
7c25227b10c9: Pull complete
Digest: sha256:ddee337acd50f05781dcb0cb1f67a7695da33580c7d7a86f22ecdea623c11c46
Status: Downloaded newer image for registry.phoenix.com:5000/k8s.gcr.io/kube-controller-manager:v1.15.0
v1.15.0: Pulling from k8s.gcr.io/kube-scheduler
39fafc05754f: Already exists
0d46767881f9: Pull complete
Digest: sha256:caeaa9ad9caed1f255ad30773a2b14d2bce9c821a473de03fe9bed8f9a7e696e
Status: Downloaded newer image for registry.phoenix.com:5000/k8s.gcr.io/kube-scheduler:v1.15.0
1.3.1: Pulling from k8s.gcr.io/coredns
Digest: sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b
Status: Image is up to date for registry.phoenix.com:5000/k8s.gcr.io/coredns:1.3.1
3.3.10: Pulling from k8s.gcr.io/etcd
Digest: sha256:240bd81c2f54873804363665c5d1a9b8e06ec5c63cfc181e026ddec1d81585bb
Status: Image is up to date for registry.phoenix.com:5000/k8s.gcr.io/etcd:3.3.10
3.1: Pulling from k8s.gcr.io/pause
Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
Status: Image is up to date for registry.phoenix.com:5000/k8s.gcr.io/pause:3.1

master主机查看镜像：

[root@k8s ~]# docker images
REPOSITORY                                                     TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                                          v1.15.0             d235b23c3570        8 days ago          82.4MB
registry.phoenix.com:5000/k8s.gcr.io/kube-proxy                v1.15.0             d235b23c3570        8 days ago          82.4MB
k8s.gcr.io/kube-apiserver                                      v1.15.0             201c7a840312        8 days ago          207MB
registry.phoenix.com:5000/k8s.gcr.io/kube-apiserver            v1.15.0             201c7a840312        8 days ago          207MB
k8s.gcr.io/kube-scheduler                                      v1.15.0             2d3813851e87        8 days ago          81.1MB
registry.phoenix.com:5000/k8s.gcr.io/kube-scheduler            v1.15.0             2d3813851e87        8 days ago          81.1MB
k8s.gcr.io/kube-controller-manager                             v1.15.0             8328bb49b652        8 days ago          159MB
registry.phoenix.com:5000/k8s.gcr.io/kube-controller-manager   v1.15.0             8328bb49b652        8 days ago          159MB
k8s.gcr.io/coredns                                             1.3.1               eb516548c180        5 months ago        40.3MB
registry.phoenix.com:5000/k8s.gcr.io/coredns                   1.3.1               eb516548c180        5 months ago        40.3MB
k8s.gcr.io/etcd                                                3.3.10              2c4adeb21b4f        6 months ago        258MB
registry.phoenix.com:5000/k8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        6 months ago        258MB
k8s.gcr.io/pause                                               3.1                 da86e6ba6ca1        18 months ago       742kB
registry.phoenix.com:5000/k8s.gcr.io/pause                     3.1                 da86e6ba6ca1        18 months ago       742kB

从master主机上可以看到，镜像都已经下好，并改好了名字。

初始化Master节点（使用kubeadm reset可以回滚）

–apiserver-advertise-address 192.168.56.111 ## master节点的内网地址
–pod-network-cidr=10.244.0.0/16 ## pod网络地址

kubeadm init --apiserver-advertise-address 192.168.56.111 --pod-network-cidr=10.244.0.0/16

初始化成功的log：

[root@k8s ~]# kubeadm init --apiserver-advertise-address 192.168.56.111 --pod-network-cidr=10.244.0.0/16
W0628 21:12:43.920083    7559 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0628 21:12:43.920407    7559 version.go:99] falling back to the local client version: v1.15.0
[init] Using Kubernetes version: v1.15.0
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s.master localhost] and IPs [192.168.56.111 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s.master localhost] and IPs [192.168.56.111 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s.master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.111]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 57.010881 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s.master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s.master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 8sbeyh.tm1j31jfnh1k06gl
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.111:6443 --token 8sbeyh.tm1j31jfnh1k06gl \
    --discovery-token-ca-cert-hash sha256:771aed98bd64190e983f804e2afba9a69ee3a883f67487ffb2daa0b00a396333
[root@k8s ~]#

下面是ctrl-c/ctrl-v来的kubeadm init解释：

kubeadm-init
kubeadm是什么
首先，kubeadm是一个构建k8s集群的工具。它提供的kubeadm init和kubeadm join两个命令是快速构建k8s集群的最佳实践。 其次，kubeadm工具只为构建最小可用集群，它只关心集群中最基础的组件，至于其他的插件（比如dashboard、CNI等）则不会涉及。 不过，从我的经验来看，这些已经足够。

通过kubeadm init [flags]命令可以启动一个master节点。其中[flags]代表了kubeadm init命令可以传入的参数。

如果查看官方介绍，可以发现kubeadm init可以传入的参数比较多，这里只简单介绍几种（其他的大家可以到 这里查看 ）：

–apiserver-bind-port int32 Default: 6443 可以通过这个参数指定API-server的工作端口，默认是6443。
–config string 可以通过这个命令传入kubeadm的配置文件，需要注意的是，这个参数是实验性质的，不推荐使用。
–dry-run 带了这个参数后，运行命令将会把kubeadm做的事情输出到标准输出，但是不会实际部署任何东西。强烈推荐！
-h, --help 输出帮助文档。
–node-name string 指定当前节点的名称。
–pod-network-cidr string 通过这个值来设定pod网络的IP地址网段；设置了这个值以后，控制平面会自动给每个节点设置CIDRs（无类别域间路由，Classless Inter-Domain Routing）。
–service-cidr string Default: “10.96.0.0/12” 设置service的CIDRs，默认为 10.96.0.0/12。
–service-dns-domain string Default: “cluster.local” 设置域名称后缀，默认为cluster.local。
其他参数。
kubeadm-init的工作流
在运行了 kubeadm init 命令以后，都进行了那些操作呢？这里主要就是跟着官方文档来翻译一遍了：

首先会运行一系列预检代码来检查系统的状态；大部分的检查只会抛出一个警告，也有一部分会抛出异常错误从而导致工作流推出（比如没有关闭swap或者没有安装docker）。官方给出一个参数–ignore-preflight-errors=， 我估计八成大家用不到，除非真的明白自己在做啥。。。
生成一个用来认证k8s组件间调用的自签名的CA（Certificate Authority，证书授权）；这个证书也可以通过–cert-dir（默认是/etc/kubernetets/pki）的方式传入，那么这一步就会跳过。
把kubelet、controller-manager和scheduler等组件的配置文件写到/etc/kubernets/目录，这几个组件会使用这些配置文件来连接API-server的服务；除了上面几个配置文件，还会生成一个管理相关的admin.conf文件。
如果参数中包含–feature-gates=DynamicKubeletConfig，会把kubelet的初始化配置文件写入/var/lib/kubelet/config/init/kubelet这个文件；官方给出一坨文字解释，这里先不探究了，因为我没有用到。。。
接下来就是创建一些 静态pod 的配置文件了，包括API-server、controller-manager和scheduler。假如没有提供外部etcd，还会另外生成一个etcd的静态Pod配置文件。这些静态pod会被写入/etc/kubernetes/manifests，kubelet进程会监控这个目录，从而创建相关的pod。
假如第五步比较顺利，这个时候k8s的控制面进程（api-server、controller-manager、scheduler）就全都起来了。
如果传入了参数–feature-gates=DynamicKubeletConfig，又会对kubelet进行一坨操作，因为没有用到，所以这里不做详细探究。
给当前的节点（Master节点）打label和taints，从而防止其他的负载在这个节点运行。
生成token，其他节点如果想加入当前节点（Master）所在的k8s集群，会用到这个token。
进行一些允许节点以 Bootstrap Tokens) 和 TLS bootstrapping 方式加入集群的必要的操作：
设置RBAC规则，同时创建一个用于节点加入集群的ConfigMap（包含了加入集群需要的所有信息）。
让Bootstrap Tokens可以访问CSR签名的API。
给新的CSR请求配置自动认证机制。
通过API-server安装DNS服务器（1.11版本后默认为CoreDNS，早期版本默认为kube-dns）和kube-proxy插件。这里需要注意的是，DNS服务器只有在安装了CNI（flannel或calico）之后才会真正部署，否则会处于挂起（pending）状态。

两段初始化失败的log：
因为主机命名问题导致失败…之前hostname命名为：k8s_master，而k8s不认下划线。

[root@k8s_master ~]# kubeadm init --apiserver-advertise-address 192.168.56.111 --pod-network-cidr=10.244.0.0/16
W0628 16:52:28.956823   10287 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0628 16:52:28.957092   10287 version.go:99] falling back to the local client version: v1.15.0
name: Invalid value: "k8s_master": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

因为kube与镜像的版本不一致导致kubeadm去官网拉取同版本的镜像而失败。
一开始下载的是1.14.3版本镜像，而yum安装时，kube没指定版本，装的是最新版1.15.0，初始化时在本地找不到1.15.0的镜像，就尝试去官网下载（能下得到才怪…）

[root@k8s ~]# kubeadm init --apiserver-advertise-address 192.168.56.111 --pod-network-cidr=10.244.0.0/16
W0628 17:01:53.524982   13185 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0628 17:01:53.525861   13185 version.go:99] falling back to the local client version: v1.15.0
[init] Using Kubernetes version: v1.15.0
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.15.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.15.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.15.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.15.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
[root@k8s ~]#

初始化成功后，按log输出的内容，还有点事情要做：

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.111:6443 --token 8sbeyh.tm1j31jfnh1k06gl \
    --discovery-token-ca-cert-hash sha256:771aed98bd64190e983f804e2afba9a69ee3a883f67487ffb2daa0b00a396333

执行：

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

需要这些配置命令的原因是:Kubernetes 集群默认需要加密方式访问。所以，这几条命令，就是 将刚刚部署生成的 Kubernetes 集群的安全配置文件，保存到当前用户的.kube 目录下，kubectl 默 认会使用这个目录下的授权信息访问 Kubernetes 集群。
如果不这么做的话，我们每次都需要通过 export KUBECONFIG 环境变量告诉 kubectl 这个安全配 置文件的位置。

创建podnetwork

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

选择需要的集群网络方案:flannel或calico

这里选择使用calico：

kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml

安装完毕后，查看docker images里会发现多了三个镜像：（这三个镜像应该可以提前准备好，在https://docs.projectcalico.org/v3.7/manifests/calico.yaml里面查看版本号，下载这个yaml到本地，按image名搜索就能搜到）

calico/node                                                    v3.7.3              bf4ff15c9db0        2 weeks ago         156MB
calico/cni                                                     v3.7.3              1a6ade52d471        2 weeks ago         135MB
calico/kube-controllers                                        v3.7.3              283860d96794        2 weeks ago         46.8MB

记录token，worker加入master时要用。
（如果安装完master节点后24小时内没有将work加入，则需要重新生成token。
kubeadm token create 生成
kubeadm token list 查看
然后使用kubeadm join加入）

kubeadm join 192.168.56.111:6443 --token 8sbeyh.tm1j31jfnh1k06gl \
    --discovery-token-ca-cert-hash sha256:771aed98bd64190e983f804e2afba9a69ee3a883f67487ffb2daa0b00a396333

初始化worker节点

检查hostname，没修改的话记得修改。
hostnamectl --static set-hostname k8s.nodea
hostnamectl --static set-hostname k8s.nodeb
检查/etc/hosts，添加ip-域名/主机名关系。
查看master节点的docker images，然后在worker节点上拉取下面这几个镜像：（join成功后会自动拉取calico的两个镜像，但是速度太慢，造成一直在pending，cni无法初始化的错觉。应该可以提前拉好，注意跟master节点的版本保持一致。如果calico/node、calico/cni两个镜像在私有镜像仓库已经准备好，另外两个镜像在私有仓库里已经存在了，这样worker节点提前从私有库pull下来就好，速度也能快很多。（此处脚本省略）

k8s.gcr.io/kube-proxy                             v1.15.0
calico/node                                       v3.7.3
calico/cni                                        v3.7.3
k8s.gcr.io/pause                                  3.1

加入master
在nodea和nodeb上执行（使用kubeadm reset可以回滚）

kubeadm join 192.168.56.111:6443 --token 8sbeyh.tm1j31jfnh1k06gl \
    --discovery-token-ca-cert-hash sha256:771aed98bd64190e983f804e2afba9a69ee3a883f67487ffb2daa0b00a396333

执行LOG：

[root@k8s ~]# kubeadm join 192.168.56.111:6443 --token 8sbeyh.tm1j31jfnh1k06gl \
>     --discovery-token-ca-cert-hash sha256:771aed98bd64190e983f804e2afba9a69ee3a883f67487ffb2daa0b00a396333
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@k8s ~]#

cni配置好后，在master节点执行kubectl get nodes查看集群状态都是Ready：

[root@k8s ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s.master   Ready    master   71m   v1.15.0
k8s.nodea    Ready       30m   v1.15.0
k8s.nodeb    Ready       29m   v1.15.0

kubectl get pods --all-namespaces

[root@k8s ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-6fb584dd97-z229f   1/1     Running   0          61m
kube-system   calico-node-lpw8c                          1/1     Running   1          61m
kube-system   calico-node-s9s5z                          1/1     Running   0          37m
kube-system   calico-node-v4vhw                          1/1     Running   0          38m
kube-system   coredns-5c98db65d4-874kc                   1/1     Running   0          78m
kube-system   coredns-5c98db65d4-grhth                   1/1     Running   0          78m
kube-system   etcd-k8s.master                            1/1     Running   0          77m
kube-system   kube-apiserver-k8s.master                  1/1     Running   0          77m
kube-system   kube-controller-manager-k8s.master         1/1     Running   0          77m
kube-system   kube-proxy-4lgzl                           1/1     Running   0          38m
kube-system   kube-proxy-ppgc6                           1/1     Running   0          78m
kube-system   kube-proxy-xxlm6                           1/1     Running   0          37m
kube-system   kube-scheduler-k8s.master                  1/1     Running   0          77m

感谢以下blogger分享文章：
https://totozhang.github.io/2019-05-10-learning-k8s-2/
https://blog.csdn.net/qq_24973447/article/details/88069601
https://blog.csdn.net/m0_37556444/article/details/86494791