使用Kubeadm搭建Kubernetes(1.13.1)集群

Kubeadm是Kubernetes官方提供的用于快速安装Kubernetes集群的工具，伴随Kubernetes每个版本的发布都会同步更新，在2018年将进入GA状态，说明离生产环境中使用的距离越来越近了。

使用Kubeadm搭建Kubernetes集群本来是件很简单的事，但由于众所周知的原因，在中国大陆是无法访问 k8s.gcr.io的。这就使我们无法按照官方的教程来创建集群。而国内的教程参差不齐，大多也无法运行成功，我也是踩了很多坑，才部署成功，故在此分享出来。
另外，我把相关的命令简单整合了一下，写成了两个脚本，分别用来部署Master和Worker Node，用起来非常方便。Kubernetes自动化部署脚本

准备

安装环境准备

IP地址	主机名	CPU	内存	磁盘
10.136.157.23	kube-node1	32C	128G	2T
10.136.157.24	kube-node2	32C	128G	2T
10.136.157.25	kube-node3	32C	128G	2T

关闭防火墙和selinux

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 禁用SELINUX
setenforce 0

vim /etc/selinux/config
SELINUX=disabled

关闭系统的Swap

Kubernetes 1.8开始要求必须禁用Swap，如果不关闭，默认配置下kubelet将无法启动。

# 关闭系统的Swap方法如下:
# 编辑`/etc/fstab`文件，注释掉引用`swap`的行，保存并重启后输入:
sudo swapoff -a

验证Mac地址和product_uuid是否唯一（可选）

Kubernetes要求集群中所有机器具有不同的Mac地址、产品uuid、Hostname。可以使用如下命令查看：

# UUID
cat /sys/class/dmi/id/product_uuid

# Mac地址
ip link

# Hostname
cat /etc/hostname

安装Docker

Kubernetes从1.6开始使用CRI(Container Runtime Interface)容器运行时接口。默认的容器运行时仍然是Docker，是使用kubelet中内置dockershim CRI来实现的。

sudo yum install -y docker

安装kubeadm, kubelet 和 kubectl

部署之前，我们需要安装一下三个包：

• kubeadm: 引导启动k8s集群的命令行工具。
• kubelet: 在群集中所有节点上运行的核心组件, 用来执行如启动pods和containers等操作。
• kubectl: 操作集群的命令行工具。

修改yum源

Kubernetes国内yum源

安装kubeadm, kubelet 和 kubectl

查看可用版本

sudo yum list --showduplicates | grep 'kubeadm\|kubectl\|kubelet'

安装 kubeadm1.13.1, kubelet1.13.1 和 kubectl1.13.1

yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 --disableexcludes=kubernetes

此时还不能启动 kubelet，先设置开机启动：

systemctl enable kubelet

初始化 Master 节点

配置文件

# 生成配置文件
kubeadm config print-default > kubeadm.conf 
# 修改配置文件
# 修改镜像仓储地址
sed -i "s#imageRepository: .*#imageRepository: registry.cn-beijing.aliyuncs.com/imcto#g" kubeadm.conf
# 修改版本号
sed -i "s/kubernetesVersion: .*/kubernetesVersion: v1.13.1/g" kubeadm.conf
sed -i "s/podSubnet: .*/podSubnet: \"10.244.0.0\/16\"/g" kubeadm.conf

初始化

拉取镜像

$ sudo kubeadm config images pull --config kubeadm.conf
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-apiserver:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-controller-manager:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-scheduler:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-proxy:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/pause:3.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/etcd:3.2.24
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/coredns:1.2.6

可以看到，已经成功拉取了需要的镜像

开始初始化

此处可能会遇到一个坑 安装Kubernetes报错：The kubelet is not running

$ sudo kubeadm init --config kubeadm.conf
[init] Using Kubernetes version: v1.13.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube-node1 localhost] and IPs [10.136.157.23 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube-node1 localhost] and IPs [10.136.157.23 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube-node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.136.157.23]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.501577 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node1" as an annotation
[mark-control-plane] Marking the node kube-node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kube-node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.136.157.23:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952

开启8080端口

# 修改配置文件
sudo sed -i 's/insecure-port=0/insecure-port=8080/g' /etc/kubernetes/manifests/kube-apiserver.yaml
# 重启docker镜像
sudo docker ps |grep 'kube-apiserver_kube-apiserver'|awk '{print $1}'|head -1|xargs sudo docker restart

设置 Node 节点加入集群

kubeadm init 命令的输出日志中已经告知我们加入新节点的方式。如果忘记了 Master 的 Token，可以在 Master 上输入以下命令查看：

$ sudo kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
abcdef.0123456789abcdef   9h        2019-03-22T19:35:14+08:00   authentication,signing           system:bootstrappers:kubeadm:default-node-token

默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Token

kubeadm token create

获取ca证书sha256编码hash值

$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952

加入新的Node节点

$ sudo kubeadm join 10.136.157.23:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.136.157.23:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.136.157.23:6443"
[discovery] Requesting info from "https://10.136.157.23:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.136.157.23:6443"
[discovery] Successfully established connection with API Server "10.136.157.23:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node2" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

启用IPV6

注意：需要在新加入的Node上启用IPV6，否则会导致 coredns 启动失败，始终处于 ContainerCreating 状态

sudo sed -i 's\ipv6.disable=1\ipv6.disable=0\g' /etc/default/grub
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

安装网络插件

为了让Pods间可以相互通信，我们必须安装一个网络插件，并且必须在部署任何应用之前安装，CoreDNS也是在网络插件安装之后才会启动的。

网络的插件完整列表，请参考 Networking and Network Policy。

在安装之前，我们先查看一下当前Pods的状态：

$ sudo kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE
kube-system   coredns-58d6b7c8d7-f5mdg             0/1       Pending   0          57m
kube-system   coredns-58d6b7c8d7-fkpmc             0/1       Pending   0          57m
kube-system   etcd-kube-node1                      1/1       Running   1          56m
kube-system   kube-apiserver-kube-node1            1/1       Running   7          10m
kube-system   kube-controller-manager-kube-node1   1/1       Running   2          56m
kube-system   kube-proxy-2xrzt                     1/1       Running   1          57m
kube-system   kube-scheduler-kube-node1            1/1       Running   2          56m

如上，可以看到CoreDND的状态是Pending，就是因为我们还没有安装网络插件。

安装RBAC

wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
sudo kubectl apply -f rbac.yaml

安装Flannel

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work

$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ sudo kubectl apply -f kube-flannel.yml
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

通过 ifconfig 可以看到新建了一张虚拟网卡：

$ ifconfig
flannel.1: flags=4163  mtu 1450
        inet 10.244.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 7a:d1:26:11:ca:ae  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Master隔离（可选）

默认情况下，由于安全原因，集群并不会将pods部署在Master节点上。但是在开发环境下，我们可能就只有一个Master节点，这时可以使用下面的命令来解除这个限制：

$ sudo kubectl taint nodes --all node-role.kubernetes.io/master-
## 输出
node/ubuntu1 untainted

等待一会，我们可以在Master节点上使用kubectl get nodes命令来查看节点的状态：

$ sudo kubectl get nodes
NAME         STATUS     ROLES    AGE    VERSION
kube-node1   Ready   master   3h3m   v1.13.3
kube-node2   Ready      118s   v1.13.3
kube-node3   Ready      95s    v1.13.3

安装Dashboard UI

Dashboard默认是不安装的，要想安装Dashboard，可以执行下面的命令：

$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml

# 修改kubernetes-dashboard.yaml
        spec:
      containers:
      - name: kubernetes-dashboard
        image: registry.cn-beijing.aliyuncs.com/imcto/kubernetes-dashboard-amd64:v1.10.1
# 创建Dashboard UI
$ sudo kubectl create -f kubernetes-dashboard.yaml

注意：如果没有必要，不用解除 --apiserver-host 的注释

访问Dashboard UI

修改kubernetes-dashboard配置

$ kubectl -n kube-system edit service kubernetes-dashboard

将type: ClusterIP 修改为 type: NodePort，并保存文件。
检查 Dashboard UI 暴露的服务端口

$ sudo kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.110.146.196           443:32275/TCP   42m

可以看到，Dashboard UI 的服务端口是32275，协议为HTTPS，我们可以https://:32275来进行访问。

遇到的问题

初始化 Kubernetes 问题（端口占用）
安装Kubernetes报错：The connection to the server localhost:8080 was refused
安装Kubernetes报错：STATUS NotReady
安装Kubernetes报错：/proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
安装Kubernetes报错：kubernetes-dashboard状态CrashLoopBackOff
kubelet启动报错：node "kube-master1" not found

参考文章

Centos7 使用 kubeadm 安装Kubernetes 1.13.3
使用Kubeadm搭建Kubernetes(1.12.2)集群
在国内如何巧妙获取kubernetes各镜像？
国内借助阿里云快速获取gcr.io上的镜像
解决kubeadm部署kubernetes集群镜像问题
kubeadm 生成的token过期后，集群增加节点
kubeadm 安装 kubernetes
Kubernetes自动化部署脚本