Kubernetes集群管理 --2双机安装Kubernetes集群

1. 集群环境 (for Master + Worker)

现集群中有两台机器，机器通过NAT技术共享同一个外网IP。内网IP地址分别为192.168.0.103和192.168.0.106。两台机器的操作系统均为Ubuntu16.04 LTS。两台机器之间可以互相ping通。

下面，我们把192.168.0.103作为Kubernetes Master节点，把192.168.0.106作为Kubernetes Worker节点。下面例子介绍如何手动构建起一个包含两节点的Kubernetes集群。

2. 清理环境 (for Master + Worker)

如果之前有安装过Kubernetes，执行“清理环境”的操作，卸载之前安装过的Kubeadm及其依赖项；反之，如果第一次安装Kubernetes，可以跳过此步。

~$ sudo kubeadm reset
~$ sudo apt-get purge kubeadm kubectl kubelet kubernetes-cni kube*   
~$ sudo apt-get autoremove  
~$ sudo rm -rf ~/.kube
~$ sudo rm /usr/local/bin/kubectl

3. 准备环境 (for Master + Worker)

所有机器上，关闭防火墙

~$ systemctl stop firewalld
~$ systemctl disable firewalld
~$ sudo ufw status
Status: inactive

所有机器上，关闭selinux

~$ sed -i 's/enforcing/disabled/' /etc/selinux/config
~$ setenforce 0
~$ getenforce
Disabled

所有机器上，清空iptables的规则，查看iptables的路由转发规则值均清空

~$ sudo iptables -F
~$ sudo iptables -X
~$ sudo iptables -Z
~$ sudo iptables -vnL
Chain INPUT (policy ACCEPT 92 packets, 8043 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 83 packets, 7026 bytes)
 pkts bytes target     prot opt in     out     source               destination

所有机器上，关闭swap

~$ sudo swapoff -a $ 关闭进程
~$ vim /etc/fstab $ 开机不启动进程
~$ free $ 查看Swap一行值均为0
joe@ubuntu00:~$ free
              total        used        free      shared  buff/cache   available
Mem:       12055260      416996     7600548       44908     4037716    11163312
Swap:             0           0           0 

所有机器上，测试机器之间能否通过内网IP访问

~$ ping -c 3 192.168.0.103
~$ ping -c 3 192.168.0.106

所有机器上，将桥接的IPv4流量传递到iptables的链

~$ sudo su -
~# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
~# sysctl --system $ 生效
~# exit
logout

4. 安装Docker-ce (for Master + Worker)

根据官方文档安装docker-ce，进行稳定版docker-ce的安装

~$ sudo su
~# apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common
~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
~# add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
~# apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu
~# cat > /etc/docker/daemon.json <

5. 安装Kubeadm (for Master + Worker)

根据官方文档安装Kubeadm，进行Kubeadm的安装

~$ sudo su
~# apt-get update && apt-get install -y apt-transport-https curl
~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
~# cat </etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
~# apt-get update
~# apt-get install -y kubelet kubeadm kubectl
~# apt-mark hold kubelet kubeadm kubectl
~# exit
exit

6. 初始化Kubernetes Master (Only for Master Node)

6.1 配置Master节点 (Only for Master Node)

注意：Master节点的内网IP地址192.168.0.103记得更改

~$ sudo su
~# kubeadm init --pod-network-cidr=10.244.10.0/16 --apiserver-advertise-address=192.168.0.103

注意：安装成功之后，复制屏幕显示的提示信息如下，

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.103:6443 --token r5qgpn.m950jgbdbln80a8p \
    --discovery-token-ca-cert-hash sha256:2f348b99af4d6d23bebf95937744ef93f463e7bf395d0f3fab141e5b9243200f

安装成功之后，根据提示信息，在用户home目录下新建文件夹和文件

~$ mkdir -p $HOME/.kube
~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
~$ ls .kube/
config

同时复制命令2，新的worker节点加入时备用，也可以通过命令kubeadm token create --print-join-command查询

kubeadm join 192.168.0.103:6443 --token r5qgpn.m950jgbdbln80a8p \
    --discovery-token-ca-cert-hash sha256:2f348b99af4d6d23bebf95937744ef93f463e7bf395d0f3fab141e5b9243200f

显示当前master节点，如果无法使用kubectl命令，请使用官方文档安装kubectl

~$ kubectl get node
NAME       STATUS     ROLES    AGE   VERSION
ubuntu00   NotReady   master   21m   v1.14.1

在没有装容器Pod网络之前，master节点的status为notready.

6.2 安装Pod网络 (Only for Master Node)

Master节点安装Kubernetes Flannel网络如下

~$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

查看kubernetes pods的运行状态

~$ kubectl get pods -n kube-system

出现coredns pod启动问题，解决方法分三步

NAME                               READY   STATUS             RESTARTS   AGE
coredns-fb8b8dccf-m98jl            0/1     CrashLoopBackOff   4          30m
coredns-fb8b8dccf-xc775            0/1     CrashLoopBackOff   4          30m

第一步：重新配置coredns，进入配置文件

~$ kubectl edit cm coredns -n kube-system

第二步：注释掉loop那一行代码

第三步：重新启动coredns

~$ kubectl delete pod coredns-fb8b8dccf-m98jl -n kube-system
~$ kubectl delete pod coredns-fb8b8dccf-xc775 -n kube-system

再次查询kubernetes pods的运行状态，显示如下，

~$ kubectl get pods -n kube-system
NAME                               READY   STATUS        RESTARTS   AGE
coredns-fb8b8dccf-w8mpq            1/1     Running       0          27s
coredns-fb8b8dccf-zd8vx            1/1     Running       0          5m11s
etcd-ubuntu00                      1/1     Running       0          38m
kube-apiserver-ubuntu00            1/1     Running       0          38m
kube-controller-manager-ubuntu00   1/1     Running       0          38m
kube-flannel-ds-amd64-p6hkl        1/1     Running       0          10m
kube-proxy-9ll6l                   1/1     Running       0          38m
kube-scheduler-ubuntu00            1/1     Running       0          37m

查看网络配置好之后，master node是否就绪

~$ kubectl get node
NAME       STATUS   ROLES    AGE   VERSION
ubuntu00   Ready    master   42m   v1.14.1

Master节点的status为ready状态

7. 初始化Kubernetes Worker (Only for Worker Node)

Worker节点主动通过下列命令2 (该命令在初始化Master节点的时候会显示在屏幕中) 请求添加到Master的集群中。Workers是运行任务的地方，Master为分发任务的地方。

kubeadm join 192.168.0.103:6443 --token r5qgpn.m950jgbdbln80a8p \
    --discovery-token-ca-cert-hash sha256:2f348b99af4d6d23bebf95937744ef93f463e7bf395d0f3fab141e5b9243200f

8. Master节点管理新加入的节点 (Only for Master Node)

在Master端看到集群中的机器

kubectl get node

Node节点加入成功，status为ready

参考文献

[1. 官方文档安装Docker-ce] https://kubernetes.io/docs/setup/cri/#docker
[2. 官方文档安装Kubeadm] https://kubernetes.io/docs/setup/independent/install-kubeadm/
[3. 官方文档安装Kubectl] https://kubernetes.io/docs/tasks/tools/install-kubectl/