centos7系统kubeadm部署kubernetes(k8s)集群

环境

系统：阿里云centos5.7，2核、2G内存
kubernetes：1.11.1
docker：17.03.3.ce

节点：
172.19.235.15 master
172.19.235.16 node1

注：k8s要求服务器配置至少要2核，2G内存。我们这里准备2台机器。

部署步骤

1.关闭防火墙 (master/node)

systemctl stop firewalld
systemctl disable firewalld 

2.关闭selinux (master/node)

sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0

3.关闭swap (master/node)

swapoff -a   # 临时
vi /etc/fstab   # 永久 

4.同步时间 (master/node)

yum install ntpdate -y
ntpdate  ntp.api.bz

5.下载docker和kubernetes源 (master/node)
docker源：

cd /etc/yum.repos.d
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

kubernetes源：
vi /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

这里我们使用aliyun的镜像。因为某些不可描述的原因，不得不使用国内镜像。

6.安装docker和kubernetes组件
master/node节点

yum localinstall https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.3.ce-1.el7.noarch.rpm 

master节点

yum install docker-ce-17.03.3.ce
yum install kubelet-1.11.1 
yum install kubeadm-1.11.1
yum install kubectl-1.11.1

node节点

yum install docker-ce-17.03.3.ce 
yum install kubelet-1.11.1 
yum install kubeadm-1.11.1

注：这里我们指定docker和k8s版本，因为官网说在docker17比较稳定。

启动docker

systemctl daemon-reload
systemctl start docker
systemctl enable docker

启动kubelet

systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet
systemctl stop kubelet
systemctl restart kubelet

下载k8s组件镜像 (master/node)

因为某些原因，我们无法直接下载k8s某些镜像。所以我们先从阿里云下载好镜像后，在本地更改tag。初始化k8s时直接使用本地镜像。

vi kubernetes-master.sh

K8S_VERSION=v1.11.1 
ETCD_VERSION=3.2.18
DASHBOARD_VERSION=v1.8.3
FLANNEL_VERSION=v0.10.0-amd64 
DNS_VERSION=1.1.3
PAUSE_VERSION=3.1
#基本组件
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION 
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION 
#网络组件
docker pull quay.io/coreos/flannel:$FLANNEL_VERSION
#修改tag
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION k8s.gcr.io/kube-apiserver-amd64:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION k8s.gcr.io/kube-controller-manager-amd64:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION k8s.gcr.io/kube-scheduler-amd64:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION k8s.gcr.io/kube-proxy-amd64:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION k8s.gcr.io/etcd-amd64:$ETCD_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION k8s.gcr.io/coredns:$DNS_VERSION

然后执行
sh kubernetes-master.sh

7.关闭swap分区 (master/node)

vi /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

8.初始化集群 (master)

kubeadm init --kubernetes-version=1.11.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12  --ignore-preflight-errors=Swap

初始化master信息：

[root@master ~]# kubeadm init --kubernetes-version=1.11.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12  --ignore-preflight-errors=Swap
[init] using Kubernetes version: v1.11.1
[preflight] running pre-flight checks
I0602 19:34:51.999003    4815 kernel_validator.go:81] Validating kernel version
I0602 19:34:51.999117    4815 kernel_validator.go:96] Validating kernel config
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.19.235.15]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [master localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [172.19.235.15 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 42.001851 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node master as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node master as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master" as an annotation
[bootstraptoken] using token: 9w7md1.7oik8335q4bzbe2e
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.19.235.15:6443 --token 9w7md1.7oik8335q4bzbe2e --discovery-token-ca-cert-hash sha256:7be7dcf83e6a94894a68c758881799f4b69fb793b609f7da6048a47d2cc96fd4

初始化成功后执行如下命令：

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

9.部署flannel网络插件 (master)

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

注：执行 kubectl get nodes 当状态变成Ready后，可以加入集群操作。

10.node加入集群

kubeadm join 172.19.235.15:6443 --token 9w7md1.7oik8335q4bzbe2e --discovery-token-ca-cert-hash sha256:7be7dcf83e6a94894a68c758881799f4b69fb793b609f7da6048a47d2cc96fd4

node加入集群信息：

[root@iZuf64jig06fagu4o0lct8Z ~]# kubeadm join 172.19.235.15:6443 --token 9w7md1.7oik8335q4bzbe2e --discovery-token-ca-cert-hash sha256:7be7dcf83e6a94894a68c758881799f4b69fb793b609f7da6048a47d2cc96fd4
[preflight] running pre-flight checks
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0602 19:56:06.189271    2067 kernel_validator.go:81] Validating kernel version
I0602 19:56:06.189373    2067 kernel_validator.go:96] Validating kernel config
[discovery] Trying to connect to API Server "172.19.235.15:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.19.235.15:6443"
[discovery] Requesting info from "https://172.19.235.15:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.19.235.15:6443"
[discovery] Successfully established connection with API Server "172.19.235.15:6443"
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.11" ConfigMap in the kube-system namespace
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[preflight] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "izuf64jig06fagu4o0lct8z" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

master节点查看已加入成功

[root@master ~]# kubectl get node
NAME                      STATUS    ROLES     AGE       VERSION
izuf64jig06fagu4o0lct8z   Ready         51m       v1.11.1
master                    Ready     master    1h        v1.11.1