前言
在2018年初接触了Kubernetes，研究学习了如何使用kubeadm安装kubernetes集群，也写了一篇文章讲如何离线安装kubernetes集群链接，然而当时水平有限，文中所讲如今看来很多漏洞，操作也过于繁琐，因此重新写一篇文章，采用更方便的在线安装方法，快捷安装kubernetes集群，本次在线安装离不开阿里的帮助，也要感谢一波阿里。最后希望本文能帮助到有需要的人。
规划
节点规划
本次安装使用了3台ubuntu server 16.04虚拟机
| ip | hostname | 用途 |
|---|---|---|
| 10.0.3.4 | k8s-001 | control plane |
| 10.0.3.5 | k8s-002 | worker |
| 10.0.3.6 | k8s-003 | worker |
kubernetes版本
采用较新的1.14.3版本
安装
〇、安装docker
标准安装方法可以参考docker官方文档
某些同学网络可能存在墙的情况，因此可以使用阿里云的源进行安装，参考阿里官方文档
一、关闭swap
在所有节点执行以下命令
关闭swap
swapoff -a关闭配置文件中的swap，防止重启机器后自动开启swap
sed -i 's/^\([^#].*swap.*$\)/#\1/' /etc/fstab二、安装kubeadm、kubectl、kubelet
按照谷歌官方文档进行安装
上一步99%的同学都会因为被墙而安装失败，因此可以按照这里寻求阿里帮助
进入阿里旗下的OPSX，
ctrl+f搜索kubernetes，找到后再点击最后的帮助，可以看到操作提示，这里直接贴出来apt-get update && apt-get install -y apt-transport-https curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat </etc/apt/sources.list.d/kubernetes.list deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main EOF apt-get update # apt-get install -y kubelet kubeadm kubectl #注意这一句先不要直接执行，见下文 注意上面最后一句命令不要直接执行，因为直接安装的话默认是最新版，而我们需要指定安装版本1.14.3，因此执行以下命令
apt-get install -y kubelet=1.14.3-00 kubeadm=1.14.3-00 kubectl=1.14.3-00有些小伙伴不知道当前源有哪些版本，这里可以使用以下命令进行查询，第二列就是版本信息
root@k8s-001:/home# apt-cache madison kubeadm kubeadm | 1.15.0-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages kubeadm | 1.14.3-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages kubeadm | 1.14.2-00 | https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 Packages . . .
三、生成kubeadm的配置文件
在k8s-001上执行以下命令
cat < ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.3
apiServerCertSANs:
- k8s-001
- 10.0.3.4
- myk8s.cluster
controlPlaneEndpoint: "10.0.3.4:6443"
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
networking:
podSubnet: "10.244.0.0/16"
EOF 以上配置文件说明
- kubernetesVersion：集群版本，本次安装为v1.14.3
- apiServerCertSANs：kubernetes集群安装时会自动生成证书，其中k8s-001，10.0.3.4两项SANs，kubernetes在生成证书时会自带，但是推荐多加一个域名（myk8s.cluster），以防止集群IP变动后证书失效
- controlPlaneEndpoint：控制平面端点，单控制平面时就是控制平面本身的IP:6443，在做高可用集群时就需要虚拟的vip或者负载均衡ip，后面教大家搭建高可用集群时可以再详细说明一下
- imageRepository：这个配置就很重要了，默认情况下kubernetes会去gcr.io拉取镜像，由于墙的原因99%的同学会失败，因此再次替换为阿里的仓库
- networking.podSubnet：pod的网段，不要和现有网段冲突即可
四、初始化control plane节点
在k8s-001上执行以下命令
kubeadm init --config kubeadm-config.yaml以上命令说明
- --config：指定配置文件，就是第三步生成的kubeadm-config.yaml
不出意外可以看到成功信息
.
.
.
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join 10.0.3.4:6443 --token xxxxxx.xxxxxxxxxxxxxxxx \
--discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
--experimental-control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.3.4:6443 --token xxxxxx.xxxxxxxxxxxxxxxx \
--discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 按照提示在k8s-001节点执行以下三条命令
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config我们需要使用Kubectl客户端连接kubernetes集群，并且连接时是需要经过认证的，上面的三步操作就是将认证文件放到默认路径
$HOME/.kube/config下，供kubectl读取
此时我们就可以执行kubectl命令查看集群信息了
root@k8s-001:/home# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-d5947d4b-k69nd 0/1 Pending 0 4m34s
kube-system coredns-d5947d4b-ll6hx 0/1 Pending 0 4m34s
kube-system etcd-k8s-001 1/1 Running 0 3m44s
kube-system kube-apiserver-k8s-001 1/1 Running 0 3m47s
kube-system kube-controller-manager-k8s-001 1/1 Running 0 4m1s
kube-system kube-proxy-p9jgp 1/1 Running 0 4m34s
kube-system kube-scheduler-k8s-001 1/1 Running 0 3m48s上面的结果显示coredns pod处于Pending状态，这是因为coredns需要依赖网络插件，而现在网络插件还未安装
五、worker节点加入集群
安装第四步提示，在k8s-002，k8s-003节点分别执行以下命令
kubeadm join 10.0.3.4:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 需要注意的是加入集群的命令中，token是有有效期的，一般为2小时，超过2小时再使用此token就会加入失败，这时需要重新生成加入命令，如下
root@k8s-001:~# kubeadm token create --print-join-command
kubeadm join 10.0.3.4:6443 --token xxxxxx.xxxxxxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 成功信息如下
.
.
.
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.此时我们可以在k8s-001节点查看集群信息了
root@k8s-001:/home# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-001 NotReady master 13m v1.14.3
k8s-002 NotReady 76s v1.14.3
k8s-003 NotReady 56s v1.14.3 root@k8s-001:/home/kubernetes/init# kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-d5947d4b-k69nd 0/1 Pending 0 14m
kube-system coredns-d5947d4b-ll6hx 0/1 Pending 0 14m
kube-system etcd-k8s-001 1/1 Running 0 13m 10.0.3.4 k8s-001
kube-system kube-apiserver-k8s-001 1/1 Running 0 13m 10.0.3.4 k8s-001
kube-system kube-controller-manager-k8s-001 1/1 Running 0 13m 10.0.3.4 k8s-001
kube-system kube-proxy-g4p5x 1/1 Running 0 2m 10.0.3.5 k8s-002
kube-system kube-proxy-p9jgp 1/1 Running 0 14m 10.0.3.4 k8s-001
kube-system kube-proxy-z9cpd 1/1 Running 0 100s 10.0.3.6 k8s-003
kube-system kube-scheduler-k8s-001 1/1 Running 0 13m 10.0.3.4 k8s-001 六、安装calico网络插件
网络插件有很多，例如flannel等，大家可以自行选用，这里使用calico
root@k8s-001:/home# kubectl apply -f https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
etworking/1.7/calico.yaml
configmap/calico-config created
service/calico-typha created
deployment.apps/calico-typha created
poddisruptionbudget.policy/calico-typha created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created网络插件安装完毕后，再次查看pod信息，就会看见coredns状态已经时Running了
root@k8s-001:/home# kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system calico-node-gf6j4 1/1 Running 0 39s 10.0.3.4 k8s-001
kube-system calico-node-l4w9n 1/1 Running 0 39s 10.0.3.5 k8s-002
kube-system calico-node-rtcnl 1/1 Running 0 39s 10.0.3.6 k8s-003
kube-system coredns-d5947d4b-k69nd 1/1 Running 0 17m 10.244.0.10 k8s-001
kube-system coredns-d5947d4b-ll6hx 1/1 Running 0 17m 10.244.1.6 k8s-002
kube-system etcd-k8s-001 1/1 Running 0 16m 10.0.3.4 k8s-001
kube-system kube-apiserver-k8s-001 1/1 Running 0 16m 10.0.3.4 k8s-001
kube-system kube-controller-manager-k8s-001 1/1 Running 0 16m 10.0.3.4 k8s-001
kube-system kube-proxy-g4p5x 1/1 Running 0 5m5s 10.0.3.5 k8s-002
kube-system kube-proxy-p9jgp 1/1 Running 0 17m 10.0.3.4 k8s-001
kube-system kube-proxy-z9cpd 1/1 Running 0 4m45s 10.0.3.6 k8s-003
kube-system kube-scheduler-k8s-001 1/1 Running 0 16m 10.0.3.4 k8s-001 七、安装ingress
我们计划将ingress启动一个副本，并且部署在k8s-003节点上，使用k8s-003的host网络（只启动了一个副本，这是非高可用的做法，后续可以讲一讲高可用该怎么操作）
下载ingress配置文件，我们需要对其进行修改
wget -O ingress.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml修改下载好的ingress.yaml
.
.
.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
nodeName: k8s-003 #新增，表示只能部署在k8s-003节点上
hostNetwork: true #新增，表示使用k8s-003的宿主机网络
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.0
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
.
.
.quay.io镜像仓库也有地方会被墙，或者下载速度很慢，因此可以使用镜像加速，替换
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.0为image: quay.mirrors.ustc.edu.cn/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1（这里替换为0.24.1版本是因为我在使用0.25.0版本时存在健康探测不通过的问题）
应用以上修改完毕的配置文件
root@k8s-001:/home# kubectl apply -f ingress.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created稍等一会再看下pod状态
root@k8s-001:/home/kubernetes/init# kubectl get pod -n ingress-nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-controller-6558b88448-mv7cz 1/1 Running 0 2m57s 10.0.3.6 k8s-003 八、测试一把
到此为止集群已经安装完毕了，现在安装一个简单的nginx应用测试一把
生成nginx应用的配置文件nginx.yaml，定义了deployment、service、ingress三种资源
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.15.0
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
name: nginx
spec:
ports:
- port: 80
targetPort: 80
selector:
app: nginx
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
labels:
app: nginx
name: nginx
spec:
rules:
- host: your.local.domain
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
path: /
执行以下命令部署
root@k8s-001:/home# kubectl apply -f nginx.yaml
deployment.apps/nginx created
service/nginx created
ingress.extensions/nginx created查看pod状态
root@k8s-001:/home# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-8cc98cb56-knszf 1/1 Running 0 12s使用curl命令测试
root@k8s-001:/home# curl -H "Host: your.local.domain" 10.0.3.6
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
大功告成！