kubernetes-HA 高可用部署
1 . 部署架构
- kubernetes组件说明
kube-apiserver：集群核心，集群API接口、集群各个组件通信的中枢；集群安全控制；
etcd：集群的数据中心，用于存放集群的配置以及状态信息，非常重要，如果数据丢失那么集群将无法恢复；因此高可用集群部署首先就是etcd是高可用集群；
kube-scheduler：集群Pod的调度中心；默认kubeadm安装情况下--leader-elect参数已经设置为true，保证master集群中只有一个kube-scheduler处于活跃状态；
kube-controller-manager：集群状态管理器，当集群状态与期望不同时，kcm会努力让集群恢复期望状态；
kubelet: kubernetes node agent，负责与node上的docker engine打交道；
kube-proxy: 每个node上一个，负责service vip到endpoint pod的流量转发，当前主要通过设置iptables规则实现。
2 . 环境
- 172.16.50.121 morepay01 CentOS 7.4.1708
- 172.16.50.122 morepay02 CentOS 7.4.1708
- 172.16.50.123 morepay03 CentOS 7.4.1708
- 172.16.50.125 morepay04 CentOS 7.4.1708
3 . 基础配置
3.1 . 配置免密登录
所有软件包上传到服务器172.16.50.121；为了方便拷贝镜像，及配置文件；配置172.16.50.121免密登录其他服务器
- 在morepay01主机
ssh-keygen
for i in 121 122 123 125 ; do ssh-copy-id [email protected].$i ; done
3.2 . 参数调整
所有主机
morepay01主机
创建`/etc/sysctl.d/k8s.conf `文件添加如下:
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1分发到其他主机
for i in 122 123 125 ; do scp /etc/sysctl.d/k8s.conf [email protected].$i:/etc/sysctl.d/k8s.conf ; done生效
for i in 121 122 123 125 ; do ssh [email protected].$i "sysctl -p /etc/sysctl.d/k8s.conf " ; done如提示
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory执行命令 modprobe br_netfilter
3.3 . 禁用SELINUX
for i in 121 122 123 125 ; do ssh [email protected].$i 'setenforce 0 && sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/sysconfig/selinux' ; done 3.4 . hosts配置
所有主机添加
- 172.16.50.121 morepay01
- 172.16.50.122 morepay02
- 172.16.50.123 morepay03
-
172.16.50.125 morepay04
for i in 122 123 125 ; do scp /etc/hosts [email protected].$i:/etc/hosts ; done
4 . etcd集群部署
参考：https://blog.51cto.com/11889458/2105040
步骤略
5 . Docker安装
for i in 121 122 123 125 ; do ssh [email protected].$i "yum install docker -y" ; done
for i in 121 122 123 125 ; do ssh [email protected].$i "systemctl start docker.service && systemctl status docker.service && systemctl enable docker.service" ; done6 . kubernetes组件安装
for i in 121 122 123 125 ; do ssh [email protected].$i "mkdir /data/soft -p" ; done
cd /data/soft 科学上网下载组件
- kubeadm-1.9.1-0.x86_64.rpm
- kubectl-1.9.1-0.x86_64.rpm
- kubelet-1.9.1-0.x86_64.rpm
- kubernetes-cni-0.6.0-0.x86_64.rpm
拷贝到其他服务器
for i in 122 123 125 ; do scp /data/soft/* [email protected].$i:/data/soft ; done安装
for i in 121 122 123 125 ; do ssh [email protected].$i "yum install /data/soft/* -y" ; done 设置kubelet开机启动
for i in 121 122 123 125 ; do ssh [email protected].$i "systemctl enable kubelet.service" ; done 7 . 初始化集群
7.1 . kubernetes基础镜像导入
科学上网下载docker镜像
- k8s-dns-dnsmasq-nanny-amd64_1.14.7.tar
- k8s-dns-kube-dns-amd64_1.14.7.tar
- k8s-dns-sidecar-amd64_1.14.7.tar
- kube-apiserver-amd64_v1.9.1.tar
- kube-controller-manager-amd64_v1.9.1.tar
- kube-proxy-amd64_v1.9.1.tar
- kube-scheduler-amd64_v1.9.1.tar
- pause-amd64_3.0.tar
分发镜像到其他服务器
for i in 121 122 123 125 ; do ssh [email protected].$i "mkdir /data/images" ; done
for i in 122 123 125 ; do scp /data/images/* [email protected].$i:/data/images ; done导入镜像
for j in `ls /data/images`; do docker load --input /data/images/$j ; done # 所有机器执行
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/google_containers/kube-apiserver-amd64 v1.9.1 e313a3e9d78d 6 days ago 210.4 MB
gcr.io/google_containers/kube-controller-manager-amd64 v1.9.1 4978f9a64966 6 days ago 137.8 MB
gcr.io/google_containers/kube-proxy-amd64 v1.9.1 e470f20528f9 6 days ago 109.1 MB
gcr.io/google_containers/kube-scheduler-amd64 v1.9.1 677911f7ae8f 6 days ago 62.7 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64 1.14.7 db76ee297b85 11 weeks ago 42.03 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64 1.14.7 5d049a8c4eec 11 weeks ago 50.27 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64 1.14.7 5feec37454f4 11 weeks ago 40.95 MB
gcr.io/google_containers/pause-amd64 3.0 99e59f495ffa 20 months ago 746.9 kB7.2 . 初始化
创建配置文件config.yaml 如下：
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
kubernetesVersion: v1.9.1
networking:
podSubnet: 10.244.0.0/16
apiServerCertSANs:
- morepay01
- morepay02
- morepay03
- 172.16.50.121
- 172.16.50.122
- 172.16.50.123
- 172.16.50.200
apiServerExtraArgs:
endpoint-reconciler-type: "lease"
etcd:
endpoints:
- http://172.16.50.121:2379
- http://172.16.50.122:2379
- http://172.16.50.123:2379
token: "deed3a.b3542929fcbce0f0"
tokenTTL: "0"172.16.50.200 作为keep
kubeadm引导集群 morepay01主机进行
kubeadm init --config=config.yaml 拷贝/etc/kubernetes/pki/目录 到 morepay02，morepay03 主机
for i in 122 123 ; do ssh [email protected].$i "mkdir /etc/kubernetes/pki" ; done
for i in 122 123 ; do scp /etc/kubernetes/pki/* [email protected].$i:/etc/kubernetes/pki ; donekubeadm引导集群 morepay02，morepay03主机
for i in 122 123 ; do ssh [email protected].$i "kubeadm init --config=/data/soft/config.yaml" ; done 拷贝配置文件
for i in 121 122 123 ; do ssh [email protected].$i "mkdir -p $HOME/.kube && sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config" ; done查看kubernetes集群状态
kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-apiserver-morepay01 1/1 Running 0 17m
kube-system kube-apiserver-morepay02 1/1 Running 0 1m
kube-system kube-apiserver-morepay03 1/1 Running 0 29s
kube-system kube-controller-manager-morepay01 1/1 Running 0 17m
kube-system kube-controller-manager-morepay02 1/1 Running 0 1m
kube-system kube-controller-manager-morepay03 1/1 Running 0 41s
kube-system kube-dns-6f4fd4bdf-hh9zg 0/3 Pending 0 18m
kube-system kube-proxy-48k7v 1/1 Running 0 1m
kube-system kube-proxy-m6m7j 1/1 Running 0 2m
kube-system kube-proxy-rl5kz 1/1 Running 0 18m
kube-system kube-scheduler-morepay01 1/1 Running 0 17m
kube-system kube-scheduler-morepay02 1/1 Running 0 1m
kube-system kube-scheduler-morepay03 1/1 Running 0 35s7.3 . 安装flannel；网络插件
kubectl create -f kube-flannel.yml 查看pods运行状态
kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-apiserver-morepay01 1/1 Running 0 21m
kube-system kube-apiserver-morepay02 1/1 Running 0 5m
kube-system kube-apiserver-morepay03 1/1 Running 0 4m
kube-system kube-controller-manager-morepay01 1/1 Running 0 21m
kube-system kube-controller-manager-morepay02 1/1 Running 0 5m
kube-system kube-controller-manager-morepay03 1/1 Running 0 4m
kube-system kube-dns-6f4fd4bdf-hh9zg 3/3 Running 0 22m
kube-system kube-flannel-ds-6vwbs 1/1 Running 0 1m
kube-system kube-flannel-ds-7gqv2 1/1 Running 0 1m
kube-system kube-flannel-ds-k8dp9 1/1 Running 0 1m
kube-system kube-proxy-48k7v 1/1 Running 0 6m
kube-system kube-proxy-m6m7j 1/1 Running 0 6m
kube-system kube-proxy-rl5kz 1/1 Running 0 22m
kube-system kube-scheduler-morepay01 1/1 Running 0 21m
kube-system kube-scheduler-morepay02 1/1 Running 0 5m
kube-system kube-scheduler-morepay03 1/1 Running 0 4m8 . keepalived 安装
for i in 121 122 123 ; do ssh [email protected].$i "yum install keepalived -y" ; done
for i in 121 122 123 ; do ssh [email protected].$i "systemctl enable keepalived" ; done 创建健康检查脚本check_apiserver.sh；如下：
#!/bin/bash
err=0
for k in $( seq 1 10 )
do
check_code=$(ps -ef|grep kube-apiserver | wc -l)
if [ "$check_code" = "1" ]; then
err=$(expr $err + 1)
sleep 5
continue
else
err=0
break
fi
done
if [ "$err" != "0" ]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi拷贝脚本至 morepay01 ，morepay02 ， morepay03 ； 添加执行权限
for i in 121 122 123 ; do scp /data/soft/check_apiserver.sh [email protected].$i:/etc/keepalived/ ; done
for i in 121 122 123 ; do ssh [email protected].$i "chmod +x /etc/keepalived/check_apiserver.sh" ; done创建keepalived.conf 配置文件；如下：
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 60
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 53
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass 4be37dc3b4c90194d1600c483e10ad1d
}
virtual_ipaddress {
172.16.50.200
}
track_script {
chk_apiserver
}
}拷贝配置文件至 morepay01 ，morepay02 ， morepay03
for i in 121 122 123 ; do scp /data/soft/keepalived.conf [email protected].$i:/etc/keepalived/keepalived.conf ; done启动keepalived
for i in 121 122 123 ; do ssh [email protected].$i "systemctl start keepalived" ; done9 . kubernetes加入node节点
for i in 125 ; do echo $i ; ssh [email protected].$i "kubeadm join --token deed3a.b3542929fcbce0f0 172.16.50.200:6443 --discovery-token-ca-cert-hash sha256:d49e5784284ad741aaa8259b9987d52a394b5d76d137d179951f4979e27eb58d" ; done 10 . kube-proxy配置
kubectl edit -n kube-system configmap/kube-proxy
server: https://172.16.50.200:6443重启重新创建pod
kubectl get pods --all-namespaces -o wide | grep proxy
kubectl delete pod -n kube-system kube-proxy-xxxxx11 . 安装插件（可选）
kubectl create -f heapster.yaml
kubectl create -f influxdb.yaml
kubectl create -f kubernetes-dashboard.yaml
kubectl create -f grafana.yaml 12 . 访问仪表盘
https://172.16.50.200:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login
12.1 . 证书导出
cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk -F ': ' '{print $2}' | base64 -d > /etc/kubernetes/pki/client.crt
cat /etc/kubernetes/admin.conf | grep client-key-data | awk -F ': ' '{print $2}' | base64 -d > /etc/kubernetes/pki/client.key
openssl pkcs12 -export -inkey /etc/kubernetes/pki/client.key -in /etc/kubernetes/pki/client.crt -out /etc/kubernetes/pki/client.pfx下载/etc/kubernetes/pki/client.pfx文件 导入浏览器
12.2 . 查看Token
kubectl get Secret --all-namespaces | grep dashboard-admin
kubectl describe Secret kubernetes-dashboard-admin-token-xxxx -n kube-system 13.自动伸缩
需要创建一个服务
kubectl create -f deploy.yaml 测试，创建deployment
kubectl run php-apache --image=172.16.50.116/google_containers/hpa-example:latest --requests=cpu=200m --expose --port=80创建 horizontalpodautoscalers
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10查看hpa
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0% / 50% 1 10 1 9m增加cpu负载
while true; do wget -q -O- 172.16.50.121:30150; done
再查看hpa
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 290% / 50% 1 10 4 11m14 . 手动生成证书
参考：https://github.com/fandaye/k8s-tls