Kubernetes集群配置高可用
高可用原理
kubelet的config参数可以监控特定目录下的yaml文件,确保指定目录下的pod可以稳定运行，我们可以在目录下创建apiserver,controller-manager,scheduler的对应yaml描述文件，通过启动kubelet来确保三者稳定运行。api-server可以使用nginx来进行负载均衡和健康状态监测，controller-manager,scheduler通过启动时设置–leader-elect=true进行选举确保同一时刻只有一台主机上的起作用，故障时standby会切换为master。
官方推荐master-standby-standby来配置高可用，一主两备。
kubelet的高可用：systemctl设置Restart=on-failure来实现，
Restart字段：定义了 kubelet 退出后，Systemd 的重启方式。
上面的例子中，Restart设为on-failure，表示任何意外的失败，就将重启kubelet。如果 kubelet 正常停止（比如执行systemctl stop命令），它就不会重启。
Restart字段可以设置的值如下。
- no（默认值）：退出后不会重启
- on-success：只有正常退出时（退出状态码为0），才会重启
- on-failure：非正常退出时（退出状态码非0），包括被信号终止和超时，才会重启
- on-abnormal：只有被信号终止和超时，才会重启
- on-abort：只有在收到没有捕捉到的信号终止时，才会重启
- on-watchdog：超时退出，才会重启
- always：不管是什么退出原因，总是重启
apiserver前端nginx高可用：keepalive+nginx不再赘述
etcd：集群形式
节点信息
| 主机IP | 操作系统 |
|---|---|
| 172.20.1.99 | centos7.2 |
| 172.20.1.100 | centos7.2 |
| 172.20.1.101 | centos7.2 |
配置过程（可直接保存为kubelet.sh执行）
#config配置文件
cat > /etc/kubernetes/config <"--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER=""
EOF
#kubelet配置文件
cat > /etc/kubernetes/kubelet <"--address=127.0.0.1"
KUBELET_HOSTNAME="--hostname-override=172.20.1.101"
KUBELET_API_SERVER=""
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=172.20.1.128:5000/pause"
KUBELET_ARGS="--config=/etc/kubernetes/manifests --register-node=false"
EOF
#启动文件
echo "
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/bin/kubelet \
\$KUBE_LOGTOSTDERR \
\$KUBE_LOG_LEVEL \
\$KUBELET_API_SERVER \
\$KUBELET_ADDRESS \
\$KUBELET_PORT \
\$KUBELET_HOSTNAME \
\$KUBE_ALLOW_PRIV \
\$KUBELET_POD_INFRA_CONTAINER \
\$KUBELET_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target
" > /usr/lib/systemd/system/kubelet.service
#生成工作目录
mkdir -p /var/lib/kubelet/pods
mkdir -p /var/lib/kubelet/plugins
#设置开机启动
systemctl enable kubelet
#生成日志文件
touch /var/log/kube-apiserver.log
touch /var/log/kube-controller-manager.log
touch /var/log/kube-scheduler.log
#创建监听目录
mkdir -p /etc/kubernetes/manifests
#kube-apiserver.yaml
cat > /etc/kubernetes/manifests/kube-apiserver.yaml <name: kube-apiserver
spec:
hostNetwork: true
containers:
- name: kube-apiserver
image: 172.20.1.128:5000/kube-apiserver:v1.3.0
command:
- /bin/sh
- -c
- /usr/local/bin/kube-apiserver --address=172.20.1.101 --etcd-servers=http://172.20.1.103:2379,http://172.20.1.99:2379,http://172.20.1.102:2379,http://172.20.1.100:2379,http://172.20.1.101:2379
--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
--v=2 --service-cluster-ip-range=10.254.0.0/16
--allow-privileged=False 1>>/var/log/kube-apiserver.log 2>&1
ports:
- containerPort: 8080
hostPort: 8080
name: local
volumeMounts:
- mountPath: /var/log/kube-apiserver.log
name: logfile
volumes:
- hostPath:
path: /var/log/kube-apiserver.log
name: logfile
EOF
#kube-controller-manager
cat > /etc/kubernetes/manifests/kube-controller-manager.yaml <name: kube-controller-manager
spec:
containers:
- command:
- /bin/sh
- -c
- /usr/local/bin/kube-controller-manager --master=172.20.1.101:8080 --cluster-name=local
--cluster-cidr=10.254.0.0/16
--v=2 --leader-elect=true 1>>/var/log/kube-controller-manager.log 2>&1
image: 172.20.1.128:5000/kube-controller-manager:v1.3.0
livenessProbe:
httpGet:
path: /healthz
port: 10252
initialDelaySeconds: 15
timeoutSeconds: 1
name: kube-controller-manager
volumeMounts:
- mountPath: /var/log/kube-controller-manager.log
name: logfile
hostNetwork: true
volumes:
- hostPath:
path: /var/log/kube-controller-manager.log
name: logfile
EOF
#kube-scheduler
cat > /etc/kubernetes/manifests/kube-scheduler.yaml <name: kube-scheduler
spec:
hostNetwork: true
containers:
- name: kube-scheduler
image: 172.20.1.128:5000/kube-scheduler:v1.3.0
command:
- /bin/sh
- -c
- /usr/local/bin/kube-scheduler --master=172.20.1.101:8080 --v=2 --leader-elect=true 1>>/var/log/kube-scheduler.log
2>&1
livenessProbe:
httpGet:
path: /healthz
port: 10251
initialDelaySeconds: 15
timeoutSeconds: 1
volumeMounts:
- mountPath: /var/log/kube-scheduler.log
name: logfile
volumes:
- hostPath:
path: /var/log/kube-scheduler.log
name: logfile
EOF
#启动kueblet
systemctl start kubelet nginx配置文件
upstream backend {
ip_hash;
server 172.20.1.99:8080 max_fails=1 fail_timeout=300s;
server 172.20.1.100:8080 max_fails=1 fail_timeout=300s;
server 172.20.1.101:8080 max_fails=1 fail_timeout=300s;
}
server {
listen 0.0.0.0:8080;
server_name apiserver;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffering off;
proxy_pass http://backend;
}
}排错
- tailf /var/log/kube-*.log
- tailf /var/log/messages
- docker logs id
注意事项
- 其它节点上执行需要替换配置文件中的主机IP
- kubelet需要拷贝
- 启动文件$符号需要转义
- /etc/kubernetes目录下不要含有多余文件
- 脚本中的镜像需要替换，这里是自己部署的私有仓库
参考
Kubernetes - Building High-Availability Clusters
http://kubernetes.io/docs/admin/high-availability/#establishing-a-redundant-reliable-data-storage-layer