企业运维----Docker-kubernetes-高可用集群部署(未完......)
Docker-kubernetes-高可用集群部署
-
-
- K8s高可用+负载均衡集群
- 部署
-
-
- haproxy负载均衡部署
- k8s高可用集群部署
-
-
K8s高可用+负载均衡集群
工作原理：
本实验通过haproxy配置k8s master主机实现负载均衡，通过k8s三台master主机实现k8s集群高可用。
实验环境：
需要6台虚拟机
至少3台master节点才能实现高可用k8s集群
| 主机名:ip | role |
|---|---|
| server1:172.25.12.1 | 提供harbor仓库服务 |
| server5:172.25.12.5 | 提供haproxy负载均衡 |
| server6:172.25.12.6 | 提供haproxy负载均衡 |
| server7:172.25.12.7 | k8s集群主机master节点 |
| server8:172.25.12.8 | k8s集群主机master节点 |
| server9:172.25.12.9 | k8s集群主机master节点 |
部署
yum源环境和主机解析
[root@server5 ~]# vim /etc/hosts
[root@server5 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.12.250 foundation12.ilt.example.com
172.25.12.1 server1 reg.westos.org
172.25.12.2 server2
172.25.12.3 server3
172.25.12.4 server4
172.25.12.5 server5
172.25.12.6 server6
172.25.12.7 server7
172.25.12.8 server8
172.25.12.9 server9
[root@server5 ~]# scp /etc/hosts server6:/etc/hosts
[root@server5 ~]# scp /etc/hosts server7:/etc/hosts
[root@server5 ~]# scp /etc/hosts server8:/etc/hosts
[root@server5 ~]# scp /etc/hosts server9:/etc/hosts
[root@server5 yum.repos.d]# vim dvd.repo
[root@server5 yum.repos.d]# cat dvd.repo
[dvd]
name=dvd
baseurl=http://172.25.12.250/rhel7.6
gpgcheck=0
[HighAvailability]
name=HighAvailability
baseurl=http://172.25.12.250/rhel7.6/addons/HighAvailability
gpgcheck=0
[root@server5 yum.repos.d]# cat docker.repo
[docker]
name=docker-ce
baseurl=http://172.25.12.250/docker-ce
gpgcheck=0
[root@server5 yum.repos.d]# scp dvd.repo docker.repo server6:/etc/yum.repos.d/
[root@server5 yum.repos.d]# scp dvd.repo docker.repo server7:/etc/yum.repos.d/
[root@server5 yum.repos.d]# scp dvd.repo docker.repo server8:/etc/yum.repos.d/
[root@server5 yum.repos.d]# scp dvd.repo docker.repo server9:/etc/yum.repos.d/
haproxy负载均衡部署
server5/6安装pacemaker相关组件，设置开机自启
[root@server5 ~]# yum install -y pacemaker pcs psmisc policycoreutils-python
[root@server6 ~]# yum install -y pacemaker pcs psmisc policycoreutils-python
[root@server5 ~]# systemctl enable --now pcsd.service
[root@server6 ~]# systemctl enable --now pcsd.service
修改server5/6 的hacluster用户密码为westos
[root@server5 ~]# passwd hacluster
Changing password for user hacluster.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
[root@server6 ~]# passwd hacluster
Changing password for user hacluster.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
pcs注册认证server5 server6
[root@server5 ~]# pcs cluster auth server5 server6
Username: hacluster
Password:
server5: Authorized
server6: Authorized
创建集群命名mycluster，server5，6为成员
[root@server5 ~]# pcs cluster setup --name mycluster server5 server6
Destroying cluster on nodes: server5, server6...
server5: Stopping Cluster (pacemaker)...
server6: Stopping Cluster (pacemaker)...
server5: Successfully destroyed cluster
server6: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'server5', 'server6'
server5: successful distribution of the file 'pacemaker_remote authkey'
server6: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
server5: Succeeded
server6: Succeeded
Synchronizing pcsd certificates on nodes server5, server6...
server5: Success
server6: Success
Restarting pcsd on the nodes in order to reload the certificates...
server5: Success
server6: Success
设置集群服务启动并开机自启
[root@server5 ~]# pcs cluster start --all
server5: Starting Cluster (corosync)...
server6: Starting Cluster (corosync)...
server5: Starting Cluster (pacemaker)...
server6: Starting Cluster (pacemaker)...
[root@server5 ~]# pcs cluster enable --all
server5: Cluster Enabled
server6: Cluster Enabled
关闭fence警告
[root@server5 ~]# crm_verify -L -V
error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
[root@server5 ~]# pcs status
Cluster name: mycluster
WARNINGS:
No stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: server6 (version 1.1.19-8.el7-c3c624ea3d) - partition with quorum
Last updated: Fri Aug 6 23:29:46 2021
Last change: Fri Aug 6 23:24:34 2021 by hacluster via crmd on server6
2 nodes configured
0 resources configured
Online: [ server5 server6 ]
No resources
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
[root@server5 ~]# pcs property set stonith-enabled=false
[root@server5 ~]# crm_verify -L -V
[root@server5 ~]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: server6 (version 1.1.19-8.el7-c3c624ea3d) - partition with quorum
Last updated: Fri Aug 6 23:30:57 2021
Last change: Fri Aug 6 23:30:48 2021 by root via cibadmin on server5
2 nodes configured
0 resources configured
Online: [ server5 server6 ]
No resources
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
创建vip 172.25.12.100
[root@server5 ~]# pcs resource create vip ocf:heartbeat:IPaddr2 ip=172.25.12.100 op monitor interval=30s
[root@server5 ~]# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:7f:34:da brd ff:ff:ff:ff:ff:ff
inet 172.25.12.5/24 brd 172.25.12.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.25.12.100/24 brd 172.25.12.255 scope global secondary eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe7f:34da/64 scope link
valid_lft forever preferred_lft forever
查看pcs状态
[root@server5 docker]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: server5 (version 1.1.19-8.el7-c3c624ea3d) - partition with quorum
Last updated: Sun Aug 8 02:03:32 2021
Last change: Sat Aug 7 03:39:44 2021 by root via cibadmin on server5
2 nodes configured
2 resources configured
Online: [ server5 server6 ]
Full list of resources:
Resource Group: haproup
vip (ocf::heartbeat:IPaddr2): Started server5
haproxy (systemd:haproxy): Started server5
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
安装haproxy服务编辑配置文件
[root@server5 ~]# yum install -y haproxy
[root@server5 ~]# cd /etc/haproxy/
[root@server5 haproxy]# vim haproxy.cfg
# 监听80端口，查看负载均衡节点状态
59 listen stats *:80
60 status uri /status
# 设定负载均衡监听端口为6443 监听模式为tcp
64 frontend main *:6443
65 mode tcp
66 default_backend app
# 添加负载均衡后端节点为k8s1/k8s2/k8s3，check各自IP的6443端口
72 backend app
73 balance roundrobin
74 mode tcp
75 server k8s1 127.0.0.1:5001 check
76 server k8s2 127.0.0.1:5002 check
77 server k8s3 127.0.0.1:5003 check
开启haproxy服务 查看监听端口是否开启
[root@server5 haproxy]# systemctl start haproxy.service
[root@server5 haproxy]# netstat -antlp |grep :6443
tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 8187/haproxy
[root@server5 haproxy]# netstat -antlp |grep :80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 8599/haproxy
k8s主机down 还未部署
为server6进行同样操作，并在访问测试后关闭haproxy服务
[root@server6 ~]# yum install -y haproxy.x86_64
[root@server5 haproxy]# systemctl stop haproxy.service
[root@server5 haproxy]# ls
haproxy.cfg
[root@server5 haproxy]# scp haproxy.cfg server6:/etc/haproxy/
root@server6's password:
haproxy.cfg 100% 2682 4.5MB/s 00:00
[root@server6 ~]# systemctl start haproxy.service
创建haproxy服务到pcs集群
[root@server5 haproxy]# pcs resource create haproxy systemd:haproxy op monitor interval=60s
[root@server5 haproxy]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: server6 (version 1.1.19-8.el7-c3c624ea3d) - partition with quorum
Last updated: Sat Aug 7 00:51:05 2021
Last change: Sat Aug 7 00:50:28 2021 by root via cibadmin on server5
2 nodes configured
2 resources configured
Online: [ server5 server6 ]
Full list of resources:
vip (ocf::heartbeat:IPaddr2): Started server5
haproxy (systemd:haproxy): Started server6
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
将vip和haproxy服务拉到一个group结点上
[root@server5 haproxy]# pcs resource group add haproup vip haproxy
[root@server5 haproxy]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: server6 (version 1.1.19-8.el7-c3c624ea3d) - partition with quorum
Last updated: Sat Aug 7 00:52:33 2021
Last change: Sat Aug 7 00:52:30 2021 by root via cibadmin on server5
2 nodes configured
2 resources configured
Online: [ server5 server6 ]
Full list of resources:
Resource Group: haproup
vip (ocf::heartbeat:IPaddr2): Started server5
haproxy (systemd:haproxy): Starting server5
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
k8s高可用集群部署
master节点 部署docker 最后全部running 重启服务failed可能是dademin.json写错
[root@server7 ~]# yum install -y docker-ce
[root@server7 ~]# systemctl enable --now docker
[root@server8 ~]# yum install -y docker-ce
[root@server8 ~]# systemctl enable --now docker
[root@server9 ~]# yum install -y docker-ce
[root@server9 ~]# systemctl enable --now docker
[root@server7 ~]# cd /etc/docker/
[root@server7 docker]# ls
key.json
[root@server7 docker]# vim daemon.json
[root@server7 docker]# cat daemon.json
{
"registry-mirrors":["https://reg.westos.org"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
[root@server7 docker]# systemctl restart docker
[root@server7 docker]# systemctl status docker
Active: active (running) since Sat 2021-08-07 22:19:17 EDT; 1s ago
[root@server7 docker]# scp daemon.json server8:/etc/docker/
[root@server7 docker]# scp daemon.json server9:/etc/docker/
[root@server8 ~]# systemctl restart docker
[root@server8 ~]# systemctl status docker
[root@server9 ~]# systemctl restart docker
[root@server9 ~]# systemctl status docker
server7 查看docker info 编写/etc/sysctl.d/docker.conf 重启内核模块 WARNING消失
[root@server7 docker]# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.15
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-957.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: server7
ID: TXAW:B7U3:XPGX:PBPE:E56R:CDQ2:OMOD:DENZ:ZRKW:JR4W:NNIA:S3NK
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
[root@server7 sysctl.d]# pwd
/etc/sysctl.d/
[root@server7 sysctl.d]# ls
99-sysctl.conf docker.conf
[root@server7 sysctl.d]# cat docker.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
[root@server7 docker]# sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/docker.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
* Applying /etc/sysctl.conf ...
[root@server7 docker]# docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.15
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-957.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: server7
ID: TXAW:B7U3:XPGX:PBPE:E56R:CDQ2:OMOD:DENZ:ZRKW:JR4W:NNIA:S3NK
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
server8，9上做一遍
[root@server7 sysctl.d]# scp docker.conf server6:/etc/sysctl.d/
[root@server7 sysctl.d]# scp docker.conf server7:/etc/sysctl.d/
[root@server8 docker]# sysctl --system
[root@server8 docker]# docker info
[root@server9 docker]# sysctl --system
[root@server9 docker]# docker info
认证
[root@server1 ~]# cd /etc/docker/
[root@server1 docker]# ls
certs.d key.json
[root@server1 docker]# scp -r certs.d/ server7:/etc/docker/
[root@server1 docker]# scp -r certs.d/ server8:/etc/docker/
[root@server1 docker]# scp -r certs.d/ server9:/etc/docker/
安装ipvs使用模块，kube_proxy使用IPVS模式，减少iptables的压力。
[root@server7 ~]# yum install -y ipvsadm.x86_64
[root@server7 ~]# ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@server7 ~]# lsmod | grep ip_vs
ip_vs 145497 0
nf_conntrack 133095 7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
[root@server8 ~]# yum install -y ipvsadm.x86_64
[root@server8 ~]# ipvsadm -l
[root@server8 ~]# lsmod | grep ip_vs
[root@server9 ~]# yum install -y ipvsadm.x86_64
[root@server9 ~]# ipvsadm -l
[root@server9 ~]# lsmod | grep ip_vs
禁用swap分区，注释掉 /etc/fstab 中对swap定义
[root@server7 ~]# swapoff -a
[root@server7 ~]# vim /etc/fstab
11 #/dev/mapper/rhel-swap swap swap defaults 0 0
[root@server8 ~]# swapoff -a
[root@server8 ~]# vim /etc/fstab
11 #/dev/mapper/rhel-swap swap swap defaults 0 0
[root@server9 ~]# swapoff -a
[root@server9 ~]# vim /etc/fstab
11 #/dev/mapper/rhel-swap swap swap defaults 0 0
地址伪装让虚拟机上网 安装kubelet组件
[root@foundation12 images]# iptables -t nat -I POSTROUTING -s 172.25.12.0/24 -j MASQUERADE
[root@server7 ~]# cd /etc/yum.repos.d/
[root@server7 yum.repos.d]# ls
docker.repo dvd.repo redhat.repo
[root@server7 yum.repos.d]# vim k8s.repo
[root@server7 yum.repos.d]# cat k8s.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=0
[root@server7 ~]# yum install -y kubeadm kubelet kubectl
[root@server7 yum.repos.d]# scp k8s.repo server8:/etc/yum.repos.d/
[root@server7 yum.repos.d]# scp k8s.repo server9:/etc/yum.repos.d/
[root@server8 ~]# yum install -y kubeadm kubelet kubectl
[root@server9 ~]# yum install -y kubeadm kubelet kubectl
启动kubelet服务 设置开机自启
[root@server7 ~]# systemctl enable --now kubelet.service
[root@server8 ~]# systemctl enable --now kubelet.service
[root@server9 ~]# systemctl enable --now kubelet.service
server7内配置高可用，导出kubeadm-init.yaml配置文件
[root@server7 ~]# kubeadm config print init-defaults > kubeadm-init.yaml
[root@server7 ~]# ls
kubeadm-1.21.3.tar.gz kubeadm-init.yaml packages
[root@server7 ~]# vim kubeadm-init.yaml
[root@server7 ~]# cat kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.25.12.7
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: server7
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "172.25.12.100:6443"
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: reg.westos.org/k8s
kind: ClusterConfiguration
kubernetesVersion: 1.21.3
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
拉取镜像
[root@server7 ~]# kubeadm config images pull --config kubeadm-init.yaml
[config/images] Pulled reg.westos.org/k8s/kube-apiserver:v1.21.3
[config/images] Pulled reg.westos.org/k8s/kube-controller-manager:v1.21.3
[config/images] Pulled reg.westos.org/k8s/kube-scheduler:v1.21.3
[config/images] Pulled reg.westos.org/k8s/kube-proxy:v1.21.3
[config/images] Pulled reg.westos.org/k8s/pause:3.4.1
[config/images] Pulled reg.westos.org/k8s/etcd:3.4.13-0
[config/images] Pulled reg.westos.org/k8s/coredns:v1.8.0
未完…