【Docker系列】Docker Swarm 多节点集群
三节点集群搭建
创建3节点swarm cluster的方法
- https://labs.play-with-docker.com/ play with docker 网站， 优点是快速方便，缺点是环境不持久，4个小时后环境会被重置
- 在本地通过虚拟化软件搭建Linux虚拟机，优点是稳定，方便，缺点是占用系统资源，需要电脑内存最好8G及其以上
- 在云上使用云主机， 亚马逊，Google，微软Azure，阿里云，腾讯云等，缺点是需要消耗金钱（但是有些云服务，有免费试用）
多节点的环境涉及到机器之间的通信需求，所以防火墙和网络安全策略组是大家一定要考虑的问题，特别是在云上使用云主机的情况，下面这些端口记得打开 防火墙 以及 设置安全策略组
- TCP port
2376 - TCP port
2377 - TCP and UDP port
7946 - UDP port
4789
为了简化，以上所有端口都允许节点之间自由访问就行。
请大家注意，请大家使用自己熟悉的方式去创建这样的三节点集群，如果熟悉vagrant和virtualbox，那可以使用我们课程的里方法，如果不熟悉想学习，请参考B站和Youtube视频
Vagrant + Virtualbox
下载安装 VirtualBox： https://www.virtualbox.org/
下载安装 Vagarnt： https://www.vagrantup.com/
Vagrant入门系列视频：https://space.bilibili.com/364122352/channel/seriesdetail?sid=1734443
本节Vagrant搭建的文件下载
- Centos 版 vagrant相关文件
- Ubuntu 版 vagrant相关文件
Vagrant的基本操作：
- 虚拟机的启动：vagrant up
- 虚拟机的停止：vagrant halt
- 虚拟机的删除：vagrant destroy
初始化manager节点报错
docker集群初始化manager节点报错：
$ docker swarm init
Error response from daemon: could not choose an IP address to advertise since this system has multiple addresses on different interfaces (192.168.0.27 on eth0 and 172.18.0.87 on eth1) - specify one with --advertise-addr
这个错误出现在电脑有多个网卡时，原因是docker不知道该选用那个ip作为地址，解决方式很简单，就是在命令后面加上参数 --advertise-addr ip地址，例如：
docker-machine ssh myvm1 “docker swarm init --advertise-addr 192.168.99.100”
问题解决。
添加节点
192.168.0.28 管理节点
$ docker swarm init --advertise-addr 192.168.0.28
Swarm initialized: current node (cthj34wxr7j2eg2gv8bwxv1ff) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-3wrw7aibkg34nxw7plpf78sxty17j5uioasjo6neg41wf6gdjr-a7bqdfzh56f8suieomn53al5i 192.168.0.28:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
[node1] (local) [email protected] ~
192.168.0.27 子节点
[node2] (local) [email protected] ~
$ docker swarm join --token SWMTKN-1-3wrw7aibkg34nxw7plpf78sxty17j5uioasjo6neg41wf6gdjr-a7bqdfzh56f8suieomn53al5i 192.168.0.28:2377
This node joined a swarm as a worker.
192.168.0.26 子节点
[node3] (local) [email protected] ~
$ docker swarm join --token SWMTKN-1-3wrw7aibkg34nxw7plpf78sxty17j5uioasjo6neg41wf6gdjr-a7bqdfzh56f8suieomn53al5i 192.168.0.28:2377
This node joined a swarm as a worker.
查看管理节点，3个节点都已经添加完成。
[node1] (local) [email protected] ~
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
cthj34wxr7j2eg2gv8bwxv1ff * node1 Ready Active Leader 20.10.0
7gb1bwx68tjw6qtvb4nd3kv9m node2 Ready Active 20.10.0
903a2d81kqjrly3ldqex4qima node3 Ready Active 20.10.0
开始体验service集群
$ docker service create --name web nginx
hwteuq05xdeggyve8bb1gltqu
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
$ docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
fovbjyzwhijm web.1 nginx:latest node2 Running Running 30 seconds ago
创建在node2 192.168.0.27节点上
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fbea63b73275 nginx:latest "/docker-entrypoint.…" About a minute ago Up About a minute 80/tcp web.1.fovbjyzwhijmbj51qi36j4g21
[node2] (local) [email protected] ~
扩展到3个service
$ docker service update web --replicas 3
web
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged
[node1] (local) [email protected] ~
$ docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
fovbjyzwhijm web.1 nginx:latest node2 Running Running 4 minutes ago
yxyjt8i6ozix web.2 nginx:latest node1 Running Running 27 seconds ago
4gmtiqy9ho33 web.3 nginx:latest node3 Running Running 27 seconds ago
[node1] (local) [email protected] ~
这样就可以看到，3个节点都部署一个service
故障模拟
强制删除node2 192.168.0.27节点的service
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fbea63b73275 nginx:latest "/docker-entrypoint.…" 6 minutes ago Up 6 minutes 80/tcp web.1.fovbjyzwhijmbj51qi36j4g21
[node2] (local) [email protected] ~
$ docker container rm -f fbea63b73275
fbea63b73275
[node2] (local) [email protected] ~
查看管理节点：
$ docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ii0o24kiysdq web.1 nginx:latest node2 Running Running 37 seconds ago
fovbjyzwhijm \_ web.1 nginx:latest node2 Shutdown Failed 43 seconds ago "task: non-zero exit (137)"
yxyjt8i6ozix web.2 nginx:latest node1 Running Running 2 minutes ago
4gmtiqy9ho33 web.3 nginx:latest node3 Running Running 2 minutes ago
[node1] (local) [email protected] ~
可以看到，node2节点被强制杀死的service，同时node2节点又重新创建了一个service
注意：不一定是在宕掉的节点重新创建service，可能在其他节点创建，看swarm的机制来决定。
docker service scale
scale Scale one or multiple replicated services 扩展一个或多个复制服务
创建第4个容器
$ docker service scale web=4
web scaled to 4
overall progress: 4 out of 4 tasks
1/4: running [==================================================>]
2/4: running [==================================================>]
3/4: running [==================================================>]
4/4: running [==================================================>]
verify: Service converged
[node1] (local) [email protected] ~
docker service logs
显示日志，可以看出是那个节点的日志
docker service logs -f 一直监听日志，打印到屏幕上
$ docker service logs web
web.3.4gmtiqy9ho33@node3 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web.3.4gmtiqy9ho33@node3 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web.3.4gmtiqy9ho33@node3 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web.3.4gmtiqy9ho33@node3 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web.3.4gmtiqy9ho33@node3 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web.3.4gmtiqy9ho33@node3 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web.3.4gmtiqy9ho33@node3 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web.3.4gmtiqy9ho33@node3 | /docker-entrypoint.sh: Configuration complete; ready for start up
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: using the "epoll" event method
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: nginx/1.21.5
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: OS: Linux 4.4.0-210-generic
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker processes
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 35
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 36
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 37
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 38
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 39
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 40
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 41
web.3.4gmtiqy9ho33@node3 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 42
web.1.ii0o24kiysdq@node2 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web.1.ii0o24kiysdq@node2 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web.1.ii0o24kiysdq@node2 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web.1.ii0o24kiysdq@node2 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web.1.ii0o24kiysdq@node2 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web.1.ii0o24kiysdq@node2 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web.1.ii0o24kiysdq@node2 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web.1.ii0o24kiysdq@node2 | /docker-entrypoint.sh: Configuration complete; ready for start up
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: using the "epoll" event method
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: nginx/1.21.5
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: OS: Linux 4.4.0-210-generic
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker processes
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 33
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 34
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 35
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 36
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 37
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 38
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 39
web.1.ii0o24kiysdq@node2 | 2022/01/24 18:30:34 [notice] 1#1: start worker process 40
web.2.yxyjt8i6ozix@node1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web.2.yxyjt8i6ozix@node1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web.2.yxyjt8i6ozix@node1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web.2.yxyjt8i6ozix@node1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web.2.yxyjt8i6ozix@node1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web.2.yxyjt8i6ozix@node1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web.2.yxyjt8i6ozix@node1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web.2.yxyjt8i6ozix@node1 | /docker-entrypoint.sh: Configuration complete; ready for start up
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: using the "epoll" event method
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: nginx/1.21.5
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: OS: Linux 4.4.0-210-generic
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker processes
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 31
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 32
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 33
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 34
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 35
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 36
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 37
web.2.yxyjt8i6ozix@node1 | 2022/01/24 18:28:27 [notice] 1#1: start worker process 38
web.4.nlxd4bobmwje@node1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
web.4.nlxd4bobmwje@node1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
web.4.nlxd4bobmwje@node1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
web.4.nlxd4bobmwje@node1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
web.4.nlxd4bobmwje@node1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
web.4.nlxd4bobmwje@node1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
web.4.nlxd4bobmwje@node1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
web.4.nlxd4bobmwje@node1 | /docker-entrypoint.sh: Configuration complete; ready for start up
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: using the "epoll" event method
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: nginx/1.21.5
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: OS: Linux 4.4.0-210-generic
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker processes
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 33
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 34
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 35
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 36
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 37
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 38
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 39
web.4.nlxd4bobmwje@node1 | 2022/01/24 18:35:41 [notice] 1#1: start worker process 40
[node1] (local) [email protected] ~