使用docker-compose部署redis集群

集群架构：3主3从
redis版本：5.0
网络模式：host
根据官网的描述，需要用host模式。

In order to make Docker compatible with Redis Cluster you need to use the host networking mode of Docker. Please check the --net=host option in the Docker documentation for more information.

在host模式下，container共用宿主机的ip，因此不需要做端口映射

端口规划

图片.png

文件规划

挂载配置文件和日志文件

修改配置文件redis.conf

分别为每个redis建一个文件夹，里面放redis.conf和redis.log
修改每个配置文件

port 6379 #这里填每台redis的端口号
bind 0.0.0.0 #绑定ip 0.0.0.0 指可以外网访问
cluster-enabled yes #开启集群
logfile "/usr/local/redis/logs/redis-server.log" #指定日志文件

docker-compose文件内容

version: '3'

services:
 master-1:
  container_name: master-1
  image: redis
  command: redis-server /etc/usr/local/redis.conf
  network_mode: "host"
  volumes:
   - /home/jasonlin/redis/master1/redis.conf:/etc/usr/local/redis.conf
   - /home/jasonlin/redis/master1/redis.log:/usr/local/redis/logs/redis-server.log

 master-2:
  container_name: master-2
  command: redis-server /etc/usr/local/redis.conf
  image: redis
  network_mode: "host"
  volumes:
   - /home/jasonlin/redis/master2/redis.conf:/etc/usr/local/redis.conf
   - /home/jasonlin/redis/master2/redis.log:/usr/local/redis/logs/redis-server.log

 master-3:
  container_name: master-3
  image: redis
  command: redis-server /etc/usr/local/redis.conf
  network_mode: "host"
  volumes:
   - /home/jasonlin/redis/master3/redis.conf:/etc/usr/local/redis.conf
   - /home/jasonlin/redis/master3/redis.log:/usr/local/redis/logs/redis-server.log

 slave-1:
  container_name: slave-1
  image: redis
  command: redis-server /etc/usr/local/redis.conf
  network_mode: "host"
  volumes:
   - /home/jasonlin/redis/slave1/redis.conf:/etc/usr/local/redis.conf
   - /home/jasonlin/redis/slave1/redis.log:/usr/local/redis/logs/redis-server.log

 slave-2:
  container_name: slave-2
  image: redis
  command: redis-server /etc/usr/local/redis.conf
  network_mode: "host"
  volumes:
   - /home/jasonlin/redis/slave2/redis.conf:/etc/usr/local/redis.conf
   - /home/jasonlin/redis/slave2/redis.log:/usr/local/redis/logs/redis-server.log
   
 slave-3:
  container_name: slave-3
  image: redis
  command: redis-server /etc/usr/local/redis.conf
  network_mode: "host"
  volumes:
   - /home/jasonlin/redis/slave3/redis.conf:/etc/usr/local/redis.conf
   - /home/jasonlin/redis/slave3/redis.log:/usr/local/redis/logs/redis-server.log

部署redis

在docker-compose.yaml的文件夹下执行(root用户)

docker-compose up -d

root@ubuntu:/home/jasonlin/redis# docker-compose up -d
Creating slave-1 ... 
Creating slave-3 ... 
Creating master-3 ... 
Creating master-2 ... 
Creating master-1 ... 
Creating slave-2 ... 
Creating slave-1
Creating slave-3
Creating master-3
Creating master-1
Creating master-2
Creating master-3 ... done

查看一下是否启动成功。如果容器列表没有，说明没启动成功，需要docker logs 容器名称/ID 看一下错误情况

root@ubuntu:/home/jasonlin/redis# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
414bdb7a4b04        redis               "docker-entrypoint.s…"   7 seconds ago       Up 5 seconds                            slave-2
2c79aaa1ed74        redis               "docker-entrypoint.s…"   7 seconds ago       Up 5 seconds                            master-2
8e89a3fbb859        redis               "docker-entrypoint.s…"   7 seconds ago       Up 5 seconds                            master-1
f5bfe6a41adb        redis               "docker-entrypoint.s…"   7 seconds ago       Up 5 seconds                            master-3
09fd8166e31f        redis               "docker-entrypoint.s…"   7 seconds ago       Up 5 seconds                            slave-1
16904e456595        redis               "docker-entrypoint.s…"   7 seconds ago       Up 5 seconds                            slave-3

再用rdm或redis-cli连一下看

图片.png

添加集群

redis部署后，只是单机部署，需要手动创建集群。
接下来随便连一台创建集群，以master1为例

root@ubuntu:/home/jasonlin/redis# docker exec -it master-1 redis-cli --cluster create 192.168.139.134:6379 \
192.168.139.134:6380 \
192.168.139.134:6381 \
192.168.139.134:6382 \
192.168.139.134:6383 \
192.168.139.134:6384 --cluster-replicas 1

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.139.134:6383 to 192.168.139.134:6379
Adding replica 192.168.139.134:6384 to 192.168.139.134:6380
Adding replica 192.168.139.134:6382 to 192.168.139.134:6381
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: 69b0a09758c68ffd96f4c869261c6b08fbfed2d3 192.168.139.134:6379
   slots:[0-5460] (5461 slots) master
M: 93d75c3497a9a89c74728e1844dc6326d8aa7a23 192.168.139.134:6380
   slots:[5461-10922] (5462 slots) master
M: 4e17362768c8b7b42f46a7a3682b4e2ff3163840 192.168.139.134:6381
   slots:[10923-16383] (5461 slots) master
S: a6521ef0f0e07cf1ade9516eb455cadc7c05d1df 192.168.139.134:6382
   replicates 69b0a09758c68ffd96f4c869261c6b08fbfed2d3
S: 038a2b1407acdc7d250b0359273cb9c89af9719b 192.168.139.134:6383
   replicates 93d75c3497a9a89c74728e1844dc6326d8aa7a23
S: 4d69e6d54ca0980c03c2d90e0ac5e57114d038b6 192.168.139.134:6384
   replicates 4e17362768c8b7b42f46a7a3682b4e2ff3163840
Can I set the above configuration? (type 'yes' to accept): 

输入yes，确定该配置

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
........
>>> Performing Cluster Check (using node 192.168.139.134:6379)
M: 69b0a09758c68ffd96f4c869261c6b08fbfed2d3 192.168.139.134:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 4e17362768c8b7b42f46a7a3682b4e2ff3163840 192.168.139.134:6381
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 93d75c3497a9a89c74728e1844dc6326d8aa7a23 192.168.139.134:6380
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: a6521ef0f0e07cf1ade9516eb455cadc7c05d1df 192.168.139.134:6382
   slots: (0 slots) slave
   replicates 69b0a09758c68ffd96f4c869261c6b08fbfed2d3
S: 038a2b1407acdc7d250b0359273cb9c89af9719b 192.168.139.134:6383
   slots: (0 slots) slave
   replicates 93d75c3497a9a89c74728e1844dc6326d8aa7a23
S: 4d69e6d54ca0980c03c2d90e0ac5e57114d038b6 192.168.139.134:6384
   slots: (0 slots) slave
   replicates 4e17362768c8b7b42f46a7a3682b4e2ff3163840
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

输入创建集群命令后，会return节点的信息以及主从关系
输入yes后，
更新节点配置
分配不同的配置epoch（纪元）给每个节点
发送MEET消息到每个节点加入到集群中
加入后，可以看到
3个Master，3个slave，前面一长串的是每个节点的ID，后面是IP，端口以及分配的slot

测试集群

从master1中set一个value，看是否会redirect到其他节点上

root@ubuntu:/home/jasonlin/redis# docker exec -it master-1 redis-cli -c
127.0.0.1:6379> set name jasonlin
-> Redirected to slot [5798] located at 192.168.139.134:6380
OK

可以看到，保存到了第5798个slot里，这个slot是分在了master2

请注意，集群模式下需要以集群方式使用redis-cli不然在redirect时候报错

root@ubuntu:/home/jasonlin/redis# docker exec -it master-1 redis-cli 
127.0.0.1:6379> set key ggg mmm
(error) MOVED 12539 192.168.139.134:6381

总结

这次使用docker-compose部署redis集群，遇到最大问题还是这个网络模式。
用bridge模式，会卡在添加集群那一块，一直在wait。
查了文档，除了通信用的端口需要开启，还需要开启一个叫做集群总线端口，即通信端口+10000
然后我也做了相应的映射，不行。
我也试过添加节点时候分别输入宿主机IP和container IP、把虚拟网卡添加到防火墙 ，也不行。

另外，也出现了诡异的问题。
查看日志发现，每个节点都做了config Epoch设置
但在IP address for this node updated时候，在master1上显示的是网关的ip，在master2、3显示的是container自己的ip