docker搭建redis高可用集群

目标：docker搭建redis高可用集群

1、架构：六个redis容器，三主三从，主从复制，主机宕机从机自动替代

2、网络架构设计：设计一个专属redis的docker网络

docker network create --driver bridge  --subnet 192.178.0.0/16 --gateway 192.178.0.1 redis

–driver bridge 使用桥接的网络模式

–subnet 子网地址，16支持255*255-2个子网地址分配

–gateway 网关地址

[root@iZwz98n1yl2yxhd3sp1etjZ ~]# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
04ba3d8a1774   bridge    bridge    local
c13d2c5e5614   host      host      local
4f1f0037adf8   none      null      local
3a2d9037d3ce   redis     bridge    local
[root@iZwz98n1yl2yxhd3sp1etjZ ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:16:3e:16:d8:60 brd ff:ff:ff:ff:ff:ff
    inet 172.16.70.247/20 brd 172.16.79.255 scope global dynamic eth0
       valid_lft 315218268sec preferred_lft 315218268sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:58:b6:cb:a6 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
22: br-3a2d9037d3ce: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:f3:5c:05:52 brd ff:ff:ff:ff:ff:ff
    inet 192.178.0.1/16 brd 192.178.255.255 scope global br-3a2d9037d3ce
       valid_lft forever preferred_lft forever
##docker0默认有一个网关

3、shell脚本批量创建redis.conf

redis集群搭建官方指南https://redis.io/topics/cluster-tutorial

for port in $(seq 1 6)
do
mkdir -p /mydata/redis/node-${port}/conf
touch /mydata/redis/node-${port}/conf/redis.conf
cat << EOF > /mydata/redis/node-${port}/conf/redis.conf
port 6379
bind 0.0.0.0
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-announce-ip 192.178.0.1${port}
cluster-announce-port 6379
cluster-announce-bus-port 16379
appendonly yes
EOF
done

解析：

seq:生成1-6循环

cat >:输出重定向

<< EOF：以EOF作为结束标志

cluster-announce-port 6379
cluster-announce-bus-port 16379：每个Redis集群中的节点都需要打开两个TCP连接。一个连接用于正常的给Client提供服务，比如6379，还有一个额外的端口（通过在这个端口号上加10000）作为数据端口，比如16379。第二个端口（本例中就是16379）用于集群总线，这是一个用二进制协议的点对点通信信道。这个集群总线（Cluster bus）用于节点的失败侦测、配置更新、故障转移授权，等等。客户端从来都不应该尝试和这些集群总线端口通信，它们只应该和正常的Redis命令端口进行通信。注意，确保在你的防火墙中开放着两个端口，否则，Redis集群节点之间将无法通信。命令端口和集群总线端口的偏移量总是10000。

4、docker启动六个redis容器

for port in $(seq 1 6)
do
docker run -it -p 637${port}:6379 -p 1637${port}:16379 --name redis-${port} --net redis --ip 192.178.0.1${port} -v /mydata/redis/node-${port}/data:/data -v /mydata/redis/node-${port}/conf/redis.conf:/etc/redis/redis.conf -d redis redis-server /etc/redis/redis.conf
done

解析：启动后面追加命令启动server

-p:端口绑定

-v:挂载外部配置和数据安装（docker运行是没有配置文件redis.conf的，需要挂载到宿主机的文件上面）

踩坑：

docker: Error response from daemon: Invalid address 192.168.0.11: It does not belong to any of this network's subnets.
docker: Error response from daemon: Invalid address 192.168.0.12: It does not belong to any of this network's subnets.
docker: Error response from daemon: Invalid address 192.168.0.13: It does not belong to any of this network's subnets.
docker: Error response from daemon: Invalid address 192.168.0.14: It does not belong to any of this network's subnets.
docker: Error response from daemon: Invalid address 192.168.0.15: It does not belong to any of this network's subnets.
docker: Error response from daemon: Invalid address 192.168.0.16: It does not belong to any of this network's subnets.

ip地址没有分配正确，不属于分配的网络的子网范围

排查：

[root@iZwz98n1yl2yxhd3sp1etjZ /]# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
04ba3d8a1774   bridge    bridge    local
c13d2c5e5614   host      host      local
4f1f0037adf8   none      null      local
3a2d9037d3ce   redis     bridge    local
[root@iZwz98n1yl2yxhd3sp1etjZ /]# docker network inspect redis
[
    {
        "Name": "redis",
        "Id": "3a2d9037d3ceade2d239727256f3f66f2c3e73b1bb421baa36552b81f3b4127e",
        "Created": "2022-03-19T11:59:58.772370048+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.178.0.0/16",
                    "Gateway": "192.178.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
[root@iZwz98n1yl2yxhd3sp1etjZ /]# 

问题：配置的redis网络子网为"Subnet": “192.178.0.0/16”，但是运行的时候是-p 192.168.0.1${port}，粗心导致。

解决：

docker rm $(docker ps -aq)
docker rmi redis
重新执行运行脚本

5、为redis-cli创建集群

未搭建集群查看：

127.0.0.1:6379> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

搭建集群:

docker exec -it redis-1 /bin/sh
redis-cli --cluster create 192.178.0.11:6379 192.178.0.12:6379 192.178.0.13:6379 192.178.0.14:6379 192.178.0.15:6379 192.178.0.16:6379 --cluster-replicas 1
# redis-cli -c
127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:118
cluster_stats_messages_pong_sent:127
cluster_stats_messages_sent:245
cluster_stats_messages_ping_received:122
cluster_stats_messages_pong_received:118
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:245
127.0.0.1:6379> cluster nodes
0a0257ee7e69899b54472d996db1ab806c0bb55e 192.178.0.11:6379@16379 myself,master - 0 1647677931000 1 connected 0-5460
2cab4d3d0cd62544f7f5c3f04737f676a5c7b325 192.178.0.14:6379@16379 slave 5334b88cec1fe35ee5c238d602556152b921a876 0 1647677935025 3 connected
50ac37e1b1e971adaf449386faec30b40abab26b 192.178.0.16:6379@16379 slave 7e98e45d51856a2fccce1767622525645e1c61c9 0 1647677933521 2 connected
4e740ae98d0994fa23221b010ab5f77654b1991d 192.178.0.15:6379@16379 slave 0a0257ee7e69899b54472d996db1ab806c0bb55e 0 1647677934000 1 connected
5334b88cec1fe35ee5c238d602556152b921a876 192.178.0.13:6379@16379 master - 0 1647677934000 3 connected 10923-16383
7e98e45d51856a2fccce1767622525645e1c61c9 192.178.0.12:6379@16379 master - 0 1647677935000 2 connected 5461-10922

6、测试高可用

由50ac37e1b1e971adaf449386faec30b40abab26b 192.178.0.16:6379@16379 slave 7e98e45d51856a2fccce1767622525645e1c61c9 0 1647677933521 2 connected 可知6是2的从机

192.178.0.13:6379> set cao 176
-> Redirected to slot [10089] located at 192.178.0.12:6379
OK
192.178.0.12:6379> get cao
"176"
192.178.0.12:6379> 
# exit
[root@iZwz98n1yl2yxhd3sp1etjZ /]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                              NAMES
35d664093077   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16376->16376/tcp, 0.0.0.0:6376->6379/tcp   redis-6
8653d9e8e461   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16375->16375/tcp, 0.0.0.0:6375->6379/tcp   redis-5
0de4693aa971   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16374->16374/tcp, 0.0.0.0:6374->6379/tcp   redis-4
cecf6af415a8   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16373->16373/tcp, 0.0.0.0:6373->6379/tcp   redis-3
7b83ace71801   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16372->16372/tcp, 0.0.0.0:6372->6379/tcp   redis-2
2b43f6f4de05   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16371->16371/tcp, 0.0.0.0:6371->6379/tcp   redis-1
[root@iZwz98n1yl2yxhd3sp1etjZ /]# docker stop redis-2
redis-2
[root@iZwz98n1yl2yxhd3sp1etjZ /]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                              NAMES
35d664093077   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16376->16376/tcp, 0.0.0.0:6376->6379/tcp   redis-6
8653d9e8e461   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16375->16375/tcp, 0.0.0.0:6375->6379/tcp   redis-5
0de4693aa971   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16374->16374/tcp, 0.0.0.0:6374->6379/tcp   redis-4
cecf6af415a8   redis     "docker-entrypoint.s…"   14 minutes ago   Up 14 minutes   0.0.0.0:16373->16373/tcp, 0.0.0.0:6373->6379/tcp   redis-3
2b43f6f4de05   redis     "docker-entrypoint.s…"   15 minutes ago   Up 14 minutes   0.0.0.0:16371->16371/tcp, 0.0.0.0:6371->6379/tcp   redis-1
[root@iZwz98n1yl2yxhd3sp1etjZ /]# docker exec -it redis-1 /bin/sh
# redis-cli -c
127.0.0.1:6379> get cao
-> Redirected to slot [10089] located at 192.178.0.16:6379
"176"
192.178.0.16:6379> get cao
"176"
192.178.0.16:6379> cluster nodes
5334b88cec1fe35ee5c238d602556152b921a876 192.178.0.13:6379@16379 master - 0 1647678241000 3 connected 10923-16383
0a0257ee7e69899b54472d996db1ab806c0bb55e 192.178.0.11:6379@16379 master - 0 1647678241000 1 connected 0-5460
7e98e45d51856a2fccce1767622525645e1c61c9 192.178.0.12:6379@16379 master,fail - 1647678161619 1647678159511 2 connected
2cab4d3d0cd62544f7f5c3f04737f676a5c7b325 192.178.0.14:6379@16379 slave 5334b88cec1fe35ee5c238d602556152b921a876 0 1647678241856 3 connected
50ac37e1b1e971adaf449386faec30b40abab26b 192.178.0.16:6379@16379 myself,master - 0 1647678240000 7 connected 5461-10922
4e740ae98d0994fa23221b010ab5f77654b1991d 192.178.0.15:6379@16379 slave 0a0257ee7e69899b54472d996db1ab806c0bb55e 0 1647678242859 1 connected
192.178.0.16:6379> 

redis-cli -c:以集群模式

说明：2写入数据，宕机后6替代并且有数据。
总结：主从复制，主机宕机后从机替代成为主机，即使主机恢复后也只能做小弟！！！