Redis Cluster 集群部署

2.4：Redis Cluster 部署

2.4.1：部署 Redis Cluster 的前提

1. 时间同步；
2. Redis Node 均采用相同的硬件配置、相同的密码、相同的 Redis 版本；
3. Redis Node 必须开启的参数：
   • cluster-enabled yes：启用集群，开启后 Redis 进程会有 Cluster 显示；
   • cluster-config-file nodes-6380.conf：指定集群配置文件，此文件由 Redis Cluster 自动创建和维护，开启即可，不需要任何手动配置；
4. 所有 Redis Node 必须没有任何数据；

2.4.2：拓扑图

共 6 台 Redis 服务器组成 Redis Cluster，三主三从，即集群中的 Redis Node 为 3 个；
奇数为 Master，偶数为 Slave，方便后期维护管理时辨别；

• Masters：Redis1、Redis3、Redis5；

• Slaves：Redis2、Redis4、Redis6；

各 Redis 服务器编译安装 redis-4.0.14 和 redis-5.0.10（为了测试 redis 5 版本的集群创建和维护）；

2.4.3：各节点配置并启动 Redis

编辑各节点的 Redis 配置文件：

配置 masterauth（如果有 requirepass）；
开启集群；
指定集群配置文件；

masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf

各节点启动为单机 Redis：

systemctl start redis && systemctl enable redis

2.4.4：创建集群（Redis-3/4）

准备 redis-trib.rb 使用环境

Redis 3 和 4 版本创建集群时，需要使用集群管理工具 redis-trib.rb，这个工具是 Redis 官方推出的管理 Redis 集群的工具，集成在 Redis 源码的 src 目录下，是基于 Redis 提供的集群命令而封装的简单、便捷、实用的操作工具；

redis-trib.rb 是 Redis 作者用 Ruby 开发完成的，CentOS yum 安装的 Ruby 存在版本较低问题，需要编译安装 Ruby：

只需在一台 Redis 服务器上准备环境即可；

[root@redis1 ~]# yum remove ruby rubygems -y
[root@redis1 ~]# cd /usr/local/src
[root@s1 src]# wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz
[root@redis1 src]# tar xf ruby-2.5.5.tar.gz 
[root@redis1 src]# cd ruby-2.5.5/
[root@redis1 ruby-2.5.5]# ./configure
[root@redis1 ruby-2.5.5]# make -j 2
[root@redis1 ruby-2.5.5]# make install

安装 Ruby 的 Redis 模块：

[root@redis1 ruby-2.5.5]# gem install redis

如果无法在线安装 Ruby 的 Redis 模块，可以先下载 Redis 模块（https://rubygems.org/gems/redis），再离线安装：

gem install -l redis-3.3.0.gem

创建 redis-trib.rb 软链接：

[root@redis1 ~]# ln -sv /usr/local/src/redis-4.0.14/src/redis-trib.rb /usr/bin/redis-trib
‘/usr/bin/redis-trib’ -> ‘/usr/local/src/redis-4.0.14/src/redis-trib.rb’

查看 redis-trib.rb 使用帮助：

[root@redis1 ~]# redis-trib 
Usage: redis-trib   

  create          host1:port1 ... hostN:portN    # 创建集群，指定集群中的节点；
                  --replicas                # 指定每个Master的副本数量（即一个Master有几个Slave）；
  check           host:port						 # 检查集群状态
  info            host:port						 # 查看集群信息
  fix             host:port						 # 修复集群
                  --timeout 
  reshard         host:port						 # 重新分片（热迁移slots）；
                  --from 
                  --to 
                  --slots 
                  --yes
                  --timeout 
                  --pipeline 
  rebalance       host:port						 # 平衡集群中各节点的slot数量；
                  --weight 
                  --auto-weights
                  --use-empty-masters
                  --timeout 
                  --simulate
                  --pipeline 
                  --threshold 
  add-node        new_host:new_port existing_host:existing_port
  												 # 向集群中添加节点；
                  --slave
                  --master-id 
  del-node        host:port node_id				 # 从集群中删除节点；
  set-timeout     host:port milliseconds		 # 设置节点的超时时间（单位毫秒）；
  call            host:port command arg arg .. arg
  												 # 在集群的所有节点上执行命令；
  import          host:port						 # 导入外部redis服务器的数据到当前集群；
                  --from 
                  --copy
                  --replace
  help            (show this help)

For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

为 Ruby Redis 模块配置 Redis 连接密码

[root@redis1 ~]# vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.2.5/lib/redis/client.rb
class Redis
  class Client
    # Defaults are also used for converting string keys to symbols.
    DEFAULTS = {
    ……
    password: 123456,

创建 Redis Cluster

创建时，配置在前的节点会成为 Master：

[root@redis1 ~]# redis-trib create --replicas 1 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.201:6379
192.168.1.203:6379
192.168.1.205:6379
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
   slots:0-5460 (5461 slots) master
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
   slots:5461-10922 (5462 slots) master
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
   slots:10923-16383 (5461 slots) master
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
   replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
   replicates 469e0c16495598b5fadba28f856203e206464b99
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
   replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
   slots: (0 slots) slave
   replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
   slots: (0 slots) slave
   replicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
   slots: (0 slots) slave
   replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

2.4.5：创建集群（Redis-5）

Redis 5 版本可以直接使用 redis-cli 创建 Redis Cluster：

[root@redis1 ~]# redis-cli -a 123456 --cluster create 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379 --cluster-replicas 1

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
   slots:[0-5460] (5461 slots) master
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
   slots:[5461-10922] (5462 slots) master
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
   slots:[10923-16383] (5461 slots) master
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
   replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
   replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
   replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e

Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
   slots: (0 slots) slave
   replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
   slots: (0 slots) slave
   replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
   slots: (0 slots) slave
   replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-cli -a 123456 --cluster create 192.168.7.101:6379
192.168.7.101:6380 192.168.7.102:6379 192.168.7.102:6380 1
92.168.7.103:6379 192.168.7.103:6380 --cluster-replicas 1

2.4.6：验证集群创建

集群中任一节点均可查看集群状态和信息；

CLUSTER 指令

查看集群状态：

127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:127
cluster_stats_messages_pong_sent:125
cluster_stats_messages_meet_sent:3
cluster_stats_messages_sent:255
cluster_stats_messages_ping_received:121
cluster_stats_messages_pong_received:130
cluster_stats_messages_meet_received:4
cluster_stats_messages_received:255

查看 Node 对应关系：

127.0.0.1:6379> CLUSTER NODES
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379@16379 master - 0 1609749076000 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379@16379 myself,slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609749076000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 master - 0 1609749077996 1 connected 0-5460
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609749075000 4 connected
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609749076973 5 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609749075960 3 connected 10923-16383

redis-trib.rb（Redis-3/4）

指定集群中的任一节点均可查看集群的状态信息；

查看集群信息：

[root@redis1 ~]# redis-trib info 192.168.1.206:6379
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.201:6379 (469e0c16...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.

检查集群状态：

[root@redis1 ~]# redis-trib check 192.168.1.203:6379
>>> Performing Cluster Check (using node 192.168.1.203:6379)
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
   slots: (0 slots) slave
   replicates 469e0c16495598b5fadba28f856203e206464b99
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
   slots: (0 slots) slave
   replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
   slots: (0 slots) slave
   replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-cli（Redis-5）

[root@redis1 ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
   slots: (0 slots) slave
   replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
   slots: (0 slots) slave
   replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
   slots: (0 slots) slave
   replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群配置文件

Redis-4.0.14：

[root@redis1 ~]# cat /apps/redis/data/nodes-6379.conf 
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379@16379 master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379@16379 slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

Redis-5.0.10：

[root@redis1 ~]# cat /apps/redis5/data/nodes-6379.conf 
df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379@16379 master - 0 1609811160000 3 connected 10923-16383
c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379@16379 master - 0 1609811161000 2 connected 5461-10922
b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379@16379 slave c2b48542344ece56e69d8ca9404ee29a48ae7b8e 0 1609811162000 6 connected
90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379@16379 slave feb6b43233fdcc01e0d5425fd03e0116b74f0833 0 1609811162884 5 connected
d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379@16379 slave df921d013a6cbf1588c1ad51809435fe39f6c25c 0 1609811162000 4 connected
feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379@16379 myself,master - 0 1609811160000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

2.4.7：测试集群读写

集群数据写入

Redis1 节点上尝试写入：

提示 foo1 这个 key 的 CRC 结果被调度到了 13431 槽位，该槽位在 192.168.1.205:6379 节点上；

[root@redis1 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
(error) MOVED 13431 192.168.1.205:6379

Redis5 节点 写入 foo1：

[root@redis5 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
OK

各节点只保存各自槽位的数据，所以 Redis1 和 Redis3 都没有刚刚写入的 foo1：

[root@redis3 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> KEYS foo1
(empty list or set)

集群数据读取

因为各节点只保存各自槽位的数据，所以数据的读取也只能到相应的节点上进行；

而且集群中的 Slave 节点，读写服务均不提供；

查看 Redis5 的 Slave 节点：

Redis5 的 ID 为：1b897eda6acfb806eb27326f151e4067f0d47f7b；
可以查到对应的 Slave 为 Redis2（192.168.1.202）；

[root@redis1 ~]# cat /apps/redis/data/nodes-6379.conf 
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379@16379 master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379@16379 slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0

到 Redis2 读取刚刚写入到 Redis5 的 foo1：

可以查到有 foo1 的数据，但是读取还是得到 Redis5 上；

127.0.0.1:6379> KEYS foo1
1) "foo1"
127.0.0.1:6379> GET foo1
(error) MOVED 13431 192.168.1.205:6379