Redis Cluster 6.0版本安装部署
1. 环境说明
1.1 版本说明
本安装部署步骤针对Redis 6.0.6 版本，涉及到部署工具更新Redis 5.x 以上版本该文不可作为参考。
Redis版本3或4，需使用旧版本的 redis-trib.rb的 工具，5.0版本后集群管理命令等功能集成到了
redis-cli
| 操作系统版本 | CentOS Linux release 7.5.1804 (Core) |
|---|---|
| Reids数据版本 | Redis 6.0.0 |
1.2 目录规划
该手册为测试环境部署示例，其它环境部署对应目录路径、命名需相应调整。
| 实例角色 | 实例IP | 实例端口 | 数据目录 |
|---|---|---|---|
| Redis node1 Master | 127.0.0.1 | 6401 | /data/redis/data/6401 |
| Redis node1 Slave | 127.0.0.1 | 6402 | /data/redis/data/6402 |
| Redis node2 Master | 127.0.0.1 | 6403 | /data/redis/data/6403 |
| Redis node2 Slave | 127.0.0.1 | 6404 | /data/redis/data/6404 |
| Redis node3 Master | 127.0.0.1 | 6405 | /data/redis/data/6405 |
| Redis node3 Slave | 127.0.0.1 | 6406 | /data/redis/data/6406 |
2. 环境准备
2.1 目录创建
mkdir -p /data/redis/{bin,data,conf,log}
mkdir -p /data/redis/data/{6401..6406}
2.2 安装Redis
tar xfz redis-6.0.6.tar.gz
cd redis-6.0.6
yum install -y gcc-c++ centos-release-scl devtoolset-9-gcc devtoolset-9-gcc-c++ devtoolset-9-binutils
scl enable devtoolset-9 bash
cd ./deps/
make hiredis lua linenoise jemalloc
cd ../
make
make PREFIX=/data/redis/bin install
# 配置环境变量
vi /etc/profile
REDIS_BIN=/data/redis/base/bin
export PATH=$PATH:$REDIS_BIN
提前准备好各实例的配置文件，以6401为例
bind 0.0.0.0
port 6401
pidfile /var/run/redis_6401.pid
logfile "/data/redis/log/redis_6401.log"
dir /data/redis/data/6401
cluster-enabled yes
cluster-node-timeout 15000
cluster-config-file "nodes-6401.conf"
daemonize yes
protected-mode yes
tcp-backlog 511
timeout 0
tcp-keepalive 300
supervised no
loglevel notice
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
启动各Redis实例
redis-server /data/redis/conf/redis_6401.conf
redis-server /data/redis/conf/redis_6402.conf
redis-server /data/redis/conf/redis_6403.conf
redis-server /data/redis/conf/redis_6404.conf
redis-server /data/redis/conf/redis_6405.conf
redis-server /data/redis/conf/redis_6406.conf
2.3 搭建Redis Cluster
Redis 6.x版本后使用 redis-cli工具对Redis Cluster进行集群创建、管理，功能与使用方法与redis-trib.rb相近。
# cluster-replicas 设置为1表示分配1个Slave节点
redis-cli --cluster-replicas 1 --cluster create \
127.0.0.1:6401 \
127.0.0.1:6402 \
127.0.0.1:6403 \
127.0.0.1:6404 \
127.0.0.1:6405 \
127.0.0.1:6406
# 输出信息如下
>>> Performing hash slots allocation on 6 nodes... // 可以看到16383个槽均匀分发到3个Redis Master节点
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 127.0.0.1:6405 to 127.0.0.1:6401 // 为每一个Redis Master配置了Slave节点
Adding replica 127.0.0.1:6406 to 127.0.0.1:6402
Adding replica 127.0.0.1:6404 to 127.0.0.1:6403
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401
slots:[0-5460] (5461 slots) master
M: eddff016e90610c84e37231902c640d6dc6746a4 127.0.0.1:6402
slots:[5461-10922] (5462 slots) master
M: 32b8bd5bd189d3608928b1dc09411803e2d46459 127.0.0.1:6403
slots:[10923-16383] (5461 slots) master
S: ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404
replicates 32b8bd5bd189d3608928b1dc09411803e2d46459
S: e48a1af608b64f43d4adfd1e5dfd480b71bbe92a 127.0.0.1:6405
replicates b2b8d874cb650462d05e411f0947e258004f7784
S: b97736a34db550f9f3c460e88d90bf05f4dd286a 127.0.0.1:6406
replicates eddff016e90610c84e37231902c640d6dc6746a4
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 127.0.0.1:6401)
M: b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 32b8bd5bd189d3608928b1dc09411803e2d46459 127.0.0.1:6403
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: b97736a34db550f9f3c460e88d90bf05f4dd286a 127.0.0.1:6406
slots: (0 slots) slave
replicates eddff016e90610c84e37231902c640d6dc6746a4
S: e48a1af608b64f43d4adfd1e5dfd480b71bbe92a 127.0.0.1:6405
slots: (0 slots) slave
replicates b2b8d874cb650462d05e411f0947e258004f7784
S: ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404
slots: (0 slots) slave
replicates 32b8bd5bd189d3608928b1dc09411803e2d46459
M: eddff016e90610c84e37231902c640d6dc6746a4 127.0.0.1:6402
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
检查集群是否搭建完成、槽完全被分配
redis-cli --cluster check 127.0.0.1:6401
# 输出信息如下
127.0.0.1:6401 (b2b8d874...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6403 (32b8bd5b...) -> 0 keys | 5461 slots | 1 slaves.
127.0.0.1:6402 (eddff016...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
3. 集群运维
3.1 添加新节点到集群
添加新节点只需要如2.2安装Redis一样，准备一个空实例。
新节点可以作为主节点使用（需要迁移数据到新节点）、也可以作为从节点使用。
新节点刚加入集群后由于未分配槽，所以不会存储任何数据、也不会参与选举。
发
## 新节点准备，以下示例准备7401做为已创建好的新节点
redis-server 0.0.0.0:7401
## 添加新节点到集群，第一个地址为新节点IP和端口，第二个地址为集群中任意存在的节点即可
# redis-cli -p 6401 --cluster add-node 127.0.0.1:7401 127.0.0.1:6401
>>> Adding node 127.0.0.1:7401 to cluster 127.0.0.1:6401
>>> Performing Cluster Check (using node 127.0.0.1:6401)
M: b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404
slots: (0 slots) slave
replicates 32b8bd5bd189d3608928b1dc09411803e2d46459
......
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:7401 to make it join the cluster.
[OK] New node added correctly.
## 新节点添加成功后，可以使用 cluster nodes命令查看集群节点列表信息
# redis-cli -p 6401 cluster nodes
ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404@16404 slave 32b8bd5bd189d3608928b1dc09411803e2d46459 0 1597825031000 3 connected
498faeeffcddcbe0518cf0c559e4be308fdc9a48 127.0.0.1:7401@17401 master - 0 1597825031887 0 connected
......
3.2 给新节点分配槽
Redis Cluster集群如果16384个槽全部被分配，那么分配槽给新加节点则需要使用reshard命令
## 下面地址为集群中任意地址即可
# redis-cli -p 6401 --cluster reshard 127.0.0.1:6401
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 3000 ## 需要分配多少槽 比如 3000
What is the receiving node ID? 498faeeffcddcbe0518cf0c559e4be308fdc9a48 ## 将槽分配给哪个node_id，这里选择新节点7401的node_id
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: all ## 槽的source源于哪个node_id，如果填入 all，则表示从集群所有主节点均匀拆分一定数量的槽用于分配
Moving slot 7126 from eddff016e90610c84e37231902c640d6dc6746a4
Moving slot 7127 from eddff016e90610c84e37231902c640d6dc6746a4
...
Moving slot 12587 from 32b8bd5bd189d3608928b1dc09411803e2d46459
Moving slot 12588 from 32b8bd5bd189d3608928b1dc09411803e2d46459
Do you want to proceed with the proposed reshard plan (yes/no)? yes ## 是否同意该reshard方案
...
Moving slot 7126 from 127.0.0.1:6402 to 127.0.0.1:7401:
Moving slot 7127 from 127.0.0.1:6402 to 127.0.0.1:7401:
...
Moving slot 12587 from 127.0.0.1:6403 to 127.0.0.1:7401:
Moving slot 12588 from 127.0.0.1:6403 to 127.0.0.1:7401:
## 分配完毕后，通过cluster nodes查看槽位分配情况
# redis-cli -p 6401 cluster nodes
ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404@16404 slave 32b8bd5bd189d3608928b1dc09411803e2d46459 0 1597826341549 3 connected
498faeeffcddcbe0518cf0c559e4be308fdc9a48 127.0.0.1:7401@17401 master - 0 1597826340000 7 connected 0-1665 5461-7127 10923-12588 ## 数目大概在3000左右
b97736a34db550f9f3c460e88d90bf05f4dd286a 127.0.0.1:6406@16406 slave eddff016e90610c84e37231902c640d6dc6746a4 0 1597826342553 2 connected
b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401@16401 myself,master - 0 1597826341000 1 connected 1666-5460
e48a1af608b64f43d4adfd1e5dfd480b71bbe92a 127.0.0.1:6405@16405 slave b2b8d874cb650462d05e411f0947e258004f7784 0 1597826341000 1 connected
eddff016e90610c84e37231902c640d6dc6746a4 127.0.0.1:6402@16402 master - 0 1597826340545 2 connected 7128-10922
32b8bd5bd189d3608928b1dc09411803e2d46459 127.0.0.1:6403@16403 master - 0 1597826340000 3 connected 12589-16383
3.3 新节点置为从库
新节点加入集群后，不但可以做为单独分片的主节点使用，还可用配置为集群中现有主节点的从库
新节点置为从库有两种方式：
-
第一种是随机被分配到从库较少的主节点（--cluster-slave）
## 准备一个新节点 7402 redis-server 0.0.0.0:7402 ## 使用--cluster-slave选项，注意第二个地址参数随意指定集群中任意节点都可 # redis-cli -p 6401 --cluster add-node 127.0.0.1:7402 127.0.0.1:6401 --cluster-slave >>> Adding node 127.0.0.1:7402 to cluster 127.0.0.1:6401 >>> Performing Cluster Check (using node 127.0.0.1:6401) M: b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401 slots:[1666-5460] (3795 slots) master 1 additional replica(s) S: ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404 slots: (0 slots) slave replicates 32b8bd5bd189d3608928b1dc09411803e2d46459 M: 498faeeffcddcbe0518cf0c559e4be308fdc9a48 127.0.0.1:7401 ## 7401该主节点不存在从库 slots:[0-1665],[5461-7127],[10923-12588] (4999 slots) master S: b97736a34db550f9f3c460e88d90bf05f4dd286a 127.0.0.1:6406 slots: (0 slots) slave replicates eddff016e90610c84e37231902c640d6dc6746a4 S: e48a1af608b64f43d4adfd1e5dfd480b71bbe92a 127.0.0.1:6405 slots: (0 slots) slave replicates b2b8d874cb650462d05e411f0947e258004f7784 M: eddff016e90610c84e37231902c640d6dc6746a4 127.0.0.1:6402 slots:[7128-10922] (3795 slots) master 1 additional replica(s) ## 其它主节点都存在从库 M: 32b8bd5bd189d3608928b1dc09411803e2d46459 127.0.0.1:6403 slots:[12589-16383] (3795 slots) master 1 additional replica(s) [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. Automatically selected master 127.0.0.1:7401 ## 集群自动选择新节点与从库较少的主节点7401建立复制 >>> Send CLUSTER MEET to node 127.0.0.1:7402 to make it join the cluster. Waiting for the cluster to join >>> Configure node as replica of 127.0.0.1:7401. [OK] New node added correctly. ## 检查到复制已正常建立 # redis-cli -p 6401 cluster nodes|grep 740 498faeeffcddcbe0518cf0c559e4be308fdc9a48 127.0.0.1:7401@17401 master - 0 1597827661107 7 connected 0-1665 5461-7127 10923-12588 96899c1d4264e00ab1f9391a504901f842c3e68c 127.0.0.1:7402@17402 slave 498faeeffcddcbe0518cf0c559e4be308fdc9a48 0 1597827658095 7 connected
-
第二种指定主节点建立复制关系，灾备场景、跨机房场景非常适用（--cluster-slave --cluster-master-id node_id）
## 准备一个新节点 7403 redis-server 0.0.0.0:7403 ## 除--cluster-slave外，还需使用--cluster-master-id指定需要复制的主节点node_id，该示例选择7401该主节点作为新节点的主库 # redis-cli -p 6401 --cluster add-node 127.0.0.1:7403 127.0.0.1:6401 --cluster-slave --cluster-master-id 498faeeffcddcbe0518cf0c559e4be308fdc9a48 >>> Adding node 127.0.0.1:7403 to cluster 127.0.0.1:6401 >>> Performing Cluster Check (using node 127.0.0.1:6401) M: b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401 slots:[1666-5460] (3795 slots) master 1 additional replica(s) ...... [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node 127.0.0.1:7403 to make it join the cluster. Waiting for the cluster to join >>> Configure node as replica of 127.0.0.1:7401. [OK] New node added correctly. ## 检查复制已正常建立 # redis-cli -p 6401 cluster nodes|grep 740 498faeeffcddcbe0518cf0c559e4be308fdc9a48 127.0.0.1:7401@17401 master - 0 1597827937056 7 connected 0-1665 5461-7127 10923-12588 96899c1d4264e00ab1f9391a504901f842c3e68c 127.0.0.1:7402@17402 slave 498faeeffcddcbe0518cf0c559e4be308fdc9a48 0 1597827939000 7 connected 4261a11603a1da2925e33266873e2026f4dfb339 127.0.0.1:7403@17403 slave 498faeeffcddcbe0518cf0c559e4be308fdc9a48 0 1597827939063 7 connected
4. 集群命令
4.1 集群全局命令
-
cluster nodes列出Redis Cluster各节点信息与槽位分布
# redis-cli -p 6401 cluster nodes
ef3c20a5dd0be9113b440550e45501faa003e18f 127.0.0.1:6404@16404 slave 32b8bd5bd189d3608928b1dc09411803e2d46459 0 1597824012241 3 connected
b97736a34db550f9f3c460e88d90bf05f4dd286a 127.0.0.1:6406@16406 slave eddff016e90610c84e37231902c640d6dc6746a4 0 1597824012000 2 connected
b2b8d874cb650462d05e411f0947e258004f7784 127.0.0.1:6401@16401 myself,master - 0 1597824011000 1 connected 0-5460
e48a1af608b64f43d4adfd1e5dfd480b71bbe92a 127.0.0.1:6405@16405 slave b2b8d874cb650462d05e411f0947e258004f7784 0 1597824013243 1 connected
eddff016e90610c84e37231902c640d6dc6746a4 127.0.0.1:6402@16402 master - 0 1597824011000 2 connected 5461-10922
32b8bd5bd189d3608928b1dc09411803e2d46459 127.0.0.1:6403@16403 master - 0 1597824012000 3 connected 10923-16383
-
cluster info输出集群整体信息
# redis-cli -p 6401 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:773
cluster_stats_messages_pong_sent:780
cluster_stats_messages_sent:1553
cluster_stats_messages_ping_received:780
cluster_stats_messages_pong_received:769
cluster_stats_messages_received:1549
4.2 集群节点命令
- cluster replicate 将当前节点与指定node_id的主节点建立复制
- cluster forget 将指定node_id的节点从集群中移除