Redis还是挺好玩的,今天测试了集群的添加、删除节点、重分配slot等。更深入的理解redis的游戏规则。步骤繁多,但是详细。
环境解释:
我是在一台Centos 6.9上测试的,各个redis节点以端口号区分。文中针对各个redis,我只是以端口号代表。
~~~~Master Node~~~~~ 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 ~~~~Slave Node~~~~~ 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002 ~~~~用来折腾的Node~~~~~ 172.16.32.116:9000 172.16.32.116:9001
1. 创建redis集群
注:更多redis集群创建,请参阅
Redis Cluster集群部署搭建
# ./redis-trib.rb create --replicas 1 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 172.16.32.116:8000 172.16.32.116:8001 172.16.32.116:8002 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 172.16.32.116:7000 172.16.32.116:7001 172.16.32.116:7002 Adding replica 172.16.32.116:8000 to 172.16.32.116:7000 Adding replica 172.16.32.116:8001 to 172.16.32.116:7001 Adding replica 172.16.32.116:8002 to 172.16.32.116:7002 M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:0-5460 (5461 slots) master M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5461-10922 (5462 slots) master M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:10923-16383 (5461 slots) master S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8 S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 replicates 273107e5ac994d675749be0979556e761274bb93 S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 replicates 88fe075375295b59eabe69fa1438ed7c7c314f43 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 172.16.32.116:7000) M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:0-5460 (5461 slots) master M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5461-10922 (5462 slots) master M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:10923-16383 (5461 slots) master M: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slots: (0 slots) master replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8 M: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slots: (0 slots) master replicates 273107e5ac994d675749be0979556e761274bb93 M: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slots: (0 slots) master replicates 88fe075375295b59eabe69fa1438ed7c7c314f43 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
2. 检查集群状态
# ./redis-trib.rb check 172.16.32.116:7000 >>> Performing Cluster Check (using node 172.16.32.116:7000) M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slots: (0 slots) slave replicates 88fe075375295b59eabe69fa1438ed7c7c314f43 S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slots: (0 slots) slave replicates 273107e5ac994d675749be0979556e761274bb93 S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slots: (0 slots) slave replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
~~~~~~~~~~~~~~~~~~~~~~~~~~~添加节点~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3. 添加新节点redis-trib.rb add-node 新增节点名 原集群节点名
# ./redis-trib.rb add-node 172.16.32.116:9000 172.16.32.116:7000 >>> Adding node 172.16.32.116:9000 to cluster 172.16.32.116:7000 >>> Performing Cluster Check (using node 172.16.32.116:7000) M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slots: (0 slots) slave replicates 88fe075375295b59eabe69fa1438ed7c7c314f43 S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slots: (0 slots) slave replicates 273107e5ac994d675749be0979556e761274bb93 S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slots: (0 slots) slave replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage...
4. 查看当前集群状态,9000是一个空的Master
# ./redis-cli -p 9000 cluster nodes a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505321254767 1 connected 0-5460 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505321250759 2 connected 5461-10922 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505321251761 3 connected 10923-16383 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505321255769 3 connected aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505321253765 1 connected a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505321256771 2 connected
5. 为9000分配slot, redis的solt是固定的,就16384个,只能从其他节点获取slot,然后分配到9000
# ./redis-trib.rb reshard 172.16.32.116:9000 >>> Performing Cluster Check (using node 172.16.32.116:9000) M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 slots: (0 slots) master 0 additional replica(s) M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slots: (0 slots) slave replicates 88fe075375295b59eabe69fa1438ed7c7c314f43 S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slots: (0 slots) slave replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8 M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slots: (0 slots) slave replicates 273107e5ac994d675749be0979556e761274bb93 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? What is the receiving node ID? 364ae8322ab2627e25b05d45b702448c74afad10 Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:all Ready to move 300 slots. Source nodes: M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) Destination node: M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 slots: (0 slots) master 0 additional replica(s) Resharding plan: Moving slot 5461 from 273107e5ac994d675749be0979556e761274bb93 Moving slot 5469 from 273107e5ac994d675749be0979556e761274bb93 Do you want to proceed with the proposed reshard plan (yes/no)? yes Moving slot 5461 from 172.16.32.116:7001 to 172.16.32.116:9000:
6. 可以看到,9000已经分配到的slot是0-98 5461-5561 10923-11021
# ./redis-cli -p 9000 cluster nodes a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 master - 0 1505324905062 1 connected 99-5460 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 master - 0 1505324910075 2 connected 5562-10922 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 myself,master - 0 0 7 connected 0-98 5461-5561 10923-11021 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slave 88fe075375295b59eabe69fa1438ed7c7c314f43 0 1505324908070 3 connected aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slave a0b91f48e933c1f1d427c54917ce970bd25d29f8 0 1505324911077 1 connected 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 master - 0 1505324902057 3 connected 11022-16383 a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slave 273107e5ac994d675749be0979556e761274bb93 0 1505324909073 2 connected
~~~~~~~~~~~~~~~~~~~~~~~~~~~将9000变为slave~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7. 希望将9000变成7000的slave,但是由于有slot,执行失败,需要先转移slot
# redis-cli -c -p 9000 cluster replicate a0b91f48e933c1f1d427c54917ce970bd25d29f8 (error) ERR To set a master the node must be empty and without assigned slots.
8. 删除节点也是不可以的,总之,只要上面有slot。redis是不会让你删除的,而且需要人工介入,rebalance这些slot之后才行
# ./redis-trib.rb del-node 172.16.32.116:9000 364ae8322ab2627e25b05d45b702448c74afad10 >>> Removing node 364ae8322ab2627e25b05d45b702448c74afad10 from cluster 172.16.32.116:9000 [ERR] Node 172.16.32.116:9000 is not empty! Reshard data away and try again.
9.重新分配9000的slot到7000上
# ./redis-trib.rb reshard 172.16.32.116:9000 <<<<<<重新分配slot >>> Performing Cluster Check (using node 172.16.32.116:9000) M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 slots:0-98,5461-5561,10923-11021 (299 slots) master 0 additional replica(s) M: a0b91f48e933c1f1d427c54917ce970bd25d29f8 172.16.32.116:7000 slots:99-5460 (5362 slots) master 1 additional replica(s) M: 273107e5ac994d675749be0979556e761274bb93 172.16.32.116:7001 slots:5562-10922 (5361 slots) master 1 additional replica(s) S: 3d27f60a1cc4d9c8f09aca928b03f0e083722d3b 172.16.32.116:8002 slots: (0 slots) slave replicates 88fe075375295b59eabe69fa1438ed7c7c314f43 S: aeb684429d220c0fd1392574d193cc1ae7577782 172.16.32.116:8000 slots: (0 slots) slave replicates a0b91f48e933c1f1d427c54917ce970bd25d29f8 M: 88fe075375295b59eabe69fa1438ed7c7c314f43 172.16.32.116:7002 slots:11022-16383 (5362 slots) master 1 additional replica(s) S: a96cad95dca2a8e1e0302bff4f835260d92e3d31 172.16.32.116:8001 slots: (0 slots) slave replicates 273107e5ac994d675749be0979556e761274bb93 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. How many slots do you want to move (from 1 to 16384)? 300 <<<9000节点全部需要迁移的节点 What is the receiving node ID? a0b91f48e933c1f1d427c54917ce970bd25d29f8 <<<<<<7000的ID Please enter all the source node IDs. Type 'all' to use all the nodes as source nodes for the hash slots. Type 'done' once you entered all the source nodes IDs. Source node #1:364ae8322ab2627e25b05d45b702448c74afad10 <<<<<<9000的ID Source node #2:done Ready to move 300 slots. Source nodes: M: 364ae8322ab2627e25b05d45b702448c74afad10 172.16.32.116:9000 <<<<<