Linux环境下配置搭建Redis分布式集群

久前, 玩过redis集群服务的搭建, 那种哨兵模式有点类似数据库的读写分离, 但是业务写入数据还是全部由一个逻辑上的master承担的, 具体方法可见 [哨兵]; 今天来依据Redis的官方说明部署把玩下分布式集群, 这种模式可以配置多个master均分16384个hash slot, 使得不同hash key分落到不同的master节点上, 真正意义的分布式;

本文的部署环境是 ubuntu16.04, redis-3.2.8

部署方案及环境

本示例中采用6节点, 三个master和三个slaves两两成组的方式共同形成一个分布式集群, 对外提供存储服务;

获取redis

下载 redis, 得到应该是源码, 先进行解压, 然后make即可;

redis 配置介绍

# redis 提供服务时绑定的本机网络地址，可以是多个满足多块网卡分别对外网内网的访问, 按需要改
bind 127.0.0.1
# 提供服务的端口, 供客户端连接
port 7000
# 启动redis服务时,进程pid存储位置
pidfile /disk/develop/redis/cluster/7000/redis.pid
# redis 日志文件所在
logfile "/disk/develop/redis/cluster/7000/log.txt"
# 持久化文件存储目录
dir /disk/develop/redis/cluster/7000
# 开启集群模式
cluster-enabled yes
# 集群模式下的节点配置信息
cluster-config-file nodes.conf
# 集群中各节点间连接超时时间
cluster-node-timeout 5000
# 允许数据持久化追加
appendonly yes

创建集群目录

比如上面我把下载的redis源码解压到了 ‘/disk/develop/redis/cluster/redis-3.2.8’, 同时创建其他六个节点目录

cd /disk/develop/redis/cluster/
mkdir 7000 7001 7002 7003 7004 7005 

开始配置

copy 原始的配置文件

cp redis-3.2.8/redis.conf 7000/
cp redis-3.2.8/redis.conf 7001/
cp redis-3.2.8/redis.conf 7002/
cp redis-3.2.8/redis.conf 7003/
cp redis-3.2.8/redis.conf 7004/
cp redis-3.2.8/redis.conf 7005/

修改配置文件, 分别进入到上面的六个新建的文件夹中, 按照上面的配置介绍中的样例进行对应修改

启动服务

cd /disk/develop/redis/cluster/redis-3.2.8
src/redis-server ../7000/redis.conf
src/redis-server ../7001/redis.conf
src/redis-server ../7002/redis.conf
src/redis-server ../7003/redis.conf
src/redis-server ../7004/redis.conf
src/redis-server ../7005/redis.conf

查看下服务呢

ps -ef|grep redis-server
hinsteny 21971  2242  0 22:48 ?        00:00:02 src/redis-server 127.0.0.1:7000 [cluster]
hinsteny 22062  2242  0 22:50 ?        00:00:02 src/redis-server 127.0.0.1:7001 [cluster]
hinsteny 22149  2242  0 22:53 ?        00:00:02 src/redis-server 127.0.0.1:7002 [cluster]
hinsteny 22226  2242  0 22:55 ?        00:00:02 src/redis-server 127.0.0.1:7003 [cluster]
hinsteny 22257  2242  0 22:57 ?        00:00:02 src/redis-server 127.0.0.1:7004 [cluster]
hinsteny 22280  2242  0 22:58 ?        00:00:02 src/redis-server 127.0.0.1:7005 [cluster]
hinsteny 23831 11259  0 23:35 pts/10   00:00:00 grep --color=auto redis-server

虽然六个服务都起来了, 但这时他们之间还不知道彼此的存在, 无法组成集群提供服务

src/redis-cli -c -p 7000
127.0.0.1:7000> cluster info
cluster_state:fail   //集群状态失败
cluster_slots_assigned:0 // hash 槽点分布也是失败的
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
127.0.0.1:7000> 

接下来让我们用一个工具 ‘redis-trib.rb’ 将上面的六个节点连接起来

插播, redis-trib是需要ruby环境并在其环境下安装对应的redis插件

# 先安装ruby环境
apt-get install ruby
# 再安装redis需要的插件
gem install redis

关联组建集群

src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 >>> Creating cluster >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 Adding replica 127.0.0.1:7003 to 127.0.0.1:7000 Adding replica 127.0.0.1:7004 to 127.0.0.1:7001 Adding replica 127.0.0.1:7005 to 127.0.0.1:7002 M: dc5da8bfce379f9cc2c11966709d7d9f275d06b1 127.0.0.1:7000 slots:0-5460 (5461 slots) master M: 2b22012f983698f504b9e61d0958fc645112c805 127.0.0.1:7001 slots:5461-10922 (5462 slots) master M: df396e45f08019525bf668ff4704d626014db2d6 127.0.0.1:7002 slots:10923-16383 (5461 slots) master S: 90740536f6e5b191300253b84fb4d9b908d48d54 127.0.0.1:7003 replicates dc5da8bfce379f9cc2c11966709d7d9f275d06b1 S: 52ba80c45ffd165b9cec01c8409c605386d1a8b2 127.0.0.1:7004 replicates 2b22012f983698f504b9e61d0958fc645112c805 S: 888848820f300331a76803755dc1c15302f2ce98 127.0.0.1:7005 replicates df396e45f08019525bf668ff4704d626014db2d6 Can I set the above configuration? (type 'yes' to accept): yes >>> Nodes configuration updated >>> Assign a different config epoch to each node >>> Sending CLUSTER MEET messages to join the cluster Waiting for the cluster to join... >>> Performing Cluster Check (using node 127.0.0.1:7000) M: dc5da8bfce379f9cc2c11966709d7d9f275d06b1 127.0.0.1:7000 slots:0-5460 (5461 slots) master 1 additional replica(s) M: 2b22012f983698f504b9e61d0958fc645112c805 127.0.0.1:7001 slots:5461-10922 (5462 slots) master 1 additional replica(s) S: 90740536f6e5b191300253b84fb4d9b908d48d54 127.0.0.1:7003 slots: (0 slots) slave replicates dc5da8bfce379f9cc2c11966709d7d9f275d06b1 S: 52ba80c45ffd165b9cec01c8409c605386d1a8b2 127.0.0.1:7004 slots: (0 slots) slave replicates 2b22012f983698f504b9e61d0958fc645112c805 M: df396e45f08019525bf668ff4704d626014db2d6 127.0.0.1:7002 slots:10923-16383 (5461 slots) master 1 additional replica(s) S: 888848820f300331a76803755dc1c15302f2ce98 127.0.0.1:7005 slots: (0 slots) slave replicates df396e45f08019525bf668ff4704d626014db2d6 [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. 

验证集群配置

上面显示已经成功组成集群了, 节点[7000,7001,7002]为三个master, 剩下的[7003,7004,7005]分别为前三者的复制节点, 并且可以看到16384个槽点被全部覆盖分布, 现在再查看下集群状态

src/redis-cli -c -p 7000127.0.0.1:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_sent:487
cluster_stats_messages_received:487

最后通过set与get操作验证下

src/redis-cli -c -p 7000
127.0.0.1:7000> set hisoka welcome
-> Redirected to slot [9822] located at 127.0.0.1:7001
OK
127.0.0.1:7001> get hisoka
"welcome"
127.0.0.1:7001> set hinsteny 'love sing'
OK
127.0.0.1:7001> get hinsteny
"love sing"
127.0.0.1:7001> set k3 k3_val
-> Redirected to slot [4576] located at 127.0.0.1:7000
OK
127.0.0.1:7000> set k5 k5_val
-> Redirected to slot [12582] located at 127.0.0.1:7002
OK
127.0.0.1:7002> 

That’s all, Thanks!