redis集群的部署

注意事项：

1.如果是使用redis-trib.rb工具构建集群，集群构建完成前不要配置密码，集群构建完毕再通过config set + config rewrite命令逐个机器设置密码

2.如果对集群设置密码，那么requirepass和masterauth都需要设置，否则发生主从切换时，就会遇到授权问题，可以模拟并观察日志

3.各个节点的密码都必须一致，否则Redirected就会失败

config set masterauth 123456  
config set requirepass 123456
config rewrite

1. 要求6台服务器

  三主三从

2.在三台服务器上分别启动redis

 分别在每台服务器上部署redis，并启动起来

启动之前需要和redis单机安装不同的的配置文件参数不一样

vi /opt/redids/conf/配置文件

  database 1  #默认有16个库选择一个

 cluster-enabled yes  #开启集群

cluster-config-file #/opt/redis/conf/nodes.conf   #开启集群的时候自动生成和redis的配置文件没有任何关系

appendonly  yes #开启持久化存储

3.启动所有的redis服务，启动以后只能说明启动但是6台服务器还未构成集群#参考单机部署

4.集群工作

  1.yum  -y install ruby  rubygems

   gem  install redis

  cp  /root/redis-3.2.2/src/redis-trib.rb  /opt/redis/bin/  #redis bin  目录已经在环境变量了

  redis-trib.rb  create  --replicas 1  IP1:6379 IP2:6379 IP3:6379 IP4:6379 IP5:6379 IP6 :6379 

 --replicas 1  是指一个主节点有个备节点 就是说三主三被

结束以后会发现/opt/redis/conf/nodes.conf 里面已经生成了集群的信息

5.查看集群的状态

 redis-cli  -h IP cluster  nodes

6.设置key  value 手动验证

具体实验步骤

说明：视频上用的是
192.168.1.0/24   我自己用的是 192.168.56.0/24 网段

redis 集群的通讯的端口默认是服务端口加上10000

1.所有机器 停止selinux 和 iptables （iptables 有的话端口允许）（all）
yum  -y install  gcc  ctl

2.每台服务器都上传reids(all)

[root@localhost src]# pwd
/usr/local/src
[root@localhost src]# ls
redis-3.0.6.tar.gz
[root@localhost src]#
3.创建目录(all)

mkdir  /usr/local/redis3  -p

4.安装

cd /usr/local/src/

tar xf   redis-3.0.6.tar.gz

cd  redis-3.0.6
make  PREFIX=/usr/local/redis3  install

5. 初始化操作

 mkdir  -p /usr/local/redis3/cluster/7111    #192.168.56.111

 mkdir  -p /usr/local/redis3/cluster/7112   #192.168.56.112

 mkdir  -p /usr/local/redis3/cluster/7113  #192.168.56.113

 mkdir  -p /usr/local/redis3/cluster/7114   #192.168.56.114

 mkdir  -p /usr/local/redis3/cluster/7115  #192.168.56.115

 mkdir  -p /usr/local/redis3/cluster/7116   #192.168.56.116

 scp redis.conf 192.168.56.111:/usr/local/redis3/cluster/7111/redis-7111.conf

 scp redis.conf 192.168.56.112:/usr/local/redis3/cluster/7112/redis-7112.conf
  
 scp redis.conf 192.168.56.113:/usr/local/redis3/cluster/7113/redis-7113.conf

 scp redis.conf 192.168.56.114:/usr/local/redis3/cluster/7114/redis-7114.conf
  
 scp redis.conf 192.168.56.115:/usr/local/redis3/cluster/7115/redis-7115.conf
  
 scp redis.conf 192.168.56.116:/usr/local/redis3/cluster/7116/redis-7116.conf

6.修改redis配置文件

vim  /usr/local/redis3/cluster/7111/redis-7111.conf

daemonize yes
pidfile /var/run/redis-7111.pid  #此处修改
port 7111
tcp-backlog 511
timeout 0
tcp-keepalive 0
loglevel notice
logfile ""
databases  1
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir ./
slave-serve-stale-data yes
slave-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
slave-priority 100
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
lua-time-limit 5000
cluster-enabled yes
cluster-config-file  /usr/local/redis3/cluster/7111/nodes.conf  #此处修改
cluster-node-timeout 15000
cluster-migration-barrier 1
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

sed -i 's#7111#7116#g' redis-7116.conf   其余的配置文件拷贝过去 替换

7.启动所有的redis

/usr/local/redis3/bin/redis-server   /usr/local/redis3/cluster/7111/redis-7111.conf

/usr/local/redis3/bin/redis-server   /usr/local/redis3/cluster/7112/redis-7112.conf

/usr/local/redis3/bin/redis-server   /usr/local/redis3/cluster/7113/redis-7113.conf

/usr/local/redis3/bin/redis-server   /usr/local/redis3/cluster/7114/redis-7114.conf

/usr/local/redis3/bin/redis-server   /usr/local/redis3/cluster/7115/redis-7115.conf

/usr/local/redis3/bin/redis-server   /usr/local/redis3/cluster/7116/redis-7116.conf

检查所有服务的端口都已经启动

[root@localhost 7111]# netstat  -lntup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1104/sshd           
tcp        0      0 0.0.0.0:17111           0.0.0.0:*               LISTEN      5379/redis-server *
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1603/master         
tcp        0      0 0.0.0.0:7111            0.0.0.0:*               LISTEN      5379/redis-server *
tcp6       0      0 :::22                   :::*                    LISTEN      1104/sshd           
tcp6       0      0 :::17111                :::*                    LISTEN      5379/redis-server *
tcp6       0      0 ::1:25                  :::*                    LISTEN      1603/master         
tcp6       0      0 :::7111                 :::*                    LISTEN      5379/redis-server *
[root@localhost 7111]#
[root@localhost 7111]# ps -ef |grep redis
root       5379      1  0 08:13 ?        00:00:00 /usr/local/redis3/bin/redis-server *:7111 [cluster]
root       5410   5275  0 08:19 pts/0    00:00:00 grep --color=auto redis
[root@localhost 7111]#

8.创建集群

yum  -y install ruby  rubygems   (all)
安装redis ruby的接口

gem  install redis  (all)

cd  /usr/local/src/redis-3.0.6/src
cp  redis-trib.rb  /usr/local/bin/redis-trib

如果是如下创建的话可以看主 111，112，113 为master    114,115,116为slave，不符合要求

--replicas 1 表示一个master有一个slave
 (创建命令在任意一台redis执行)

[root@localhost src]# redis-trib create     --replicas 1  192.168.56.111:7111  192.168.56.112:7112  192.168.56.113:7113  192.168.56.114:7114  192.168.56.115:7115 192.168.56.116:7116
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.56.111:7111
192.168.56.112:7112
192.168.56.113:7113
Adding replica 192.168.56.114:7114 to 192.168.56.111:7111
Adding replica 192.168.56.115:7115 to 192.168.56.112:7112
Adding replica 192.168.56.116:7116 to 192.168.56.113:7113
M: 8c1890f102a035a05836dd3a5621499d69ad97eb 192.168.56.111:7111
   slots:0-5460 (5461 slots) master
M: f9455c930ac9a932ed9b4bcb2cb6d929ab912941 192.168.56.112:7112
   slots:5461-10922 (5462 slots) master
M: 289acc8a4c879f9cd9ca40f29825dd67a64a71c8 192.168.56.113:7113
   slots:10923-16383 (5461 slots) master
S: 28b18432a99ffc67f31e7b94f63199acae8bcc24 192.168.56.114:7114
   replicates 8c1890f102a035a05836dd3a5621499d69ad97eb
S: 4fd1072ae882c2a9fc332c968a5888fa38b8435e 192.168.56.115:7115
   replicates f9455c930ac9a932ed9b4bcb2cb6d929ab912941
S: d6f6fd5082874ee0ebe9a8bdc2575a1dcff23095 192.168.56.116:7116
   replicates 289acc8a4c879f9cd9ca40f29825dd67a64a71c8
Can I set the above configuration? (type 'yes' to accept):


如果不符合要求的话
调整IP的顺序，创建完毕

 (创建命令在任意一台redis执行)

[root@localhost src]# redis-trib create     --replicas 1  192.168.56.111:7111  192.168.56.112:7112  192.168.56.113:7113  192.168.56.114:7114  192.168.56.115:7115 192.168.56.116:7116
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.56.111:7111
192.168.56.112:7112
192.168.56.113:7113
Adding replica 192.168.56.114:7114 to 192.168.56.111:7111
Adding replica 192.168.56.115:7115 to 192.168.56.112:7112
Adding replica 192.168.56.116:7116 to 192.168.56.113:7113
M: 8c1890f102a035a05836dd3a5621499d69ad97eb 192.168.56.111:7111
   slots:0-5460 (5461 slots) master
M: f9455c930ac9a932ed9b4bcb2cb6d929ab912941 192.168.56.112:7112
   slots:5461-10922 (5462 slots) master
M: 289acc8a4c879f9cd9ca40f29825dd67a64a71c8 192.168.56.113:7113
   slots:10923-16383 (5461 slots) master
S: 28b18432a99ffc67f31e7b94f63199acae8bcc24 192.168.56.114:7114
   replicates 8c1890f102a035a05836dd3a5621499d69ad97eb
S: 4fd1072ae882c2a9fc332c968a5888fa38b8435e 192.168.56.115:7115
   replicates f9455c930ac9a932ed9b4bcb2cb6d929ab912941
S: d6f6fd5082874ee0ebe9a8bdc2575a1dcff23095 192.168.56.116:7116
   replicates 289acc8a4c879f9cd9ca40f29825dd67a64a71c8
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 192.168.56.111:7111)
M: 8c1890f102a035a05836dd3a5621499d69ad97eb 192.168.56.111:7111
   slots:0-5460 (5461 slots) master
M: f9455c930ac9a932ed9b4bcb2cb6d929ab912941 192.168.56.112:7112
   slots:5461-10922 (5462 slots) master
M: 289acc8a4c879f9cd9ca40f29825dd67a64a71c8 192.168.56.113:7113
   slots:10923-16383 (5461 slots) master
M: 28b18432a99ffc67f31e7b94f63199acae8bcc24 192.168.56.114:7114
   slots: (0 slots) master
   replicates 8c1890f102a035a05836dd3a5621499d69ad97eb
M: 4fd1072ae882c2a9fc332c968a5888fa38b8435e 192.168.56.115:7115
   slots: (0 slots) master
   replicates f9455c930ac9a932ed9b4bcb2cb6d929ab912941
M: d6f6fd5082874ee0ebe9a8bdc2575a1dcff23095 192.168.56.116:7116
   slots: (0 slots) master
   replicates 289acc8a4c879f9cd9ca40f29825dd67a64a71c8
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@localhost src]#

生辰文件的节点信息（all）

[root@localhost 7111]# ll
total 52
-rw-r--r--. 1 root root     0 Jun  3 08:13 appendonly.aof
-rw-r--r--. 1 root root    18 Jun  3 08:39 dump.rdb
-rw-r--r--. 1 root root   763 Jun  3 08:39 nodes.conf
-rw-r--r--. 1 root root 41584 Jun  3 08:03 redis-7111.conf
[root@localhost 7111]# pwd
/usr/local/redis3/cluster/7111
[root@localhost 7111]# cat  nodes.conf
4fd1072ae882c2a9fc332c968a5888fa38b8435e 192.168.56.115:7115 slave f9455c930ac9a932ed9b4bcb2cb6d929ab912941 0 1496493590416 5 connected
f9455c930ac9a932ed9b4bcb2cb6d929ab912941 192.168.56.112:7112 master - 0 1496493588394 2 connected 5461-10922
d6f6fd5082874ee0ebe9a8bdc2575a1dcff23095 192.168.56.116:7116 slave 289acc8a4c879f9cd9ca40f29825dd67a64a71c8 0 1496493586373 3 connected
28b18432a99ffc67f31e7b94f63199acae8bcc24 192.168.56.114:7114 slave 8c1890f102a035a05836dd3a5621499d69ad97eb 0 1496493589405 4 connected
289acc8a4c879f9cd9ca40f29825dd67a64a71c8 192.168.56.113:7113 master - 0 1496493587385 3 connected 10923-16383
8c1890f102a035a05836dd3a5621499d69ad97eb 192.168.56.111:7111 myself,master - 0 0 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
[root@localhost 7111]#

9验证redis集群（可以看出 redis集群生效 slot为第三个master的）

[root@localhost 7116]# cd /usr/local/redis3/
[root@localhost redis3]# ls
bin  cluster
[root@localhost redis3]# ./bin/redis-cli   -c  -p 7116
127.0.0.1:7116> set  liuhx  sanlang'
Invalid argument(s)
127.0.0.1:7116> set  liuhx  sanlang
-> Redirected to slot [14006] located at 192.168.56.113:7113
OK
192.168.56.113:7113>

[root@localhost redis3]# ./bin/redis-cli  -c  -p  7111
127.0.0.1:7111> get  liuhx
-> Redirected to slot [14006] located at 192.168.56.113:7113
"sanlang"
192.168.56.113:7113>

10.查看节点的状态信息

[root@localhost bin]# ./redis-cli   -p 7114  cluster  nodes
289acc8a4c879f9cd9ca40f29825dd67a64a71c8 192.168.56.113:7113 master - 0 1496494716960 3 connected 10923-16383
8c1890f102a035a05836dd3a5621499d69ad97eb 192.168.56.111:7111 master - 0 1496494716455 1 connected 0-5460
4fd1072ae882c2a9fc332c968a5888fa38b8435e 192.168.56.115:7115 slave f9455c930ac9a932ed9b4bcb2cb6d929ab912941 0 1496494719985 5 connected
f9455c930ac9a932ed9b4bcb2cb6d929ab912941 192.168.56.112:7112 master - 0 1496494718977 2 connected 5461-10922
28b18432a99ffc67f31e7b94f63199acae8bcc24 192.168.56.114:7114 myself,slave 8c1890f102a035a05836dd3a5621499d69ad97eb 0 0 4 connected
d6f6fd5082874ee0ebe9a8bdc2575a1dcff23095 192.168.56.116:7116 slave 289acc8a4c879f9cd9ca40f29825dd67a64a71c8 0 1496494717968 6 connected
[root@localhost bin]# pwd
/usr/local/redis3/bin
[root@localhost bin]#