Redis主从自动failover

 Redis主从架构持久化存在一个问题，即前次测试的结论，持久化需要配置在主实例上才能跨越实例保证数据不丢失，这样以来主实例在持久化数据到硬盘的过程中，势必会造成磁盘的I/O等待，经过实际测试，这个持久化写硬盘的过程给应用程序带来的影响无法忍受；因而在大多数场景下，会考虑把持久化配置在从实例上，当主实例宕机后，通过手动或者自动的方式将从实例提升为主实例，继续提供服务！当主实例恢复后，先从原从实例上同步数据，同步完成后再恢复到原始的主从状态！要实现这种的要求，需要有keepalive的配合，一方面keepalive提供了VIP，可以避免修改应用程序连接，同时redis实例的配置文件监听部分也需要修改为全网监听；另一方面keepalive定时调度脚本来监控主从实例的状态，根据具体情况进行切换！本文将重点介绍下使用keepalive实现redis主从自动failover！

环境介绍
操作系统版本均为：Centos6.0 64bit
redis版本：2.6.4
redis实例端口均为：6379
VIP：172.16.1.200
主实例为Master-redis(172.16.1.170)
从实例为Slave-redis(172.16.1.171，开启快照持久化)

一：安装keepalive软件,server11安装完成后直接scp至server12上即可

[root@Master-redis ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz

[root@Master-redis ~]# tar zxf keepalived-1.1.19.tar.gz 

[root@Master-redis ~]# cd keepalived-1.1.19/

[root@Master-redis keepalived-1.1.19]# ./configure  --prefix=/usr/local/keepalived/ && make && make install

二：配置主节点Master-redis配置文件

[root@Master-redis ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf

! Configuration File for keepalived  

 

global_defs {  

 router_id LVS_DEVEL 

}  

 

vrrp_script Monitor_redis {  

 script "/usr/local/scripts/redis_monitor.sh"  

 interval 2   

 weight 2    

}  

 

vrrp_instance VI_1 {   

 state MASTER  

 interface eth0  

 virtual_router_id 20

 mcast_src_ip 172.16.1.170

 priority  100  

 advert_int 1  

 authentication {  

 auth_type PASS  

 auth_pass 1111

}  

 track_script {  

 Monitor_redis  

}  

 virtual_ipaddress {  

 172.16.1.200

 }  

 notify_fault  /usr/local/scripts/redis_fault.sh    

 notify_stop   /usr/local/scripts/redis_stop.sh

三：配置从节点Slave-redis配置文件

四：准备相关的脚本，主从实例上都需要存在这些脚本，同时注意脚本需要由可执行权限

[root@Master-redis ~]# cat /usr/local/scripts/redis_monitor.sh 

#!/bin/bash

ALIVE=`/usr/local/bin/redis-cli PING`

if [ "$ALIVE" == "PONG" ] ; then

 echo $ALIVE

 exit 0

else

echo $ALIVE

killall -9 keepalived

service network restart

exit 1

Fi

[root@Master-redis ~]# sh /usr/local/scripts/redis_monitor.sh 

PONG

[root@Slave-redis ~]# cat /usr/local/scripts/redis_master.sh 

#!/bin/bash

REDISCLI="/usr/local/bin/redis-cli -h 172.16.1.171"

LOGFILE="/usr/local/src/logs/keepalived-redis-state.log"

echo "[master]" >>$LOGFILE

date >>$LOGFILE

echo "Being master ...." >>$LOGFILE 2>&1

echo "Run SLAVEOF cmd...." >>$LOGFILE

echo "Run SLAVEOF NO ONE cmd.." >>$LOGFILE

$REDISCLI SLAVEOF NO ONE >>$LOGFILE 2>&1

[root@Slave-redis ~]# cat /usr/local/scripts/redis_backup.sh 

#!/bin/bash

REDISCLI="/usr/local/bin/redis-cli -h 172.16.1.171"

LOGFILE="/usr/local/src/logs/keepalived-redis-state.log"

echo "[backup]" >>$LOGFILE

date >>$LOGFILE

echo "Being slave...." >>$LOGFILE 2>&1

sleep 15

echo "Run SLAVEOF cmd..." >>$LOGFILE

$REDISCLI SLAVEOF 172.16.1.170 6379 >>$LOGFILE 2>&1

[root@Slave-redis ~]# cat /usr/local/scripts/redis_stop.sh 

#!/bin/bash

LOGFILE="/usr/local/src/logs/keepalived-redis-state.log"

echo  "[stop]" >>$LOGFILE

date >>$LOGFILE

[root@Slave-redis ~]# cat /usr/local/scripts/redis_fault.sh 

#!/bin/bash

LOGFILE="/usr/local/src/logs/keepalived-redis-state.log"

echo "[fault]" >> $LOGFILE  

date >> $LOGFILE

五：主从实例分别启动keepalive进程，测试VIP是否正常

[root@Master-redis ~]# /usr/local/keepalived/sbin/keepalived  -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf

[root@Master-redis ~]# tail -f /var/log/messages

Jan 17 15:59:58 Master-redis Keepalived_healthcheckers: Configuration is using : 5707 Bytes

Jan 17 15:59:58 Master-redis Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...

Jan 17 15:59:58 Master-redis Keepalived_vrrp: VRRP_Script(Monitor_redis) succeeded

Jan 17 15:59:58 Master-redis Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE

Jan 17 15:59:59 Master-redis Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE

Jan 17 15:59:59 Master-redis Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.

Jan 17 15:59:59 Master-redis Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.16.1.200

Jan 17 15:59:59 Master-redis Keepalived_healthcheckers: Netlink reflector reports IP 172.16.1.200 added

Jan 17 15:59:59 Master-redis avahi-daemon[1240]: Registering new address record for 172.16.1.200 on eth0.IPv4.

Jan 17 16:00:02 Master-redis Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 172.16.1.200

[root@Master-redis ~]# ip a | grep 172

    inet 172.16.1.170/22 brd 172.16.3.255 scope global eth0

    inet 172.16.1.200/32 scope global eth0

[root@Slave-redis ~]# /usr/local/keepalived/sbin/keepalived  -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf

[root@Slave-redis ~]# tail -f /var/log/messages 

Jan 17 16:16:31 Slave-redis Keepalived: Starting VRRP child process, pid=11377

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Registering Kernel netlink reflector

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Registering Kernel netlink command channel

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Registering gratutious ARP shared channel

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Opening file '/usr/local/keepalived/etc/keepalived/keepalived.conf'.

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Configuration is using : 64882 Bytes

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Using LinkWatch kernel netlink reflector...

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) Entering BACKUP STATE

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: VRRP_Script(Monitor_redis) succeeded

[root@Master-redis ~]# redis-cli  -h 172.16.1.200 info | grep -A 3 'Replication'

# Replication

role:master

connected_slaves:1

slave0:172.16.1.171,6379,online

六：主实例写入测试数据

[root@Master-redis ~]# for i in {1..10000}

> do

> redis-cli -h 172.16.1.200 set $i abc

> done

 [root@Master-redis ~]# redis-cli  -h 172.16.1.200  info 

# Keyspace

db0:keys=10673,expires=0

[root@Master-redis data]# ls -lh 

总用量 1.4M

-rw-r--r-- 1 root root 883K 1月  17 14:33 appendonly.aof

-rw-r--r-- 1 root root 472K 1月  17 15:58 dump.rdb

[root@Slave-redis data]# ls -lh 

total 1.6M

-rw-r--r-- 1 root root 883K Jan 17 15:59 appendonly.aof

-rw-r--r-- 1 root root 472K Jan 17 15:59 dump.rdb

七：模拟主实例故障，观察日志输出，验证从实例是否能成功接管VIP，同时将实例变成读写模式

[root@Master-redis ~]# killall redis-server

[root@Master-redis ~]# ip a | grep 172

    inet 172.16.1.170/22 brd 172.16.3.255 scope global eth0

[root@Master-redis ~]# ps -ef | grep redis-server

root     32461 32379  0 16:23 pts/1    00:00:00 grep --color=auto redis-server

[root@Master-redis ~]# ps -ef | grep keepalived

root     32500 32379  0 16:24 pts/1    00:00:00 grep --color=auto keepalived

[root@Slave-redis data]# tail -f /usr/local/src/logs/keepalived-redis-state.log 

OK

Run SLAVEOF NO ONE cmd..

OK

[master]

Thu Jan 17 16:22:25 CST 2013

Being master ....

Run SLAVEOF cmd....

Could not connect to Redis at 172.16.1.170:6379: Connection refused

Run SLAVEOF NO ONE cmd..

Could not connect to Redis at 172.16.1.170:6379: Connection refused

[root@Slave-redis data]# tail -f /var/log/messages 

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: Using LinkWatch kernel netlink reflector...

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) Entering BACKUP STATE

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(9,10)]

Jan 17 16:16:31 Slave-redis Keepalived_vrrp: VRRP_Script(Monitor_redis) succeeded

Jan 17 16:22:24 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) Transition to MASTER STATE

Jan 17 16:22:25 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) Entering MASTER STATE

Jan 17 16:22:25 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) setting protocol VIPs.

Jan 17 16:22:25 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 172.16.1.200

Jan 17 16:22:25 Slave-redis avahi-daemon[1259]: Registering new address record for 172.16.1.200 on eth0.IPv4.

Jan 17 16:22:30 Slave-redis Keepalived_vrrp: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 172.16.1.200

[root@Slave-redis data]# ip a | grep 172

    inet 172.16.1.171/22 brd 172.16.3.255 scope global eth0

    inet 172.16.1.200/32 scope global eth0

[root@Slave-redis data]# redis-cli  -h 172.16.1.200 info | grep -A 3 'Replication'

# Replication

role:slave

master_host:172.16.1.170

master_port:6379

九:主实例角色的恢复过程，使用shell脚本自动恢复

[root@Slave-redis ~]# cat /usr/local/scripts/remover_master.sh 

#!/bin/sh  

ALIVE=`/usr/local/bin/redis-cli -h 171.16.1.170 -p 6379 PING`

MDB=/usr/local/src/data/dump.rdb  

SDB=/usr/local/src/data/

 

if [ "$ALIVE" == "PONG" ]; then   

    echo $ALIVE    

    scp $MDB [email protected]:$SDB  

    else   

    echo $ALIVE   

    exit 1   

fi   

 

redis-server /etc/redis.conf  

/usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf