通过Keepalived实现Redis Failover自动故障切换功能(整理中) .
通过Keepalived实现Redis Failover自动故障切换功能
详细可参考http://heylinux.com/archives/1942.html
http://blog.csdn.net/qingrx/article/details/9146779
设计思路：
redis 主和 redis从 正常同步；
redis主keepalived 启动，执行redis_master.sh脚本，vip在自己身上；
redis 从keepalived 启动，执行redis_slave.sh脚本；
redis主服务关闭，则redis主执行redis_fault.sh脚本， vip落在redis 从上，redis从执行redis_master.sh脚本，变成主；
redis主服务开启，则redis主执行redis_slave.sh脚本变成从，vip还在redis 从上，还是主；
redis从服务关闭，则redis从执行redis_fault.sh脚本， vip落在redis主上，redis主执行redis_master.sh脚本，变成主；
redis从服务开启，则redis从执行redis_slave.sh脚本变成从， vip还在redis主上，redis主继续做主，此时为redis初试主从状态；
环境：
10.7.13.43 redis
10.7.13.44 redis-slave
10.7.13.45 vip
一:安装keepalived
(redis和redis-slave两台服务器都安装)
1. 安装操作系统的 ipvsadm
yum -y install ipvsadm
否则出现如下错误：
如果这个不安装 make 会有如下错误
make
make -C lib || exit 1;
make[1]: Entering directory `/httx/download/keepalived-1.2.10/lib'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/httx/download/keepalived-1.2.10/lib'
make -C keepalived
make[1]: Entering directory `/httx/download/keepalived-1.2.10/keepalived'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/core'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/core'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/check'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/check'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/vrrp'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/vrrp'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/libipvs-2.6'
gcc -g -O2 -I/usr/src/linux/include -I/usr/src/linux/include -DLIBIPVS_DONTUSE_NL -Wall -Wunused -c -o libipvs.o libipvs.c
libipvs.c: In function ‘nlerr2syserr’:
libipvs.c:61: error: ‘NLE_BAD_SOCK’ undeclared (first use in this function)
libipvs.c:61: error: (Each undeclared identifier is reported only once
libipvs.c:61: error: for each function it appears in.)
libipvs.c:62: error: ‘NLE_EXIST’ undeclared (first use in this function)
libipvs.c:63: error: ‘NLE_NOADDR’ undeclared (first use in this function)
libipvs.c:64: error: ‘NLE_OBJ_NOTFOUND’ undeclared (first use in this function)
libipvs.c:65: error: ‘NLE_INTR’ undeclared (first use in this function)
libipvs.c:66: error: ‘NLE_AGAIN’ undeclared (first use in this function)
libipvs.c:67: error: ‘NLE_INVAL’ undeclared (first use in this function)
libipvs.c:68: error: ‘NLE_NOACCESS’ undeclared (first use in this function)
libipvs.c:69: error: ‘NLE_NOMEM’ undeclared (first use in this function)
libipvs.c:70: error: ‘NLE_AF_NOSUPPORT’ undeclared (first use in this function)
libipvs.c:71: error: ‘NLE_PROTO_MISMATCH’ undeclared (first use in this function)
libipvs.c:72: error: ‘NLE_OPNOTSUPP’ undeclared (first use in this function)
libipvs.c:73: error: ‘NLE_PERM’ undeclared (first use in this function)
libipvs.c:74: error: ‘NLE_BUSY’ undeclared (first use in this function)
libipvs.c:75: error: ‘NLE_RANGE’ undeclared (first use in this function)
libipvs.c:76: error: ‘NLE_NODEV’ undeclared (first use in this function)
make[2]: *** [libipvs.o] Error 1
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/libipvs-2.6'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived'
make: *** [all] Error 2
1.wget http://www.keepalived.org/software/keepalived-1.2.10.tar.gz
2.安装操作系统组件 openssl-devel
yum -y install openssl-devel
否则出现如下错误：
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files.
2.安装 tar zxvf keepalived-1.2.10.tar.gz
cd keepalived-1.2.10
3. ./configure --prefix=/httx/run/keepalived
4. yum -y install ipvsadm
如果这个不安装 make 会有如下错误
make
make -C lib || exit 1;
make[1]: Entering directory `/httx/download/keepalived-1.2.10/lib'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/httx/download/keepalived-1.2.10/lib'
make -C keepalived
make[1]: Entering directory `/httx/download/keepalived-1.2.10/keepalived'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/core'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/core'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/check'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/check'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/vrrp'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/vrrp'
make[2]: Entering directory `/httx/download/keepalived-1.2.10/keepalived/libipvs-2.6'
gcc -g -O2 -I/usr/src/linux/include -I/usr/src/linux/include -DLIBIPVS_DONTUSE_NL -Wall -Wunused -c -o libipvs.o libipvs.c
libipvs.c: In function ‘nlerr2syserr’:
libipvs.c:61: error: ‘NLE_BAD_SOCK’ undeclared (first use in this function)
libipvs.c:61: error: (Each undeclared identifier is reported only once
libipvs.c:61: error: for each function it appears in.)
libipvs.c:62: error: ‘NLE_EXIST’ undeclared (first use in this function)
libipvs.c:63: error: ‘NLE_NOADDR’ undeclared (first use in this function)
libipvs.c:64: error: ‘NLE_OBJ_NOTFOUND’ undeclared (first use in this function)
libipvs.c:65: error: ‘NLE_INTR’ undeclared (first use in this function)
libipvs.c:66: error: ‘NLE_AGAIN’ undeclared (first use in this function)
libipvs.c:67: error: ‘NLE_INVAL’ undeclared (first use in this function)
libipvs.c:68: error: ‘NLE_NOACCESS’ undeclared (first use in this function)
libipvs.c:69: error: ‘NLE_NOMEM’ undeclared (first use in this function)
libipvs.c:70: error: ‘NLE_AF_NOSUPPORT’ undeclared (first use in this function)
libipvs.c:71: error: ‘NLE_PROTO_MISMATCH’ undeclared (first use in this function)
libipvs.c:72: error: ‘NLE_OPNOTSUPP’ undeclared (first use in this function)
libipvs.c:73: error: ‘NLE_PERM’ undeclared (first use in this function)
libipvs.c:74: error: ‘NLE_BUSY’ undeclared (first use in this function)
libipvs.c:75: error: ‘NLE_RANGE’ undeclared (first use in this function)
libipvs.c:76: error: ‘NLE_NODEV’ undeclared (first use in this function)
make[2]: *** [libipvs.o] Error 1
make[2]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived/libipvs-2.6'
make[1]: *** [all] Error 1
make[1]: Leaving directory `/httx/download/keepalived-1.2.10/keepalived'
make: *** [all] Error 2
5. make && make install
6. 日志文件 /httx/logs/keepalived/keepalived-redis-state.log
7. chmod +x *.sh
8. 复制配置文件
keepalived/sbin/keepalived -d -D -S 0 --启动命令 http://www.linuxidc.com/Linux/2014-02/96081.htm 参数意义待确认，暂时用安装文件
cp /httx/run/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ -- 并修改该文件的参数 /var/lock/subsys/$prog 为 /httx/run/keepalived/lock/$prog
chmod 755 /etc/init.d/keepalived
cp /httx/run/keepalived/sbin/keepalived /usr/sbin/
cp /httx/run/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ --配置服务
#运行 由于keepalived配置文件不是在启动时一次性加载并读取完成的，所以必须是完整路径，可以根据log判断是否加载了正确的配置文件
keepalived –f /usr/local/etc/keepalived/keepalived.conf
运行过程可以查看log：
tail -f /var/log/message
二:安装redis
(redis和redis-slave两台服务器都安装)
tar -zxvf redis-2.4.17.tar.gz
cd redis-2.4.17
./configure
make
make install
redis-slve只需注意一点
[root@redis bin]# pwd
/usr/local/bin
[root@redis bin]# vim redis.conf
116 slaveof 192.168.1.235 6379
三：通过Keepalived实现Redis Failover自动故障切换
修改Master和Slave的/etc/hosts文件
[root@redis rc.d]# cat /etc/hosts
192.168.1.235 redis
192.168.1.236 redis-slave
Master上是
[root@redis rc.d]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=redis
Slave上是
[root@redis-slave linux]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=redis-slave
修改完主机名重启机器生效
默认安装完成keepalived之后是没有配置文件的，因此我们需要手动创建：
首先，在Master上创建如下配置文件：
[root@redis linux]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
[email protected]
}
notification_email_from keepalived@redis
router_id jt-db-redis-m1
}
vrrp_script chk_redis {
script "/httx/run/keepalived/etc/keepalived/scripts/redis_check.sh"
interval 1
# weight 4
}
vrrp_instance mes_Redis {
state MASTER
# interface eth0
interface br0
garp_master_delay 10
smtp_alert
virtual_router_id 3
priority 100
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.7.13.46
}
track_script {
chk_redis
}
notify_master /httx/run/keepalived/etc/keepalived/scripts/redis_master.sh
notify_backup /httx/run/keepalived/etc/keepalived/scripts/redis_slave.sh
notify_fault /httx/run/keepalived/etc/keepalived/scripts/redis_fault.sh
notify_stop /httx/run/keepalived/etc/keepalived/scripts/redis_stop.sh
}
然后，在Slave上创建如下配置文件：
[root@redis-slave linux]# vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
[email protected]
}
notification_email_from keepalived@redis-slave
router_id jt-db-redis-s1
}
vrrp_script chk_redis {
script "/httx/run/keepalived/etc/keepalived/scripts/redis_check.sh"
interval 1
# weight 4
}
vrrp_instance mes_Redis {
state BACKUP
# interface eth0
interface br0
garp_master_delay 10
smtp_alert
virtual_router_id 3
priority 99
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.7.13.46
}
track_script {
chk_redis
}
notify_master /httx/run/keepalived/etc/keepalived/scripts/redis_master.sh
notify_backup /httx/run/keepalived/etc/keepalived/scripts/redis_slave.sh
notify_fault /httx/run/keepalived/etc/keepalived/scripts/redis_fault.sh
notify_stop /httx/run/keepalived/etc/keepalived/scripts/redis_stop.sh
}
在Master和Slave上创建监控Redis的脚本
mkdir /etc/keepalived/scripts
vim /etc/keepalived/scripts/redis_check.sh
#!/bin/bash
ALIVE=`/usr/local/bin/redis-cli PING`
if [ "$ALIVE" == "PONG" ]; then
echo $ALIVE
exit 0
else
echo $ALIVE
exit 1
fi
编写以下负责运作的关键脚本：
notify_master /etc/keepalived/scripts/redis_master.sh
notify_backup /etc/keepalived/scripts/redis_slave.sh
notify_fault /etc/keepalived/scripts/redis_fault.sh
notify_stop /etc/keepalived/scripts/redis_stop.sh
因为Keepalived在转换状态时会依照状态来呼叫：
当进入Master状态时会呼叫notify_master
当进入Backup状态时会呼叫notify_backup
当发现异常情况时进入Fault状态呼叫notify_fault
当Keepalived程序终止时则呼叫notify_stop
首先，在Redis Master上创建notity_master与notify_backup脚本：
vim /etc/keepalived/scripts/redis_master.sh
#!/bin/bash
REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.236 6379 >> $LOGFILE 2>&1
sleep 10 #延迟10秒以后待数据同步完成后再取消同步状态
echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
vim /etc/keepalived/scripts/redis_slave.sh
#!/bin/bash
REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
sleep 15 #延迟15秒待数据被对方同步完成之后再切换主从角色
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.236 6379 >> $LOGFILE 2>&1
接着，在Redis Slave上创建notity_master与notify_backup脚本：
vim /etc/keepalived/scripts/redis_master.sh
#!/bin/bash
REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.235 6379 >> $LOGFILE 2>&1
sleep 10 #延迟10秒以后待数据同步完成后再取消同步状态
echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
vim /etc/keepalived/scripts/redis_slave.sh
#!/bin/bash
REDISCLI="/usr/local/bin/redis-cli"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
sleep 15 #延迟15秒待数据被对方同步完成之后再切换主从角色
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.235 6379 >> $LOGFILE 2>&1
然后在Master与Slave创建如下相同的脚本：
vim /etc/keepalived/scripts/redis_fault.sh
#!/bin/bash
LOGFILE=/var/log/keepalived-redis-state.log
echo "[fault]" >> $LOGFILE
date >> $LOGFILE
vim /etc/keepalived/scripts/redis_stop.sh
#!/bin/bash
LOGFILE=/var/log/keepalived-redis-state.log
echo "[stop]" >> $LOGFILE
date >> $LOGFILE
在Master与Slave给脚本都加上可执行权限：
chmod +x /etc/keepalived/scripts/*.sh
脚本创建完成以后，我们开始按照如下流程进行测试：
1.启动Master上的Redis
[root@redis bin]# pwd
/usr/local/bin
[root@redis bin]# ./redis-server redis.conf
2.启动Slave上的Redis
[root@redis bin]# pwd
/usr/local/bin
[root@redis bin]# ./redis-server redis.conf
3.启动Master上的Keepalived
/etc/init.d/keepalived start
4.启动Slave上的Keepalived
/etc/init.d/keepalived start
5.尝试通过VIP连接Redis:
[root@redis bin]#pwd
/usr/local/bin
[root@redis bin]# ./redis-cli -h 192.168.1.237 info
role:master
slave0:192.168.1.236,6379,online
连接成功，Slave也连接上来了
6.尝试插入一些数据：
[root@redis bin]# ./redis-cli -h 192.168.1.237 SET Hello Redis
从VIP读取数据
[root@redis bin]# ./redis-cli -h 192.168.1.237 GET Hello
"Redis"
从Master读取数据
[root@redis bin]# ./redis-cli -h 192.168.1.235 GET Hello
"Redis"
从Slave读取数据
[root@redis-slave bin]# ./redis-cli -h 192.168.1.235 GET Hello
"Redis"
下面，模拟故障产生：
将Master上的Redis进程杀死：
[root@redis bin]# ./redis-cli shutdown
查看Master上的Keepalived日志
[root@redis scripts]# tail /var/log/keepalived-redis-state.log
[fault]
Thu Sep 27 08:29:01 CST 2012
同时Slave上的日志显示：
[root@redis-slave scripts]# tail /var/log/keepalived-redis-state.log
[master]
Thu Nov 15 12:06:04 CST 2012
Being master....
Run SLAVEOF cmd ...
OK
Run SLAVEOF NO ONE cmd ...
OK
然后我们可以发现，Slave已经接管服务，并且担任Master的角色了。
./redis-cli -h 192.168.1.237 info
./redis-cli -h 192.168.1.236 info
role:master
然后我们恢复Master的Redis进程
主变成slave
然后把236redis停掉
235恢复主的角色，在把236redis开启
恢复235是主，236是备
自动切换成功！