基于keepalived+sentinel+redis主从的redis高可用性架构方案

基于keepalived+sentinel+redis主从的redis高可用性架构方案

环境说明

在一般小项目中的我们常用redis主从模式，redis数据库是单进程单线程的架构，使用单机或简单的主从模式，来尽量保证缓存数据库的可持久化，当单节点出问题时，我们可以从节点来获取数据。
而当在规模大的项目或生产环境要求高的时，一般redis高可用性架构。 一般来说，redis高可用性架构有在致三类高可用性： redis主从模式、redis哨兵模式、redis cluster模式。

redis主从模式：设置一个redis server master 主机，一个redis server slave（从机）。 从机定期从主机同步数据。 一般使用的配置是master端提供的读写服务，而在slave端提供只读服务。中心化的架构
redis哨兵模式：Redis-Sentinel是Redis官方推荐的高可用性(HA)解决方案，当用Redis做Master-slave的高可用方案时，假如master宕机了，Redis本身(包括它的很多客户端)都没有实现自动进行主备切换，而Redis-sentinel本身也是一个独立运行的进程，它能监控多个master-slave集群，发现master宕机后能进行自动切换。Sentinel由一个或多个Sentinel 实例 组成的Sentinel 系统可以监视任意多个主服务器，以及这些主服务器属下的所有从服务器，并在被监视的主服务器进入下线状态时，自动将下线主服务器属下的某个从服务器升级为新的主服务器。 中心化的架构。
redis cluster模式：redis cluster在设计的时候，就考虑到了去中心化，也就是说，集群中的每个节点都是平等的关系，都是对等的，每个节点都保存各自的数据和整个集群的状态。每个节点都和其他所有节点连接，而且这些连接保持活跃，这样就保证了我们只需要连接集群中的任意一个节点，就可以获取到其他节点的数据。去中心化的架构。

这次高可用性实验，我们实践基于keepalived+redis-sentinel+redis(master-slave)实现的高可用性架构方案，通过keepalived来实现对外统一的VIP服务，当集群中redis节点出现问题时，则可自动进行故障切换、VIP飘逸，以此来实现对应用端的高可用性访问。

环境信息

操作中系统	ip	主机名	软件
Linux7.5	192.168.174.129	srvb	keepalived+sentinel+redis (master)
Linux7.5	192.168.174.130	srvc	keepalived+sentinel+redis(slave)
Linux7.5	192.168.174.131	srvd	keepalivedd+sentinel+redis

对外提供应用连接的IP地址（VIP）为：192.168.174.130

架构图

环境搭建

安装redis

wget http://download.redis.io/releases/redis-5.0.5.tar.gz
tar zxvf redis-5.0.5.tar.gz
cd redis-5.0.5
make PREFIX=/usr/local/redis install # 将redis安装/usr/local/redis目录

配置命令行引用
ln -s /usr/local/redis/bin/redis-server /usr/bin/redis-server
ln -s /usr/local/redis/bin/redis-server /usr/bin/redis-sentinel
ln -s /usr/local/redis/bin/redis-cli /usr/bin/redis-cli

以上srvb srvc srvd三个节点都执行相应的操作

安装keepalived

目录规划：
/usr/local/keepalived #安装目录

下载：
https://www.keepalived.org/download.html

cd /usr/local/src/keepalived-1.2.6
configure --prefix=/usr/local/keepalived
make
make install
查看安装后文件：
ls -l /usr/local/keepalived/
总用量 16
drwxr-xr-x 2 root root 4096 8月 29 14:46 bin
drwxr-xr-x 5 root root 4096 8月 29 14:46 etc
drwxr-xr-x 2 root root 4096 8月 29 14:46 sbin
drwxr-xr-x 3 root root 4096 8月 29 14:46 share

安装后文件说明：
etc配置示例及开机启动的配置文件
ls -l etc
drwxr-xr-x 3 root root 4096 8月 29 14:46 keepalived
drwxr-xr-x 3 root root 4096 8月 29 14:46 rc.d
drwxr-xr-x 2 root root 4096 8月 29 14:46 sysconfig

安装之后的目录结构：
.
├── bin
│ └── genhash
├── etc
│ ├── keepalived
│ │ ├── keepalived.conf
│ │ └── samples
│ │ ├── client.pem
│ │ ├── dh1024.pem
│ │ ├── keepalived.conf.fwmark
│ │ ├── keepalived.conf.HTTP_GET.port
│ │ ├── keepalived.conf.inhibit
│ │ ├── keepalived.conf.IPv6
│ │ ├── keepalived.conf.misc_check
│ │ ├── keepalived.conf.misc_check_arg
│ │ ├── keepalived.conf.quorum
│ │ ├── keepalived.conf.sample
│ │ ├── keepalived.conf.SMTP_CHECK
│ │ ├── keepalived.conf.SSL_GET
│ │ ├── keepalived.conf.status_code
│ │ ├── keepalived.conf.track_interface
│ │ ├── keepalived.conf.virtualhost
│ │ ├── keepalived.conf.virtual_server_group
│ │ ├── keepalived.conf.vrrp
│ │ ├── keepalived.conf.vrrp.localcheck
│ │ ├── keepalived.conf.vrrp.lvs_syncd
│ │ ├── keepalived.conf.vrrp.routes
│ │ ├── keepalived.conf.vrrp.scripts
│ │ ├── keepalived.conf.vrrp.static_ipaddress
│ │ ├── keepalived.conf.vrrp.sync
│ │ ├── root.pem
│ │ └── sample.misccheck.smbcheck.sh
│ ├── mkdir
│ ├── rc.d
│ │ └── init.d
│ │ └── keepalived
│ └── sysconfig
│ └── keepalived
├── sbin
│ └── keepalived
└── share
└── man
├── man1
│ └── genhash.1
├── man5
│ └── keepalived.conf.5
└── man8
└── keepalived.8

sbin/keepalived keepalived执行文件
bin/gensh 执行文件

将keepalived注册为系统服务：

cp /usr/local/keepalived/sbin/keepalived /usr/local/sbin
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig

etc/sysconfig/keepalived 这个配置文件，在使用service管理启动服务的时候，默认的从这个文件中读取启动参数。也就是启动脚本的参数文件。
读取KEEPALIVED_OPTIONS中配置的参数来启动

通过查看/etc/rc.d/init.d/keepalived执行脚本内容
我们可以分析keepalived-1.2.6版本的启动配置文件为 /etc/keepalived/keepalived.conf配置
脚本部分内容：

cat /etc/rc.d/init.d/keepalived
#!/bin/sh
#
# Startup script for the Keepalived daemon
#
# processname: keepalived
# pidfile: /var/run/keepalived.pid  #PID文件存放的位置
# config: /etc/keepalived/keepalived.conf  # 这个配置文件存放的目录，所以我新建这个目录
# chkconfig: - 21 79
# description: Start and stop Keepalived

# Source function library
. /etc/rc.d/init.d/functions

# Source configuration file (we set KEEPALIVED_OPTIONS there)
. /etc/sysconfig/keepalived

RETVAL=0

prog="keepalived"

start() {
    echo -n $"Starting $prog: "
    daemon keepalived ${KEEPALIVED_OPTIONS}
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
}

stop() {
    echo -n $"Stopping $prog: "
    killproc keepalived
    RETVAL=$?
    echo
    [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
}

reload() {
    echo -n $"Reloading $prog: "
    killproc keepalived -1
    RETVAL=$?
    echo
}

mkdir -p /etc/keepalived/
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

/usr/local/keepalived/etc/keepalived/keepalived.conf 这个文件是安装之后的示例配置文件。 我们需要修改的内容，可以根据这个进行相应的修改

启动：
service keepalived start
默认会有三个进程，一个father进程，两个子进程。
father负责监控两个子进程，分别是VRRP和checkers

存在的问题：
问题1：
[root@JHGYlmsHSTEST init.d]# service keepalived start
正在启动 keepalived：/bin/bash: keepalived: command not found
[失败]

原因和处理：
cp /usr/local/keepalived/sbin/keepalived /usr/sbin

使用serivce管理服务时，则执行命令必有是在/usr/sbin目录下才可以的。

问题2：
2、MASTER主机message日志中不断有Received lower prio advert, forcing new election记录：
原因：防火墙导致
解决方法：1、关闭防火墙；
2、或者在防火墙中开放目标主机中的数据包

配置redis主从

配置主从的架构

目录规划：
配置文件存放：/usr/local/redis/conf
数据文件存放：/usr/local/redis/data

redis master配置文件：

#master : redis.conf文件
port 6379
bind 0.0.0.0
pidfile "/var/run/redis_6379.pid"
#cluster-enabled yes
#cluster-config-file nodes.conf
#cluster-node-timeout 5000
appendonly yes
daemonize yes
#requirepass redis

dbfilename "dump.rdb"
dir "/usr/local/redis/data"
save 900 1
save 300 10
save 60 1000
# Generated by CONFIG REWRITE
#replicaof 192.168.174.131 6379

redis slave配置文件

[root@srvc ~]# cat /usr/local/redis/conf/redis.conf 
port 6379
bind 0.0.0.0
pidfile "/var/run/redis_6379.pid"
#cluster-enabled yes
#cluster-config-file nodes.conf
#cluster-node-timeout 5000
appendonly yes
daemonize yes
#requirepass redis

dbfilename "dump.rdb"
dir "/usr/local/redis/data"
save 900 1
save 300 10
save 60 1000
slaveof 192.168.174.129 6379

启动主从模式
start_redis.sh

[root@srvb ~]# cat redis/start_redis.sh 

REDIS_DIR=/usr/local/redis

redis-server ${REDIS_DIR}/conf/redis.conf

ssh srvc <

查看redis主从状态：

[root@srvb redis]# cat chk_redis.sh 
redis-cli -p 6379 info|grep role
[root@srvb redis]# ./chk_redis.sh 
role:master

配置redis-sentinel哨兵模式

配置文件：
/usr/local/redis/conf/sentinel
端口规划：
我们在此环境中设置redis-sentinel使用的是27000端口

配置文件内容

[root@srvb redis]# cat /usr/local/redis/conf/sentinel.conf 
port 27000
bind 0.0.0.0
daemonize yes
dir "/usr/local/redis/data"
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 192.168.174.129 6379 2
sentinel config-epoch mymaster 4
sentinel leader-epoch mymaster 4
sentinel known-replica mymaster 192.168.174.130 6379

在srvb srvc srvd三个主机都需要做相同的配置。 注意：这里我们使用的是/usr/local/redis/data做为sentinel的数据存放目录，因为没有配置缓存数据的持久化文件rdb或aof文件。所以可以直接使用。如果需要将缓存数据持久化，则需要指定不同的rdb和aof文件名来区分是redis还是的sentinel的的持久化文件

启动三个节点的sentinel:

[root@srvb redis]# cat start_sentinel.sh 

REDIS_DIR=/usr/local/redis

redis-sentinel ${REDIS_DIR}/conf/sentinel.conf

ssh srvc <

查看redis-sentinel状态：
redis-cli -p 27000
info sentinel

# 在主节点执行
127.0.0.1:27000> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.174.129:6379,slaves=2,sentinels=3

配置keepalived

配置文件目录：
/etc/keepalived
配置文件
keepalived.conf

查看配置文件内容

srvb(redis-master)

[root@srvb redis]# cat /etc/keepalived/keepalived.conf
global_defs {
   router_id HA
}

vrrp_script check_redis {
        script "redis-cli -h 192.168.174.129 -p 6379 info|grep role:master >/dev/null 2>&1"
        interval 1 
        timeout 3 
        fall 3
        rise 1
}

vrrp_instance VI_1 {
    state BACKUP
#    state MASTER
    interface eno16777736
    virtual_router_id 52
    priority 100
    advert_int 1
    nopreempt
    #preempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script {
                check_redis
        }

    virtual_ipaddress {
    192.168.174.137/24
    }
}

srvc(redis-slave):

[root@srvc ~]# cat redis/start_redis.sh 

REDIS_DIR=/usr/local/redis

redis-server ${REDIS_DIR}/conf/redis.conf

ssh srvc </dev/null 2>&1"
        interval 1 
        timeout 3 
        fall 3
        rise 1
}

vrrp_instance VI_1 {
    state BACKUP
#    state MASTER
    interface eno16777736
    virtual_router_id 52
    priority 90
    advert_int 1
    nopreempt
    #preempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script {
                check_redis
        }

    virtual_ipaddress {
    192.168.174.137/24
    }
}

srvd: 第三个节点的keepalived
这个节点上只部署有keepalived+sentinel

配置文件内容：

[root@srvd ~]# cat /etc/keepalived/keepalived.conf
global_defs {
   router_id HA
}

vrrp_script check_redis {
        script "redis-cli -h 192.168.174.130 -p 6379 info|grep role:master >/dev/null 2>&1"
        interval 1 
        timeout 3 
        fall 3
        rise 1
}

vrrp_instance VI_1 {
    state BACKUP
#    state MASTER
    interface eno16777736
    virtual_router_id 52
    priority 80
    advert_int 1
    nopreempt
    #preempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
        track_script {
                check_redis
        }

    virtual_ipaddress {
    192.168.174.137/24
    }
}

三个节点的keepalived的配置通过优先级的值的不同来设置，srvb是redis-master所在节点，在初始阶段我们将设置优先级最高。

启动keepalived:

[root@srvb redis]# cat start_keepalived.sh 
service keepalived start

ssh srvb << EOF
service keepalived start
EOF

ssh srvc << EOF
service keepalived start
EOF
[root@srvb redis]# 

启动之后查看
srvb节点:

# ip a
[root@srvb redis]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:c3:d5:ce brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.129/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.174.137/24 scope global secondary eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fec3:d5ce/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0:  mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:e2:5a:ca brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic:  mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:e2:5a:ca brd ff:ff:ff:ff:ff:ff
[root@srvb redis]# 

srvc节点

[root@srvc ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:de:98:68 brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.130/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fede:9868/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0:  mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic:  mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff

srvd节点

[root@srvd ~]#  ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:d5:e2:8b brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.131/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fed5:e28b/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0:  mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:1f:76:64 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic:  mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:1f:76:64 brd ff:ff:ff:ff:ff:ff

至此，keepalived+sentinel+redis(master-slave)高可用性架构搭建完成。

切换测试

关闭主redis-master:
在srvb上执行

[root@srvb redis]# ps -ef|grep redis
root      22580      1  0 20:03 ?        00:00:53 redis-server 0.0.0.0:6379
root      22628      1  0 20:05 ?        00:01:11 redis-sentinel 0.0.0.0:27000 [sentinel]
root      58417  44097  0 22:28 pts/3    00:00:00 grep --color=auto redis
[root@srvb redis]# redis-cli -p 6379 shutdown 
[root@srvb redis]# ps -ef|grep redis
root      22628      1  0 20:05 ?        00:01:12 redis-sentinel 0.0.0.0:27000 [sentinel]
root      58524  44097  0 22:29 pts/3    00:00:00 grep --color=au

至srvc上查看slave提升为主：

[root@srvc redis]# cat chk_redis.sh 
redis-cli -p 6379 info|grep role
[root@srvc redis]# ./chk_redis.sh 
role:master

可以看到此时备的slave上已提升为主了。

查看VIP是否切换到srvc节点上：

[root@srvc redis]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:de:98:68 brd ff:ff:ff:ff:ff:ff
    inet 192.168.174.130/24 brd 192.168.174.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 192.168.174.137/24 scope global secondary eno16777736
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fede:9868/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr0:  mtu 1500 qdisc noqueue state DOWN 
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic:  mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 500
    link/ether 52:54:00:82:fc:00 brd ff:ff:ff:ff:ff:ff

可以看到：
inet 192.168.174.137/24 scope global secondary eno16777736
valid_lft forever preferred_lft forever

VIP地址已从srvb切换到srvc节点上。 对前端应用来说这种是平滑的切换。
业务可能感知到短时间的中断，但是很快会恢复。

补充说明

1、当redis-sentinel模式检测并发现redis服务主从切换时，在这个过程中会修改原主从redis的配置文件，配置修改成切换后的应用状态。以便服务恢复时，能够按照新的配置来启动主从架构
2、当启动redis-sentinel时，本身的sentinel.conf配置文件会加入一此些集群的配置。