一，keepalived介绍


keepalived是一个可以实现某些资源高可用的开源软件，其主要的组件包括core，check，vrrp，libipfwc，libipvs，这里说下各个组件的功能。

core：keepalived的核心组件，负责主进程的启动和维护以及加载解析配置文件等。

check：负责healthchecker,负责各种健康检查方式，和对应的配置解析以及LVS的配置解析。

vrrp：vrrpd的子进程。

libipfwc：结合iptables的ipchains库来使用。

libipvs：结合LVS使用。

keepalived启动后会生成3个进程，master主进程，VRRP子进程，healthchecker子进程。

VRRP协议是实现keepalived高可用的一个基础，下面说一下VRRP的实现原理：

VRRP虚拟路由(VRRP router)，VRRP是一个“选举”协议，它能够动态地将一个虚拟路由器的责任指定至同一个VRRP组中的其它路由器上，VRRP的优势：

冗余：可以使用多个路由器设备作为LAN客户端的默认网关，大大降低了默认网关成为单点故障的可能性；

负载共享：允许来自LAN客户端的流量由多个路由器设备所共享；

多VRRP组：在一个路由器物理接口上可配置多达255个VRRP组；

多IP地址：基于接口别名在同一个物理接口上配置多个IP地址，从而支持在同一个物理接口上接入多个子网；

抢占：在master故障时允许优先级更高的backup成为master；

通告协议：使用IANA所指定的组播地址224.0.0.18进行VRRP通告；

VRRP追踪：基于接口状态来改变其VRRP优先级来确定最佳的VRRP路由器成为master；


二，实验环境：

192.168.30.116 OS：Centos 6.4 x86_64   master.luojianlong.com

192.168.30.117 OS：Centos 6.4 x86_64   backup.luojianlong.com

keepalived版本：keepalived-1.2.7

keepalived实现haproxy高可用详解_第1张图片


首先，分别在2台服务器上，安装keepalived,haproxy，由于系统版本是Centos 6.4，这2个软件已经被整合在内部了，所以使用yum来安装

[root@master ~]# yum -y install keepalived haproxy
[root@backup ~]# yum -y install keepalived haproxy


接下来，先配置master与backup服务器的优先级，virtual_router_id（同一个实例2台服务器必须相同），编辑配置文件:

[root@master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@backup ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
[root@master ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.30.230
    }
}
[root@backup ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.30.230
    }
}


global_defs：全局配置标识破；

notification_email：表示告警时发送的邮件地址；

notification_email_from：表示发送邮件的源地址；

smtp_server：发送邮件的smtp服务器地址；

router_id：机器标识；

vrrp_instance：定义一个vrrp实例；

state：state指定instance(Initial)的初始状态，就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定，里如果这里设置为master，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为master；

interface：实例绑定的网卡，因为在配置虚拟IP的时候必须是在已有的网卡上添加的；

virtual router id：这里设置VRID，这里非常重要，相同的VRID为一个组，他将决定多播的MAC地址；

priority 100：设置本节点的优先级，优先级高的为master；

advert int：检查间隔，默认为1秒；

virtual ipaddress：这里设置的就是VIP，也就是虚拟IP地址，他随着state的变化而增加删除，当state为master的时候就添加，当state为backup的时候删除，这里主要是有优先级来决定的，和state设置的值没有多大关系，这里可以设置多个IP地址；

authentication：这里设置认证；

auth type：认证方式，可以是PASS或AH两种认证方式；

auth pass：认证密码；


启动俩台服务器的keepalived

[root@master ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@backup ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@master ~]# tail -f /var/log/messages
Jan 10 11:40:56 localhost Keepalived_healthcheckers[19368]: Using LinkWatch kernel netlink reflector...
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Configuration is using : 63019 Bytes
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: Using LinkWatch kernel netlink reflector...
Jan 10 11:40:56 localhost Keepalived_vrrp[19369]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 11:40:57 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 11:40:58 localhost Keepalived_healthcheckers[19368]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 11:40:58 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 11:41:03 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Jan 10 11:41:14 localhost Keepalived_vrrp[19369]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
[root@master ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0
    inet 192.168.30.230/32 scope global eth0
    inet6 fe80::20c:29ff:fef3:fcba/64 scope link
       valid_lft forever preferred_lft forever

发现刚才定义的virtual ipaddress在master服务器上，因为优先级较高


停止master服务器的keepalived服务器，看IP会不会转移到backup

[root@master ~]# service keepalived stop
Stopping keepalived:                                       [  OK  ]
[root@backup ~]# tail -f /var/log/messages
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Configuration is using : 63017 Bytes
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: Using LinkWatch kernel netlink reflector...
Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Opening file '/etc/keepalived/keepalived.conf'.
Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Configuration is using : 7324 Bytes
Jan 10 12:12:46 localhost Keepalived_healthcheckers[18580]: Using LinkWatch kernel netlink reflector...
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
[root@backup ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.117/24 brd 192.168.30.255 scope global eth0
    inet 192.168.30.230/32 scope global eth0
    inet6 fe80::20c:29ff:fe5b:50f9/64 scope link
       valid_lft forever preferred_lft forever

发现IP已经转移到backup服务器

下面重新启动master的keepalived

[root@master ~]# service keepalived start
Starting keepalived:                                       [  OK  ]
[root@backup ~]# tail -f /var/log/messages
Jan 10 12:12:46 localhost Keepalived_vrrp[18581]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert
Jan 10 12:12:47 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 10 12:16:27 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 12:16:28 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 12:16:28 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:16:33 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Received higher prio advert
Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 10 12:18:20 localhost Keepalived_vrrp[18581]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 10 12:18:20 localhost Keepalived_healthcheckers[18580]: Netlink reflector reports IP 192.168.30.230 removed
[root@master ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0
    inet 192.168.30.230/32 scope global eth0
    inet6 fe80::20c:29ff:fef3:fcba/64 scope link
       valid_lft forever preferred_lft forever

发现IP已经重新转移到master服务器


现在编写haproxy状态检测脚本，来实现haproxy的健康检测：

[root@master ~]# cat haproxy_pid.sh
#!/bin/bash
while :
do
haproxypid=`ps -C haproxy --no-header | wc -l`
if [ $haproxypid -eq 0 ];then
   service haproxy start
   sleep 5
   haproxypid=`ps -C haproxy --no-header | wc -l`
   echo $haproxypid
     if [ $haproxypid -eq 0 ];then
   /etc/init.d/keepalived stop
     fi
fi
sleep 5
done
# 启动backup的haproxy
[root@backup ~]# service haproxy start
Starting haproxy:                                          [  OK  ]

模拟故障，先让httpd进程开启，修改haproxy监听端口为80，使得haproxy进程无法启动，看资源会不会转移到backup服务器

[root@master ~]# vi /etc/haproxy/haproxy.cfg
frontend  main *:5000 改为frontend  main *:80
# 启动httpd进程
[root@master ~]# scp -pr /etc/haproxy/haproxy.cfg [email protected]:/etc/haproxy/
[root@master ~]# service httpd start
Starting httpd: httpd: apr_sockaddr_info_get() failed for master.luojianlong.com
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
                                                           [  OK  ]
[root@master ~]# netstat -antpl | grep :80
tcp        0      0 :::80                       :::*                        LISTEN      19965/httpd
#运行状态检测脚本
[root@master ~]# nohup /root/haproxy_pid.sh &
[root@master ~]# scp -pr haproxy_pid.sh [email protected]:/root/
[root@bakcup ~]# nohup /root/haproxy_pid.sh &
[root@master ~]# tail -f /var/log/messages
Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jan 10 12:02:29 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 10 12:02:30 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:02:30 localhost Keepalived_healthcheckers[19848]: Netlink reflector reports IP 192.168.30.230 added
Jan 10 12:02:35 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.30.230
Jan 10 12:14:49 localhost Keepalived[19847]: Stopping Keepalived v1.2.7 (02/21,2013)
Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) sending 0 priority
Jan 10 12:14:49 localhost Keepalived_vrrp[19849]: VRRP_Instance(VI_1) removing protocol VIPs.
[root@master ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:f3:fc:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.116/24 brd 192.168.30.255 scope global eth0
    inet6 fe80::20c:29ff:fef3:fcba/64 scope link
       valid_lft forever preferred_lft forever
[root@backup ~]# ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:5b:50:f9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.117/24 brd 192.168.30.255 scope global eth0
    inet 192.168.30.230/32 scope global eth0
    inet6 fe80::20c:29ff:fe5b:50f9/64 scope link
       valid_lft forever preferred_lft forever
[root@backup ~]# ps aux | grep haproxy
haproxy  19054  0.0  0.0  18688  1280 ?        Ss   12:47   0:00 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
root     19097  0.0  0.0 103248   828 pts/0    S+   12:58   0:00 grep haproxy

发现IP已经转移到backup服务器，实现了keepalived对于haproxy故障的高可用。


keepalived常见的启动报错：

5913 May 16 15:26:04 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.75
5914 May 16 15:26:04 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
5915 May 16 15:26:04 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
5916 May 16 15:26:04 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment...
5917 May 16 15:26:05 localhost Keepalived_vrrp: ip address associated with VRID not present in received packet : 192.168.57.75
5918 May 16 15:26:05 localhost Keepalived_vrrp: one or more VIP associated with VRID mismatch actual MASTER advert
5919 May 16 15:26:05 localhost Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
5920 May 16 15:26:05 localhost Keepalived_vrrp: VRRP_Instance(VI_1) ignoring received advertisment.

解决方法：

在同一网段内virtual_router_id 值不能相同，如果相同会在messages中收到VRRP错误包 ，所以需要更改 virual_router_id。