Nginx+KeepAlive 高可用性实施

由于需要利用Nginx 在网通做一个电信的反代（动态网页），目前公司所有的站点都在电信，那么Nginx反代就需要做成高可用性，防止单点故障，于是就想到了做HA的强劲软件 keepalived，keepalived做LVS健康检测的资料倒是比较多，不过做双机的资料不多，自己看了官方资料，利用VRRP做了双机高可用，特此记录：

1、服务器IP部署：
(1), Master（eth0:58.22.XXX.207/27接外网; eth1:10.2.1.207/8接内网）
(2), Backup（eth0:58.22.XXX.208/27接外网; eth1:10.2.1.208/8接内网）
VIP:58.22.XXX.202
网关：58.22.XXX.192

2、软件下载解压：
wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
tar zxvf keepalived-1.1.15.tar.gz
cd keepalived-1.1.15

3、编译安装
./configure –prefix=/
make
make install

这里需要动态链接库的支持，因为本例是使用Centos，所以可以直接使用yum 安装：
libpopt-dev
libssl-dev


4、配置
A、主机Mater配置文件
vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.139.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_instance VI_1 {
state MASTER #(主机为MASTER，备用机为BACKUP)
interface eth0 #(HA监测网络接口)
track_interface { #其他要监测状态的接口
eth1
}
virtual_router_id 51 #(主、备机的virtual_router_id必须相同)
mcast_src_ip 58.22.XXX.207 #(多播的源IP，设置为本机外网IP，与VIP同一网卡)

priority 500 #(主、备机取不同的优先级，主机值较大，备份机值较小,值越大优先级越高)
advert_int 1 #(VRRP Multicast广播周期秒数)

authentication {
auth_type PASS #(VRRP认证方式)
auth_pass XXXXX #(密码)
}
virtual_ipaddress {
58.22.XXX.202 #(VRRP HA虚拟地址)
}
}

这里不做LVS，删除后面的virtual_server内容部分

B、从机Backup配置文件

! Configuration File for keepalived

global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server smtp.139.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_instance VI_1 {
state BACKUP #(与Master不同的地方)
interface eth0
track_interface {
eth1
}
virtual_router_id 51
mcast_src_ip 58.22.XXX.208
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass XXXX
}
virtual_ipaddress {
58.22.XXX.202
}
}

track_interface的意思是将Linux中你想监控的网络接口卡监控起来，当其中的一块出现故障是keepalived都将视为路由器出现故障。

这里请注意： virtual_router_id 51，同一组master/backup中，也就是如果你设置了多个VRRP，同一组这个ID必须相同，不然启动的时候主从都会把VIP给启动了，我就是在这里郁闷了很久，这里是看了官方的反而把我搞得很郁闷！！英文不够强劲啊

5、启动
在Master上

ip addr list
启动keepalived 前

[root@L1-FC-XXX-207 keepalived-1.1.15]# ip addr list
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d7 brd ff:ff:ff:ff:ff:ff
inet 58.22.XXX.207/27 brd 58.22.XXX.223 scope global eth0
inet6 fe80::213:72ff:fe68:1cd7/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d8 brd ff:ff:ff:ff:ff:ff
inet 10.2.1.207/8 brd 10.255.255.255 scope global eth1
inet6 fe80::213:72ff:fe68:1cd8/64 scope link
valid_lft forever preferred_lft forever
4: sit0: mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0
启动keepalived
/etc/init.d/keepalived start
启动keepalived 后
1: lo: mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d7 brd ff:ff:ff:ff:ff:ff
inet 58.22.XXX.207/27 brd 58.22.XXX.223 scope global eth0
inet 58.22.XXX.202/27 scope global secondary eth0
inet6 fe80::213:72ff:fe68:1cd7/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:13:72:68:1c:d8 brd ff:ff:ff:ff:ff:ff
inet 10.2.1.207/8 brd 10.255.255.255 scope global eth1
inet6 fe80::213:72ff:fe68:1cd8/64 scope link
valid_lft forever preferred_lft forever
4: sit0: mtu 1480 qdisc noop
link/sit 0.0.0.0 brd 0.0.0.0

可以看到多了一个虚拟IP

inet 58.22.XXX.202/27 scope global secondary eth0

查看keepalived日志

less /var/log/messages
Nov 19 14:27:59 FC-L103207 Keepalived: Terminating on signal
Nov 19 14:27:59 FC-L103207 Keepalived_vrrp: Terminating VRRP child process on signal
Nov 19 14:27:59 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 19 14:27:59 FC-L103207 Keepalived: Stopping Keepalived v1.1.15 (11/19,2008)
Nov 19 14:27:59 FC-L103207 keepalived: keepalived shutdown succeeded
Nov 19 14:29:39 FC-L103207 keepalived: keepalived shutdown failed
Nov 19 14:29:39 FC-L103207 Keepalived: Starting Keepalived v1.1.15 (11/19,2008)
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Using MII-BMSR NIC polling thread…
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Registering Kernel netlink reflector
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Registering Kernel netlink command channel
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Registering gratutious ARP shared channel
Nov 19 14:29:39 FC-L103207 Keepalived: Starting VRRP child process, pid=7098
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: Configuration is using : 37126 Bytes
Nov 19 14:29:39 FC-L103207 keepalived: keepalived startup succeeded
Nov 19 14:29:39 FC-L103207 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(7,8)]
Nov 19 14:29:40 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 19 14:29:41 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 19 14:29:41 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 19 14:29:41 FC-L103207 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 58.22.XXX.202

可以看出状态为MASTER

此时在BACKUP机上启动keepalived，再查看日志

Nov 19 14:29:56 FC-L103208 Keepalived: Terminating on signal
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Terminating VRRP child process on signal
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Nov 19 14:29:56 FC-L103208 Keepalived: Stopping Keepalived v1.1.15 (11/19,2008)
Nov 19 14:29:56 FC-L103208 keepalived: keepalived shutdown succeeded
Nov 19 14:29:56 FC-L103208 Keepalived: Starting Keepalived v1.1.15 (11/19,2008)
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Using MII-BMSR NIC polling thread…
Nov 19 14:29:56 FC-L103208 Keepalived: Starting VRRP child process, pid=20706
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Registering Kernel netlink reflector
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Registering Kernel netlink command channel
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Registering gratutious ARP shared channel
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Opening file '/etc/keepalived/keepalived.conf'.
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: Configuration is using : 37126 Bytes
Nov 19 14:29:56 FC-L103208 keepalived: keepalived startup succeeded
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 19 14:29:56 FC-L103208 Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(7,8)]
此时如果停止MASTER上的keepalived

那么在BACKUP的日志会出现

Nov 19 14:33:55 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Nov 19 14:33:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Nov 19 14:33:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Nov 19 14:33:56 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 58.22.103.202

即BACKUP接管了VIP
再启动MASTER上的keepalived
日志会出现：

Nov 19 14:34:01 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 58.22.103.202
Nov 19 14:35:40 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Nov 19 14:35:40 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Nov 19 14:35:40 FC-L103208 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
则是说明有更高优先级的机器即MASTER启动，去除VIP，还给更优先级的机器

这样就实现了如果某一主机宕机，IP自动切换到备机上继续运行

keepalived官方文档：http://www.keepalived.org/documentation.html