搭建LVS+Keepalived高可用负载均衡集群
LVS+Keepalived 介绍
LVS
LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术（VS/NAT、VS/TUN和VS/DR）；
十种调度算法（rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq）。
Keepalvied
Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现
集群IP地址规划
节点类型 |
ip |
主机名 |
Master DR |
192.168.2.128 |
centos.peng.com |
Backup DR |
192.168.2.129 |
centos1.peng.com |
Real Server (nginx) |
192.168.2.130 |
centos2.peng.com |
Real Server (nginx) |
192.168.2.131 |
centos3.peng.com |
VIP |
192.168.2.200 |
www.peng.com |
安装LVS
检查系统是否支持lvs
[root@centos ~]# modprobe -l |grep ip_vs
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_dh.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_ftp.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lblc.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lblcr.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_lc.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_nq.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_rr.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_sed.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_sh.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_wlc.ko
/lib/modules/2.6.18-348.el5/kernel/net/ipv4/ipvs/ip_vs_wrr.ko
如有类似以上输出，说明系统支持IPVS模块
安装ipvsadm
[root@centos ~]# tar zxvf ipvsadm-1.24.tar.gz
[root@centos ~]# cd ipvsadm-1.24
[root@centos ipvsadm-1.24]# ln -s /usr/src/kernels/2.6.18-348.el5-x86_64/ /usr/src/linux/
[root@centos ipvsadm-1.24]# make
[root@centos ipvsadm-1.24]# make install
安装keepalived
[root@centos1 ~]# tar zxvf keepalived-1.1.19.tar.gz
[root@centos1 ~]# cd keepalived-1.1.19
[root@centos1 keepalived-1.1.19]#./configure --with-kernel-dir=/usr/src/kernels/2.6.18-348.el5-x86_64
[root@centos1 keepalived-1.1.19]# make
[root@centos1 keepalived-1.1.19]# make install
[root@centos1 keepalived-1.1.19]# ln -s /usr/local/sbin/keepalived /sbin/
Real Server 配置
LVS的DR和TUN模式下，用户的请求不经过DR直接返回给用户，因此需要在Real Server上添加VIP。用脚本realserver.sh在Real Server上执行，实现自动添加VIP的功能。
#!/bin/bash
VIP=192.168.2.200
/etc/rc.d/init.d/functions
case "$1" in
start)
/sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP
/sbin/route add -host $VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK"
;;
stop)
/sbin/ifconfig lo:0 down
/sbin/route del $VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped"
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
exit 0
配置keepalived
Keepalived的配置文件为/etc/Keepalived/Keepalived.conf
配置如下所示：
! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from [email protected]
smtp_server smtp.qq.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.2.200
}
}
virtual_server 192.168.2.200 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
real_server 192.168.2.130 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.2.131 80 {
weight 1
TCP_CHECK {
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
Backup DR的配置文件基本相同，state MASTER 换成 state backup ，priority 100 换成prority 80 即可。
启动keepalived 和realserver.sh
[root@centos keepalived]# /etc/init.d/keepalived start
[root@centos3 ~]# ./realserver.sh
启动realserver.sh脚本后，Realserver ip地址如下所示：
[root@centos2 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:76:A3:21
inet addr:192.168.2.130 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe76:a321/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8902 errors:0 dropped:0 overruns:0 frame:0
TX packets:6316 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2713466 (2.5 MiB) TX bytes:1065526 (1.0 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:4559 errors:0 dropped:0 overruns:0 frame:0
TX packets:4559 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10562584 (10.0 MiB) TX bytes:10562584 (10.0 MiB)
lo:0 Link encap:Local Loopback
inet addr:192.168.2.200 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
测试LVS+keepalived
在两台Real Server上利用nginx新建两台web服务器，都将监听的虚拟VIP指向www.peng.com。通过两台客户端（一台win 7，一台opensuse）访问www.peng.com。访问结果如下所示：
Win7 下的IE访问：
Opensuse下的firefox访问：
可以看到，通过不同客户端访问到的是不同的realserver，一台是centos2，一台是centos3。
利用ipvsadm查看如下：
Lvs监听到了192.168.2.130:80和192.168.2.131:80两台RealServer，131产生了4个链接，130产生了1个链接。
由此可以确定通过LVS+keepalived搭建的高可用负载均衡web服务顺利完成。
模拟DR故障
当Master DR当机的时候，我们通过关闭keepalived服务来模拟这一场景。看一下，Backup DR是怎样继续提供服务的。观察此时Backup DR日志，如下所示：
[root@centos1 ~]# tail -f /var/log/messages
Aug 6 05:45:23 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 6 05:45:24 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200
Aug 6 05:45:24 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added
Aug 6 05:45:24 centos1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added
Aug 6 05:45:29 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200
可以看到Backup DR 马上将自己调整为MASTER STATE ，并接管了虚拟ip192.168.2.200，让web服务仍然能够正常访问。
当重新启动Master DR上的keepalived服务，我们的Master DR从当机中恢复之后。观察Backup DR日志如下：
Aug 6 05:45:24 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added
Aug 6 05:45:24 centos1 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added
Aug 6 05:45:29 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200
Aug 6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Aug 6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 6 05:52:34 centos1 Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Aug 6 05:52:34 centos1 Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 removed
Aug 6 05:52:34 centos1 Keepalived_healthcheckers: Netlink reflector reports IP192.168.2.200 removed
可以看到Backup DR 马上又乖乖的将自己变成了BACKUP STATE 并移除了虚拟ip192.168.2.200，继续做着万年的备胎。
模拟RealServer故障
当RealServer当机的时候，我们通过关闭RealServer上的 nginx服务来模拟这一场景。看一下，DR是怎样运行的。观察此时Master DR日志，如下所示：
Aug 6 05:52:34 centos Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 6 05:52:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200
Aug 6 05:52:35 centos Keepalived_vrrp: Netlink reflector reports IP 192.168.2.200 added
Aug 6 05:52:35 centos Keepalived_healthcheckers: Netlink reflector reports IP 192.168.2.200 added
Aug 6 05:52:40 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.2.200
Aug 6 05:58:34 centos Keepalived_healthcheckers: TCP connection to [192.168.2.130:80] failed !!!
Aug 6 05:58:34 centos Keepalived_healthcheckers: Removing service [192.168.2.130:80] from VS [192.168.2.200:80]
Aug 6 05:58:34 centos Keepalived_healthcheckers: SMTP connection ERROR to [127.0.0.1:25].
可以看到keepalived检测到192.168.2.130:80服务当机，马上将其从VS中移除。此时VS中应该只有192.168.2.131:80提供服务，我们通过浏览器看一下，如下所示：
果真如此，通过两个客户端访问到的都是centos3提供的服务。