负载均衡软件HAProxy案例二 带cookie前缀和高可用的HTTP负载均衡
案例二 带cookie前缀和高可用的HTTP负载均衡
如果不想增加多余的cookie，而是用现成的cookie。后端应用已经生成了一个名为JSESSIONID的cookie，下面将在这个cookie前面加上服务器名称的前缀。由于负载均衡器是整个架构中最为重要的部件，所以这里通过使用Keepalived为LB1作备份。
Keepalived 软件可以通过yum install keepalived 安装
LB1和LB2共享一个VIP 192.168.1.1，它们通过Keepalived进行VIP的接管。
在LB1和LB2配置HAProxy如下内容:
listen webfarm *:80 mode http balance roundrobin cookie JSESSIONID prefix option httpclose option forwardfor option httpchk HEAD /index.html HTTP/1.0 server webA 192.168.1.11:80 cookie A check server webB 192.168.1.12:80 cookie B check server webC 192.168.1.13:80 cookie C check server webD 192.168.1.14:80 cookie D check
这里需要注意一下， listen 绑定的IP地址必须是真实存在的IP地址，如果不存在HAProxy将启动失败，如，LB2还没有接管192.168.1.1这个IP地址，那么在LB2上HAProxy将启动失败。为了避免因为VIP没有被接管而启动不了HAProxy的情况发生，可以将绑定IP和端口设置成 *.80 或 0.0.0.0:80 .
在LB1和LB2配置Keepalived如下内容:
! Configuration File for keepalived
global_defs {
notification_email {
[email protected] 设置通知接收邮件地址
}
notification_email_from [email protected] 设置通知发送邮件地址
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LB1_MASTER
}
vrrp_script chk_haproxy { 检查HAProxy是否存在
script "killall -0 haproxy"
interval 2 每2秒检查一下HAProxy进程
weight 2
}
vrrp_instance VI_1 {
state MASTER LB1上设置为MASTER,LB2上设置为BACKUP
interface eth0
virtual_router_id 51
priority 100 MASTER设置为100，BACKUP设置为99，MASTER的值必须大于BACKUP
advert_int 1 设置MASTER和BACKUP之间同步检查时间间隔
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.1/24 brd 192.168.1.255 dev eth0 label eth0:ha
设置虚拟IP地址，为了便于查看虚拟IP，可以设置一个
}
track_script {
chk_haproxy
}
}
在LB1上启动HAProxy和Keepalived，查看/var/log/messages日志
Jun 2 15:26:21 LB1 Keepalived[14391]: Starting Keepalived v1.2.7 (02/21,2013) Jun 2 15:26:21 LB1 Keepalived[14392]: Starting Healthcheck child process, pid=14394 Jun 2 15:26:21 LB1 Keepalived[14392]: Starting VRRP child process, pid=14395 Jun 2 15:26:21 LB1 Keepalived_vrrp[14395]: Interface queue is empty Jun 2 15:26:21 LB1 Keepalived_vrrp[14395]: Netlink reflector reports IP 192.168.1.3 added Jun 2 15:26:21 LB1 Keepalived_vrrp[14395]: Netlink reflector reports IP fe80::a00:27ff:fec8:f51b added Jun 2 15:26:21 LB1 Keepalived_vrrp[14395]: Registering Kernel netlink reflector Jun 2 15:26:21 LB1 Keepalived_vrrp[14395]: Registering Kernel netlink command channel Jun 2 15:26:21 LB1 Keepalived_healthcheckers[14394]: Interface queue is empty Jun 2 15:26:21 LB1 Keepalived_healthcheckers[14394]: Netlink reflector reports IP 192.168.1.3 added Jun 2 15:26:21 LB1 Keepalived_healthcheckers[14394]: Netlink reflector reports IP fe80::a00:27ff:fec8:f51b added Jun 2 15:26:21 LB1 Keepalived_healthcheckers[14394]: Registering Kernel netlink reflector Jun 2 15:26:21 LB1 Keepalived_healthcheckers[14394]: Registering Kernel netlink command channel Jun 2 15:26:21 LB1 Keepalived_vrrp[14395]: Registering gratuitous ARP shared channel Jun 2 15:26:41 LB1 Keepalived_vrrp[14395]: Opening file '/etc/keepalived/keepalived.conf'. Jun 2 15:26:41 LB1 Keepalived_healthcheckers[14394]: Opening file '/etc/keepalived/keepalived.conf'. Jun 2 15:26:41 LB1 Keepalived_vrrp[14395]: Configuration is using : 38559 Bytes Jun 2 15:26:41 LB1 Keepalived_vrrp[14395]: Using LinkWatch kernel netlink reflector... Jun 2 15:26:41 LB1 Keepalived_healthcheckers[14394]: Configuration is using : 6366 Bytes Jun 2 15:26:41 LB1 Keepalived_vrrp[14395]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)] Jun 2 15:26:41 LB1 Keepalived_healthcheckers[14394]: Using LinkWatch kernel netlink reflector... Jun 2 15:26:41 LB1 Keepalived_vrrp[14395]: VRRP_Script(chk_haproxy) succeeded Jun 2 15:26:42 LB1 Keepalived_vrrp[14395]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 2 15:26:43 LB1 Keepalived_vrrp[14395]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 2 15:26:43 LB1 Keepalived_vrrp[14395]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 2 15:26:43 LB1 Keepalived_vrrp[14395]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.1 Jun 2 15:26:43 LB1 Keepalived_healthcheckers[14394]: Netlink reflector reports IP 192.168.1.1 added Jun 2 15:26:48 LB1 Keepalived_vrrp[14395]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.1 Jun 2 15:27:19 localhost haproxy[14376]: 192.168.2.101:64188 [02/Jun/2014:15:27:19.465] webfarm webfarm/webA 3/0/2/1/13 200 850 - - CDNN 1/1/0/0/0 0/0 "GET / HTTP/1.1"
从日志中可以看到Keepalived的启动过程，Keepalived主进程启动后启动Healthcheck 和VRRP两个子进程，然后Keepalived的Netlink reflector检查本地IP地址设置，然后读取/etc/keepalived.conf配置文件，最后根据是否为MASTER或BACKUP设置VIP。
在LB2上启动HAProxy和Keepalived，查看/var/log/messages日志
Jun 2 15:48:00 localhost Keepalived[13673]: Starting Keepalived v1.2.7 (02/21,2013) Jun 2 15:48:00 localhost Keepalived[13674]: Starting Healthcheck child process, pid=13675 Jun 2 15:48:00 localhost Keepalived[13674]: Starting VRRP child process, pid=13677 Jun 2 15:48:00 localhost Keepalived_vrrp[13677]: Interface queue is empty Jun 2 15:48:00 localhost Keepalived_vrrp[13677]: Netlink reflector reports IP 192.168.1.104 added Jun 2 15:48:00 localhost Keepalived_vrrp[13677]: Netlink reflector reports IP fe80::a00:27ff:fec2:457 added Jun 2 15:48:00 localhost Keepalived_vrrp[13677]: Registering Kernel netlink reflector Jun 2 15:48:00 localhost Keepalived_vrrp[13677]: Registering Kernel netlink command channel Jun 2 15:48:00 localhost Keepalived_vrrp[13677]: Registering gratuitous ARP shared channel Jun 2 15:48:00 localhost Keepalived_healthcheckers[13675]: Interface queue is empty Jun 2 15:48:00 localhost Keepalived_healthcheckers[13675]: Netlink reflector reports IP 192.168.1.104 added Jun 2 15:48:00 localhost Keepalived_healthcheckers[13675]: Netlink reflector reports IP fe80::a00:27ff:fec2:457 added Jun 2 15:48:00 localhost Keepalived_healthcheckers[13675]: Registering Kernel netlink reflector Jun 2 15:48:00 localhost Keepalived_healthcheckers[13675]: Registering Kernel netlink command channel Jun 2 15:48:20 localhost Keepalived_healthcheckers[13675]: Opening file '/etc/keepalived/keepalived.conf'. Jun 2 15:48:20 localhost Keepalived_healthcheckers[13675]: Configuration is using : 6364 Bytes Jun 2 15:48:20 localhost Keepalived_vrrp[13677]: Opening file '/etc/keepalived/keepalived.conf'. Jun 2 15:48:20 localhost Keepalived_vrrp[13677]: Configuration is using : 38557 Bytes Jun 2 15:48:20 localhost Keepalived_vrrp[13677]: Using LinkWatch kernel netlink reflector... Jun 2 15:48:20 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) Entering BACKUP STATE Jun 2 15:48:20 localhost Keepalived_vrrp[13677]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)] Jun 2 15:48:20 localhost Keepalived_healthcheckers[13675]: Using LinkWatch kernel netlink reflector... Jun 2 15:48:20 localhost Keepalived_vrrp[13677]: VRRP_Script(chk_haproxy) succeeded
查看LB1的IP地址
eth0 Link encap:Ethernet HWaddr 08:00:27:C8:F5:1B inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fec8:f51b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:111413 errors:0 dropped:0 overruns:0 frame:0 TX packets:121677 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:19049418 (18.1 MiB) TX bytes:11302763 (10.7 MiB) eth0:ha Link encap:Ethernet HWaddr 08:00:27:C8:F5:1B inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:713 errors:0 dropped:0 overruns:0 frame:0 TX packets:713 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:134641 (131.4 KiB) TX bytes:134641 (131.4 KiB
可以看到由于LB1上的Keepalived设置为MASTER状态并且HAProxy进程存在，所以LB1上的Keepalived将接管VIP 192.168.1.1。
在这个架构中，HAProxy代理将修改客户端和服务器发送的每一个cookie，确保HAProxy能够访问每个session会话中的所有请求的所有cookie很重要。所以，如果不确定客户端一定不使用keep-alive(HTTP 1.1),则不要轻易去掉 option httpcolse 这个选项。
如果一个请求不包含任何cookie，它将被转发到后端一个可用的服务器。反过来，如果一个名为JSESIONID的cookie出现，HAProxy将会把这个cookie的值修改为JSESIONID=A~xxx这样的形式。
当一个客户端再次请求服务器响应，并在请求中带有JSESSIONID=A~xxx 这样的cookie，LB1会直接将它转发给webA。在这个请求到达webA之前，cookie中的A~ 将被移除。
如果webA不可用，请求将被转发到另外的可用服务器上，响应的cookie将被重置。
在有些情况下，在同一个集群下的服务器，有些处理能力较强，有些处理能力较弱，这种情况，可以通过HAProxy为不同性能的服务器设置不同的权重，处理能力越强，权重越高。
server webA 192.168.1.11:80 cookie A weight 12 check server webB 192.168.1.12:80 cookie B weight 12 check server webC 192.168.1.13:80 cookie C weight 26 check server webD 192.168.1.14:80 cookie D weight 26 check
下面测试一下故障切换
停掉LB1上的HAProxy，查看LB1和LB2的Keepalived相关日志和IP接管情况
LB1上日志如下:
Jun 2 17:45:46 LB1 Keepalived_vrrp[21299]: VRRP_Script(chk_haproxy) failed Jun 2 17:45:47 LB1 Keepalived_vrrp[21299]: VRRP_Instance(VI_1) Received higher prio advert Jun 2 17:45:47 LB1 Keepalived_vrrp[21299]: VRRP_Instance(VI_1) Entering BACKUP STATE Jun 2 17:45:47 LB1 Keepalived_vrrp[21299]: VRRP_Instance(VI_1) removing protocol VIPs. Jun 2 17:45:47 LB1 Keepalived_healthcheckers[21298]: Netlink reflector reports IP 192.168.1.1 removed
可以看到在LB1上停掉HAProxy后，Keepalived检查HAProxy进程不存在，VRRP进入BACKUP状态，Keepalived将移除VIP 192.168.1.1
LB2上日志如下:
Jun 2 17:45:47 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) forcing a new MASTER election Jun 2 17:45:48 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 2 17:45:49 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 2 17:45:49 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 2 17:45:49 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.1 Jun 2 17:45:49 localhost Keepalived_healthcheckers[13675]: Netlink reflector reports IP 192.168.2.21 added Jun 2 17:45:54 localhost Keepalived_vrrp[13677]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.1
LB2上的VRRP进入MASTER状态，接管VIP 192.168.1.1
查看LB2的IP
eth0 Link encap:Ethernet HWaddr 08:00:27:C2:04:57 inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fec2:457/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:130629 errors:0 dropped:0 overruns:0 frame:0 TX packets:109630 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:19059352 (18.1 MiB) TX bytes:8767499 (8.3 MiB) eth0:ha Link encap:Ethernet HWaddr 08:00:27:C2:04:57 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:88 errors:0 dropped:0 overruns:0 frame:0 TX packets:88 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:17488 (17.0 KiB) TX bytes:17488 (17.0 KiB)
然后将LB1上的HAProxy重新启动，查看LB1和LB2的Keepalived日志和IP接管情况
Jun 2 18:13:16 LB1 Keepalived_vrrp[24199]: VRRP_Script(chk_haproxy) succeeded Jun 2 18:13:16 LB1 Keepalived_vrrp[24199]: VRRP_Instance(VI_1) forcing a new MASTER election Jun 2 18:13:17 LB1 Keepalived_vrrp[24199]: VRRP_Instance(VI_1) Transition to MASTER STATE Jun 2 18:13:18 LB1 Keepalived_vrrp[24199]: VRRP_Instance(VI_1) Entering MASTER STATE Jun 2 18:13:18 LB1 Keepalived_vrrp[24199]: VRRP_Instance(VI_1) setting protocol VIPs. Jun 2 18:13:18 LB1 Keepalived_vrrp[24199]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.1 Jun 2 18:13:18 LB1 Keepalived_healthcheckers[24198]: Netlink reflector reports IP 192.168.1.1 added Jun 2 18:13:23 LB1 Keepalived_vrrp[24199]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.1.1
从日志中可以看到当LB1上的HAProxy重新启动后，Keepalived检查到HAProxy进程存在，然后VRRP根据优先级重新选举MASTER，从LB2接管VIP192.168.1.1