前言
本文主要介绍,使用keepalived+lvs
实现负载均衡及高可用功能,后端webserver
使用nginx
,keepalived
使用双主模型。
keepalived基于VRRP实现:
VRRP的工作过程为:
(1) 虚拟路由器中的路由器根据优先级选举出 Master。Master 路由器通过发送免费 ARP 报文,将自己的虚拟 MAC 地址通知给与它连接的设备或者主机,从而承担报文转发任务;
(2) Master 路由器周期性发送 VRRP 报文,以公布其配置信息(优先级等)和工作状况;
(3) 如果 Master 路由器出现故障,虚拟路由器中的 Backup 路由器将根据优先级重新选举新的 Master;
(4) 虚拟路由器状态切换时,Master 路由器由一台设备切换为另外一台设备,新的 Master 路由器只是简单地发送一个携带虚拟路由器的 MAC 地址和虚拟 IP地址信息的免费 ARP 报文,这样就可以更新与它连接的主机或设备中的ARP 相关信息。网络中的主机感不到 Master 路由器已经切换为另外一台设备。
(5) Backup 路由器的优先级高于 Master 路由器时,由 Backup 路由器的工作方式(抢占方式和非抢占方式)决定是否重新选举 Master。
由此可见,为了保证Master路由器和Backup路由器能够协调工作,VRRP需要实现以下功能:
1、Master 路由器的选举;
2、Master 路由器状态的通告;
3、同时,为了提高安全性,VRRP 还提供了认证功能;
keepalived分为抢占式和非抢占式:
抢占示:当一个优先级较大的Server重新上线后,将VIP直接抢回,自己响应请求
非抢占式:优先级较大的Server上线后,看到VIP已经在其他节点工作,虽然自己的优先级高,但也不会强VIP,只会等待VIP所在节点down掉之后,才会重新上线
关于VRRP的内容可以参照VRRP白皮书
LVS-DR模型:
LVS-DR模型,是通过为请求报文重新封装一个MAC首部,源MAC为Director的MAC地址,目标MAC是被选中后端RS的MAC地址,IP首部不会变。
(1)确保前端路由器将目标IP为VIP的请求报文发往Director:
解决方案:
在路由器上静态绑定VIP和Director的MAC地址;
禁止RS响应VIP的ARP请求,禁止RS的VIP进行通告
(a) arptables
b) 修改RS的内核参数,并把VIP绑定lo的别名上;
arp_ignore, arp_announce
显然,第一种方案不可行,我们不可能去运营商修改,所以只能使用后者,通过修改后端RS,关闭arp请求,来实现
(2)RS的RIP可以使用私有地址,也可以使用公网地址;
(3) RS跟Director必须在同一物理网络;RS的网关必须不能指向DIP;
(4) 请求报文必须由Directory调度,但响应报文必须不能经由Director;
(5) 不支持端口映射;
(6) RS可以使用大多的OS;
环境介绍
Director1 | Director2 | WebServer_Nginx1 | WebServer_Nginx2 |
DIP:192.168.0.108 | DIP:192.168.0.109 | RIP:192.168.0.10 | RIP:192.168.020 |
VIP:192.168.0.111:M VIP:192.168.0.222:B |
VIP:192.168.0.111:B VIP:192.168.0.222:M |
VIP:192.168.0.111 VIP:192.168.0.222 |
VIP:192.168.0.111 VIP:192.168.0.222 |
M:表示为Master B:表示为BACKUP
实验拓扑图
配置前端Director
Director1配置
安装程序包:
# yum install keepalived ipvsadm -y
修改配置文件:
! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_mcast_group4 224.0.4.223 } vrrp_instance VI_1 { state MASTER interface eno16777736 virtual_router_id 108 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.0.111 } } virtual_server 192.168.0.111 80 { delay_loop 6 lb_algo rr lb_kind DR nat_mask 255.255.255.0 protocol TCP real_server 192.168.0.10 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.20 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } vrrp_instance VI_2 { state BACKUP interface eno16777736 virtual_router_id 109 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.0.222 } } virtual_server 192.168.0.222 80 { delay_loop 6 lb_algo rr lb_kind DR nat_mask 255.255.255.0 protocol TCP real_server 192.168.0.10 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.20 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } }
Director2配置
安装程序包:
# yum install keepalived ipvsadm -y
修改配置文件:
! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_mcast_group4 224.0.4.223 } vrrp_instance VI_1 { state BACKUP interface eno16777736 virtual_router_id 108 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.0.111 } virtual_server 192.168.0.111 80 { delay_loop 6 lb_algo rr lb_kind DR nat_mask 255.255.255.0 protocol TCP real_server 192.168.0.10 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.20 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } vrrp_instance VI_2 { state MASTER interface eno16777736 virtual_router_id 109 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.0.222 } } virtual_server 192.168.0.222 80 { delay_loop 6 lb_algo rr lb_kind DR nat_mask 255.255.255.0 protocol TCP real_server 192.168.0.10 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.0.20 80 { weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } }
# 此处暂时不启动服务,在配置好nginx之后启动服务
配置后端WebServer
禁止RS响应arp请求和通告脚本
#!/bin/bash vip1=192.168.0.111 vip2=192.168.0.222 interface1="lo:0" interface2="lo:1" case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $interface1 $vip1 broadcast $vip1 netmask 255.255.255.255 up route add -host $vip1 dev $interface1 ifconfig $interface2 $vip2 broadcast $vip2 netmask 255.255.255.255 up route add -host $vip2 dev $interface2 ;; stop) echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $interface1 down ifconfig $interface2 down ;; *) echo "Usage: `basename $0` {start|stop|status}" exit 1 esac
RS1执行脚本并查看
# bash arp.sh start # ip add show 1: lo:mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 192.168.0.111/32 brd 192.168.0.111 scope global lo:0 valid_lft forever preferred_lft forever inet 192.168.0.222/32 brd 192.168.0.222 scope global lo:1 valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:a7:a7:33 brd ff:ff:ff:ff:ff:ff inet 192.168.0.10/24 brd 192.168.0.255 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fea7:a733/64 scope link valid_lft forever preferred_lft forever # cat /proc/sys/net/ipv4/conf/all/arp_ignore 1 # cat /proc/sys/net/ipv4/conf/all/arp_announce 2
RS2执行脚本并查看
# bash arp.sh start # ip add show 1: lo:mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet 192.168.0.111/32 brd 192.168.0.111 scope global lo:0 valid_lft forever preferred_lft forever inet 192.168.0.222/32 brd 192.168.0.222 scope global lo:1 valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:ec:85:42 brd ff:ff:ff:ff:ff:ff inet 192.168.0.20/24 brd 192.168.0.255 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feec:8542/64 scope link valid_lft forever preferred_lft forever # cat /proc/sys/net/ipv4/conf/all/arp_ignore 1 # cat /proc/sys/net/ipv4/conf/all/arp_announce 2
配置Nginx,提供测试页面
WebServer1
# 安装Nginx # yum install nginx -y # rpm -q nginx nginx-1.6.3-8.el7.x86_64 # vim /usr/share/nginx/html/index/htmlServer1
# nginx #启动nginx
WebServer2
# 安装Nginx # yum install nginx -y # rpm -q nginx nginx-1.6.3-8.el7.x86_64 # vim /usr/share/nginx/html/index/htmlServer2
# nginx #启动nginx
启动前端调度器并测试
Director1:启动服务
# systemctl start keepalived
Director2:启动服务
# systemctl start keepalived
Director1:查看虚拟ip及RS是否添加
# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.111:80 rr -> 192.168.0.10:80 Route 1 0 0 -> 192.168.0.20:80 Route 1 0 0 TCP 192.168.0.222:80 rr -> 192.168.0.10:80 Route 1 0 0 -> 192.168.0.20:80 Route 1 0 0 # ip addr show 1: lo:mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:43:ad:7b brd ff:ff:ff:ff:ff:ff inet 192.168.0.108/24 brd 192.168.0.255 scope global eno16777736 valid_lft forever preferred_lft forever inet 192.168.0.111/32 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fe43:ad7b/64 scope link valid_lft forever preferred_lft forever # 此处可以看到VIP:192.168.0.111
Director2:查看虚拟ip及RS是否添加
# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.0.111:80 rr -> 192.168.0.10:80 Route 1 0 0 -> 192.168.0.20:80 Route 1 0 0 TCP 192.168.0.222:80 rr -> 192.168.0.10:80 Route 1 0 0 -> 192.168.0.20:80 Route 1 0 0 # ip addr show 1: lo:mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno16777736: mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:dd:c1:9c brd ff:ff:ff:ff:ff:ff inet 192.168.0.109/24 brd 192.168.0.255 scope global eno16777736 valid_lft forever preferred_lft forever inet 192.168.0.222/32 scope global eno16777736 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fedd:c19c/64 scope link valid_lft forever preferred_lft forever # 此处可以看到VIP:192.168.0.222
正常测试:
关闭后端Webserver1,再进行测试
步骤:
1、WebServer1关闭Nginx
2、测试机请求,可以看到 只能请求道Server2
3、在调度器查看,可以看到,其中一台webserver已经被移除,说明健康状态检查是有效的
关闭Director2,再进行测试
步骤:
1、WebServer1启动ginx
2、关闭Director2
3、在调度器查看,可以看到,此时有两个VIP
4、在测试测试,可以正常访问
文章就写到这里了,后续还会写一篇关于LNAMMP的文章,文章中会介绍session等信息。
作者:Ace
QQ1257465991
Linux运维攻城狮一只
Q/A:如有问题请慷慨提出