keepalived应用案例及其ipvs-dr模型原理解释
【LVS--的调度方法】
分类：静态方法：只根据调度方法本身来调度，不关心RS的负载情况来调度。
动态方法：根据调度方法本身和RS的负载状况来调度。
【 静态方法】
RR 算法 : ROUND ROBIN,轮询
WRR算法:加权轮询
SH: sourcehashing源地址的hash绑定。
实现的方式：
Session:绑定方式，无冗余能力丢失客户端要重来。实现方式;负载均衡器内部有一个会话表。他可以记录一个会话信息哈希编码记录（CIP,RS）。
session服务器，把session放在后端服务器，多个服务器共用一个后端服务器，这样就会导致无冗余，因此建议使用高可用。
session复制的方式：这是把后端对台主机，实现DLTE会话复制，多台主机都有会话的COOKIE，有冗余，但是适合的节点比较少的情况。
DH算法： destination hashing;
这种情况是针对我们是客户端的情况下，我们期望可以实现连接追踪的功能。因此就必须要让我们之前从一台防火墙出去的，报文也从哪台防火墙回来。因此调度时只要director给调度到相应的防火墙，回来时自然还是那个防火墙回来。源IP是防火墙，然后再由director调度回相应的主机即可。
【动态方法】
LC: leastconnection 最少连接接数
Active 后端主机的活动连接数
Inactive:后端主机的非活动连接数
Overhead=actconn*256+lnactionn
活动连接占用的资源多，因此*256。那台后端主机得值小，就挑中那一台。
WLC: 加权最少连接
Overhead=(actconn*256+lnactionn)/weighted 结果小的会被挑中
如果是相同的值，自上而下执行。
SED：short exppection delay最短期望延迟。最小的的被选中
存在问题：这种是会导致权重大的频繁被挑中，而当连接如果较少时，导致权重小的无法分担工作。
NQ:Nerver Queue永不排队
在执行之前安装权重从大到小，先一次分一个。之后在执行SED模式。
LBLC：动态DH算法；
LBLCR: 类似于session复制的一种工作机制。
众多的算法中，系统默认的是使用WLC算法。
【LVS-DR模型的实现】
【目的】实现后端服务器负载均衡。
【特点】这是一种基于MAC地址的转换方式。
当一个客户端发起一个请求时，为了防止不经过负载均衡器，我们通过arptables,静态绑定的方式，将前端路由器强行指向我们的director。由于这是一种基于MAC地址的转换方式，因此director和RS之间必须是同一个网络内。然后由负载均衡器，直接把报文直接丢给交换机，广播对应真实主机的MAC地址。当RS打开报文时，发现目标IP是VIP，那么就会转发回给VIP，这显然不是我们希望看到的，因此我们需要在每一个RS上也配置上VIP（这样就是多台RS和负载均衡器都有VIP了，那么如何解决一个网段相同的IP冲突呢？）其实我们是把真实主机的VIP配置在lo:0上的。然后通过修改内核参数arp_ignore,arp_announce参数，限制本机被请求和自己通告的方式。这样我们就可以RS打开报文时就看到是发送给自己的报文，并且不冲突director接受外部的报文。
RS的响应：RS在响应时根据之前的IP首部（CIP,VIP），将报文返回（VIP,CIP）。因为CIP是将报文发送给VIP的，因此回应时，想让CIP能够识别，就必须使用VIP来作为源IP地址。但是我们的VIP是配置在lo:0上的无法外部通信，因此我们还需要给予真实主机设置一个内部的路由转换，强制VIP通过lo:0-->eth0转发出去。
####################keepalive的几个常见应用的介绍##################################################
[root@localhost ~ ]#yum install keepalived [root@localhost ~ ]# rpm -ql keepalived ####查看生成的配置文件 /etc/keepalived /etc/keepalived/keepalived.conf ####配置文件 /etc/rc.d/init.d/keepalived ####服务脚本 /etc/sysconfig/keepalived #####定义脚本的选项的配置文件 /usr/sbin/keepalived #####二进制程序
查看配置文件使用帮助的命令是：
[root@node3 ~ ]# man keepalived.conf
配置文件：
分成三段：
全局段：
Global definitions
global_defs
# Block id
{
notification_email # To: ####定义提示信息发给的主机
{
[email protected]
...
}
notification_email_from [email protected] //定义由谁发
smtp_server 127.0.0.1 # IP
smtp_connect_timeout 30 # integer, seconds
router_id my_hostname # string identifying the machine,
# (doesn’thave to be hostname).
vrrp_mcast_group4 224.0.0.18 #optional, default 224.0.0.18
vrrp_mcast_group6 ff02::12 # optional, default ff02::12
enable_traps # enable SNMP traps
}
静态路由的定义格式
static_ipaddress
{
192.168.1.1/24 dev eth0 scope global
...
}
static_routes
{
192.168.2.0/24 via 192.168.1.100 dev
eth0
...
}
#####################################【对haproxy高可用】#############################################
[root@node3
keepalived ]# vim keepalived.conf
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@node1 //通知的邮件
6 root@node3
7 }
8 notification_email_from [email protected] ####定义发送的邮件
9 smtp_connect_timeout 3
10 smtp_server 127.0.0.1
11 router_id LVS_DEVEL
12 }
13
14 vrrp_script chk_haproxy { #####定义检测脚本
15 script "killall -0 haproxy" ####成功就是在运行，这时不管它。
16 interval 1 ####检测的间隔
17 weight -2 ####修改的权重
18 }
19
20 vrrp_script chk_mantaince_down { ###定义手动控制的脚本
21 script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0" ####如果有down文件，权重.
22 interval 1
23 weight -5
24 }
25
26 vrrp_instance VI_1 { ####定义实例
27 interface eth0
28 state MASTER ####主节点
29 priority 100 ####优先级
30 virtual_router_id 110 ####虚拟路由的ID
31 garp_master_delay 1
32
33 authentication { ####通信认证
34 auth_type PASS #####认证类型
35 auth_pass password #####密码
36 }
37 track_interface { ###追踪的网卡
38 eth0
39 }
40 virtual_ipaddress { ###虚拟IP
41 172.16.11.1/16 dev eth0 label eth0:0 ####虚拟IP/掩码 使用的设备 应用在别名上的标签
42 }
43 track_script {
44 chk_haproxy ###定义haproxy
45 chk_mantaince_down
46 }
47
48
49 notify_master"/etc/keepalived/notify.sh master" // notifiy_master是内置参数，表示如果当前状态是master
50 notify_backup"/etc/keepalived/notify.sh backup" //notify_backup 也是内置参数，表示如果当前状态是backup
51 notify_fault "/etc/keepalived/notify.sh fault"
52 }
[root@director1keepalived]# scp keepalived.conf director2:/etc/keepalived/ [root@director2keepalived ]# vim keepalived.conf State BACKUP priority 99
########################################提供外部的服务本##########################################
[root@node3 keepalived ]# vim notify.sh
1 #!/bin/bash
2 #
3 #
4
5 vip=172.16.11.1
6 contact='root@localhost'
7 notify() { //这是指定的邮件格式
8 mailsubject="`hostname` to be $1:$vip floating" //主题：主机名变成了主还是从，vip也变了。
9 mailbody="`date '+%F %H:%M:%S'`: vrrptransition, `hostname` changed to be $1" //时间，…正在转变
10 echo $mailbody | mail -s"$mailsubject" $contact //定义发送邮件的语句
11 }
12
13 case "$1" in
14 master)
15 notify master
16 /etc/rc.d/init.d/haproxy start
17 exit 0
18 ;;
19 backup)
20 notify backup
21 /etc/rc.d/init.d/haproxy stop
22 exit 0
23 ;;
24 fault)
25 notify fault
26 /etc/rc.d/init.d/haproxy stop
27 exit 0
28 ;;
29 *)
30 echo 'Usage: `basename $0` {master|backup|fault}'
31 exit 1
32 ;;
33 esac
34chmod a+x notify.sh //一定要给脚本文件执行权限
查看我们的邮件
[root@node3keepalived ]# mail Heirloom Mailversion 12.4 7/29/08. Type ? for help. "/var/spool/mail/root":2 messages 2 new >N 1 root Wed Jan 14 21:59 18/686 "node3 to be master: 172.16.11.1 floating" N 2root Wed Jan 14 21:59 18/686 "node3 to be backup: 172.16.11.1 floating" &
[root@node3keepalived ]# mail Heirloom Mail version 12.4 7/29/08. Type ? for help. "/var/spool/mail/root":2 messages 2 new >N 1 root Wed Jan 14 21:59 18/686 "node3 to be master: 172.16.11.1 floating" N 2 root Wed Jan 14 21:59 18/686 "node3 to be backup: 172.16.11.1 floating" &
##################################Nginx双主模式高可用####################################
节点一配置文件
[root@node1
keepalived ]# vim keepalived.conf
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@node1
6 root@node3
7 }
8 notification_email_from root@node3
9 smtp_connect_timeout 3
10 smtp_server 127.0.0.1
11 router_id LVS_DEVEL
12 }
13
14 vrrp_script chk_nginx {
15 script "killall -0 nginx"
16 interval 1
17 weight -5
18 }
19
20 vrrp_script chk_mantaince_down {
21 script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0"
22 interval 1
23 weight -5
24 }
25
26 vrrp_instance VI_1 {
27 interface eth0
28 state BACKUP
29 priority 98
30 virtual_router_id 110
31 garp_master_delay 1
32
33 authentication {
34 auth_type PASS
35 auth_pass password
36 }
37 track_interface {
38 eth0
39 }
40 virtual_ipaddress {
41 172.16.11.11/16 dev eth0 label eth0:1
42 }
43 track_script {
44 chk_mantaince_down
45 chk_nginx
46 }
47 notify_master "/etc/keepalived/notify.sh master"
48 notify_backup "/etc/keepalived/notify.sh backup"
49 notify_fault "/etc/keepalived/notify.sh fault"
50 }
51 vrrp_instance VI_2 {
52 interface eth0
53 state MASTER
54 priority 100
55 virtual_router_id 111
56 garp_master_delay 1 57
58 authentication {
59 auth_type PASS
60 auth_pass adminadmin
61 }
62 virtual_ipaddress {
63 172.16.11.12/16 dev eth0 label eth0:1
64 }
65 track_script {
66 chk_nginx
67 chk_mantaince_down
68 }
69
70
71 notify_master "/etc/keepalived/notify.sh master"
72 notify_backup "/etc/keepalived/notify.sh backup"
73 notify_fault "/etc/keepalived/notify.sh fault"
74 }
节点二配置文件
[root@node3keepalived ]# vim keepalived.conf
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email {
5 root@node1
6 root@node3
7 }
8 notification_email_from root@node3
9 smtp_connect_timeout 3
10 smtp_server 127.0.0.1
11 router_id LVS_DEVEL
12 }
13
14 vrrp_script chk_nginx {
15 script "killall -0 nginx"
16 interval 1
17 weight -5
18 }
19
20 vrrp_script chk_mantaince_down {
21 script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0"
22 interval 1
23 weight -5
24 }
25
26 vrrp_instance VI_1 {
27 interface eth0
28 state MASTER
29 priority 100
30 virtual_router_id 110
31 garp_master_delay 1
32
33 authentication {
34 auth_type PASS
35 auth_pass password
36 }
37 track_interface {
38 eth0
39 }
40 virtual_ipaddress {
41 172.16.11.11/16 dev eth0 label eth0:1
42 }
43 track_script {
44 chk_mantaince_down
45 chk_nginx
46 }
47 notify_master "/etc/keepalived/notify.sh master"
48 notify_backup "/etc/keepalived/notify.sh backup"
49 notify_fault "/etc/keepalived/notify.sh fault"
50 }
51 vrrp_instance VI_2 {
52 interface eth0
53 state BACKUP
54 priority 98
55 virtual_router_id 111
56 garp_master_delay 1
57
58 authentication {
59 auth_type PASS
60 auth_pass adminadmin
61 }
62 virtual_ipaddress {
63 172.16.11.12/16 dev eth0 label eth0:1
64 }
65 track_script {
66 chk_nginx
67 chk_mantaince_down
68 }
69
70
71 notify_master "/etc/keepalived/notify.sh master"
72 notify_backup "/etc/keepalived/notify.sh backup"
73 notify_fault "/etc/keepalived/notify.sh fault"
74 }
##################################外部控制添加脚本 #################################################
[root@node1
keepalived ]# vim notify.sh
1 #!/bin/bash
2 #
3 #
4
5 vip=172.16.11.1
6 contact='root@localhost'
7 notify() {
8 mailsubject="`hostname` to be $1: $vip floating"
9 mailbody="`date '+%F %H:%M:%S'`: vrrp transition, `hostname`changed to be $1"
10 echo $mailbody | mail -s "$mailsubject" $contact
11 }
12
13 case "$1" in
14 master)
15 notify master
16
17 exit 0
18 ;;
19 backup)
20 notify backup
21
22 exit 0
23 ;;
24 fault)
25 notify fault
26 /etc/rc.d/init.d/nginx stop
27 exit 0
28 ;;
29 *)
30 echo 'Usage: `basename $0` {master|backup|fault}'
31 exit 1
32 ;;
33 esac
###############################IPVS+KEEPALIVE实验准备如下##############################
Master 172.16.11.1
Backup 172.16.249.122
Node1: 172.16.249.106
Node2:172.16.249.141
vip： 172.16.11.11
##########################前端director的高可用准备####################################### #######################主机名解析一致################################################### [root@director1 ~]#vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.16.0.1 server.magelinux.com server 172.16.11.1 director1 172.16.249.122 director2 [root@director1 ~]#scp /etc/hosts 172.16.249.122:/etc/hosts ##########################时间保持同步################################################### [root@director2 ~ ]#crontab -e */3 * * * * /usr/sbin/ntpdate 172.16.0.1 &> /dev/null [root@director1 ~]# crontab -e */3 * * * * /usr/sbin/ntpdate 172.16.0.1 &> /dev/null ##########################基于主机间无密钥通信########################################### [root@director2 ~ ]# ssh-keygen -P '' [root@director2 ~ ]# ssh-copy-id -i ~/.ssh/id_rsa.pub director1 [root@director1 ~]# ssh-keygen -P '' [root@director1 ~]#ssh-copy-id -i ~/.ssh/id_rsa.pub director2
######################配置前端使用ipvs以及keepalived高可用################################
#######################添加ipvs脚本，便于写规则##########################################
[root@director1 ~]#yum -y install keepalived ipvsadm
[root@director2 ~]# yum -y install keepalived ipvsadm
#####################修改节点一配置文件#################################################
[root@director1keepalived]# cp keepalived.conf keepalived.conf.bak
[root@director1 keepalived ]# vim keepalived.conf
1 ! Configuration File for keepalived
2
3 global_defs {
4 notification_email { #####接受邮件地址
5 root@director1
6 root@director2
7 }
8 notification_email_from [email protected] 发送邮件的
9 smtp_server 127.0.0.1
10 smtp_connect_timeout 30
11 router_id LVS_DEVEL
12 }
13
14 vrrp_instance VI_1 { ###定义实例
15 state MASTER ###主节点
16 interface eth0 ###网卡接口
17 virtual_router_id 101 ###路由ID主备必须一致
18 priority 100 ###优先级，主高于备
19 advert_int 1
20 authentication { ###通信认证
21 auth_type PASS
22 auth_pass password ###密码使用随机字符串就可以，但是主备保持一致
23 }
24 virtual_ipaddress {
25 172.16.11.11/16 brd 172.16.11.11 dev eth0 label eth0:0 ##虚拟IP
26 }
27 }
28
29 virtual_server 172.16.11.11 { #####定义虚拟主机
30 delay_loop 6
31 lb_algo rr ###rr调度算法
32 lb_kind DR ###lvs的DR模型
33 nat_mask 255.255.0.0
34 persistence_timeout 50
35 protocol TCP
36
37 real_server 172.16.249.106 80 { ###后端真实主机
38 weight 1
39 HTTP_GET { ###HTTP协议检测
40 url {
41 path / ###检测请求路径
42 status_code 200 ###期望回应的状态码
43 }
44 connect_timeout 3 ###超时时长
45 nb_get_retry 3 ###重试时长
46 delay_before_retry 3 ###延迟时长
47 }
48 }
49
50
51 real_server 172.16.249.141 80 {
52 weight 1
53 HTTP_GET {
54 url {
55 path /
56 status_code 200
57 }
58 connect_timeout 3
59 nb_get_retry 3
60 delay_before_retry 3
61 }
62 }
63 }
#############################复制到第二个节点，修改如下选项##########################
[root@director2 keepalived ]# mv keepalived.conf keepalived.conf.bak
#####节点二做备份
[root@director1keepalived]# scp keepalived.conf director2:/etc/keepalived/
[root@director2keepalived ]# vim keepalived.conf
State BACKUP
priority
99
###############################确保开机自动启动#####################################
[root@director1 keepalived]# chkconfig keepalived on ;ssh director2 "chkconfig keepalived on";
[root@director1 keepalived]# chkconfig --list keepalived;ssh director2 "chkconfig --list
keepalived"
keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off
keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off
################################同时启动两个节点，观察VIP ###########################
[root@director1
keepalived]# service keepalived
start;ssh director2 "service keepalived start
###############################查看VIP，以及日志主备健康状态##########################
[root@director1
keepalived]# ifconfig
[root@director1keepalived]# tail /var/log/messages Jan 21 11:53:11director1 Keepalived_vrrp[5698]: Using LinkWatch kernel netlink reflector... Jan 21 11:53:11director1 Keepalived_healthcheckers[5697]: Using LinkWatch kernel netlinkreflector... Jan 21 11:53:11director1 Keepalived_vrrp[5698]: VRRP sockpool: [ifindex(2), proto(112),unicast(0), fd(10,11)] Jan 21 11:53:11director1 Keepalived_healthcheckers[5697]: Activating healthchecker for service[172.16.249.106]:80 Jan 21 11:53:12director1 Keepalived_vrrp[5698]: VRRP_Instance(VI_1) Transition to MASTER STATE Jan 21 11:53:13director1 Keepalived_vrrp[5698]: VRRP_Instance(VI_1) Entering MASTER STATE Jan 21 11:53:13director1 Keepalived_vrrp[5698]: VRRP_Instance(VI_1) setting protocol VIPs. Jan 21 11:53:13director1 Keepalived_vrrp[5698]: VRRP_Instance(VI_1) Sending gratuitous ARPs oneth0 for 172.16.11.11 Jan 21 11:53:13director1 Keepalived_healthcheckers[5697]: Netlink reflector reports IP172.16.11.11 added Jan 21 11:53:18director1 Keepalived_vrrp[5698]: VRRP_Instance(VI_1) Sending gratuitous ARPs oneth0 for 172.16.11.11 ########################配置主备节点之间的路由间转发########################################### [root@director1 keepalived]# echo 1 > /proc/sys/net/ipv4/ip_forward;ssh director2 "echo1 > /proc/sys/net/ipv4/ip_forward"; [root@director1 keepalived]# echo 1 > /proc/sys/net/ipv4/ip_forward;ssh director2 "echo1 > /proc/sys/net/ipv4/ip_forward"; 后端节点 vim /var/www/html/index.html www.stu11.1.com vim /var/www/html/index.html www.stu11.2.com
#####################为两个后端主机配置VIP和内核参数######################################### [root@node2 ~ ]# vim rs.sh 1 #!/bin/bash 2 # 3 vip=172.16.11.11 4 case $1 in 5 start) 6 echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore 7 echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore 8 echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 9 echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce 10 11 ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up 12 route add -host $vip dev lo:0 13 ;; 14 stop) 15 echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore 16 echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore 17 echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce 18 echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce 19 20 ifconfig lo:0 down 21 route del -host $vip dev lo:0 22 ;; 23 esac [root@node2 ~ ]#chmod +x rs.sh [root@node2 ~ ]#bash -x rs.sh start [root@node2 ~ ]#ifconfig lo:0 Link encap:Local Loopback inet addr:172.16.11.11 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:65536 Metric:1
################################查看MASTER节点的IPVS规则################################ [root@director1keepalived]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) ProtLocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.16.11.11:0 rr persistent 50 -> 172.16.249.106:80 Route 1 0 0 -> 172.16.249.141:80 Route 1 0 0
##################################测试结果##########################################################
至此IPVS+KEEPALIVED高可用全部结束！！！！！！！！