利用Keepalived实现lvs-dr的简单示例：
一共使用5台服务
a.时间服务器、调度器(Director1)：
CentOS 7.2A
b.调度器(Director2)
CentOS 7.2B
c.后端服务器(Real Server1)：
CentOS 7.2C
d.后端服务器(Real Server1)：
CentOS 7.2D
f.进行测试的客户端：
CentOS 7.5A
1.修改对应的主机名；为了方便观察
Director(CentOS 7.2A)：
~]# hostnamectl set-hostname drct1
Director(CentOS 7.2B)：
~]# hostnamectl set-hostname drct2
Real Server1(CentOS 7.2C)：
~]# hostnamectl set-hostname rs1
Real Server2(CentOS 7.2D)：
~]# hostnamectl set-hostname rs2
2.对比无keepalived的lvs-dr模型，无需在调度器上的DIP配置对应的VIP，因为后续会在keepalived的主配置文件中进行VIP的配置；
3.在rs1、rs2上利用脚本进行配置
#!/bin/bash
#
VIP=172.16.72.254
MASK=255.255.255.255
case $1 in
setup)
#调整ARP相关的内核参数：
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
#在lo的标签接口上配置VIP
ifconfig lo:0 $VIP netmask $MASK broadcast $VIP up
#为了能够使响应报文从lo:0标签接口向外封装发送数据，需要指定一条特殊的静态路由：
route add -host $VIP dev lo:0
;;
delete)
ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage: $(basename $0) { setup | delete }"
;;
esac
4.关闭防火墙、SELinux，确保调度器装上ipvsadm、keepalived、mailx；http服务所有服务器都要装上；
5.在调度器上的配置：
drct1：
~]# cd /etc/keepalived
keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct1
vrrp_mcast_group4 224.0.100.18
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 36
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0
#只在lvs-dr模型中才这样写
}
}
virtual_server 172.16.72.254 80 {
delay_loop 1
lb_algo wrr
lb_kind DR
protocol TCP
real_server 172.16.72.3 80 {
weight 1
HTTP_GET {
url {
path /index.html
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 172.16.72.4 80 {
weight 3
HTTP_GET {
url {
path /index.html
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
drct2：
~]# cd /etc/keepalived
keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct2
vrrp_mcast_group4 224.0.100.18
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 36
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0
#只在lvs-dr模型中才这样写
}
}
virtual_server 172.16.72.254 80 {
delay_loop 3
lb_algo wrr
lb_kind DR
protocol TCP
real_server 172.16.72.3 80 {
weight 1
HTTP_GET {
url {
path /index.html
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 172.16.72.4 80 {
weight 3
HTTP_GET {
url {
path /index.html
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
6.rs1、rs2上都开启http服务，并配置好相关的文件，令访问rs1、rs2时显示的内容不相同，这样能够方便观察keepalived的VRRP；实际生产环境中是使后端服务器显示的内容都相同；
7.drct1、drct2上都开启keepalived服务，利用ipvsadm查看集群配置情况：
keepalived]# systemctl start keepalived.service
keepalived]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.72.254:80 wrr
-> 172.16.72.3:80 Route 1 0 0
-> 172.16.72.4:80 Route 3 0 0
You have new mail in /var/spool/mail/root
8.客户端测试
当两台调度服务器都正常工作、后端提供的http服务也正常时；
~]# for i in {1..10} ; do curl http://172.16.72.254 ; done
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
this is CentOS 7.2C for /var/www/html/
关闭其中一台调度服务器的keepalived、后端提供的http服务也正常时：
~]# for i in {1..10} ; do curl http://172.16.72.254 ; done
This is CentOS 7.2D for /var/www/html
this is CentOS 7.2C for /var/www/html/
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
this is CentOS 7.2C for /var/www/html/
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
this is CentOS 7.2C for /var/www/html/
当两台调度服务器都正常工作、关闭rs1的http服务：
调度器上：
keepalived]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.72.254:80 wrr
-> 172.16.72.4:80 Route 3 0 0
客户端访问：
~]# for i in {1..10} ; do curl http://172.16.72.254 ; done
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
This is CentOS 7.2D for /var/www/html
//客户端测试结果显示：
1.调度器能够实现高可用
2.后端RS也能够实现负载均衡其他较为有用的配置命令：
一.在LVS CONFIGURATION配置段的virtual server的上下文添加：
sorry_server
示例：
sorry_server 127.0.0.1 80
作用：当后端RS都无法使用时，前段的调度器给客户端返回一个友好的错误页面；
1) 把RS的http服务都关闭，同时开启调度器的http服务，配置sorry_server显示的web站点；重启keepalived服务；若此前keepalived主配置文件已添加sorry_server的配置，无需重启服务；
2) 此时两台调度器都应该处于正常的keepalived状态，
在客户端测试
~]# for i in {1..10} ; do curl http://172.16.72.254 ; done
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
sorry page for 1
在调度器(drct1)上：
keepalived]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.72.254:80 wrr
-> 127.0.0.1:80 Route 1 0 10
3) 关闭调度器1(drct1)的keepalived，
在客户端测试：
~]# for i in {1..10} ; do curl http://172.16.72.254 ; done
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
sorry page for 2222222222222222222
//利用keepalived的sorry_server的配置也可实现
二.在VRRPD CONFIGURATION配置段的上下文中添加：
notify_master |
notify_backup |
notify_fault |
//设置当当前服务器状态发生变化时，所触发的脚本文件的路径及相关参数；
状态变更通知脚本示例：/etc/keepalived/notify.sh
#!/bin/bash
#
CONTACT='root@localhost'
notify() {
MAILSUBJECT="$(hostname) changed state to $1, VIP reflact."
#发送电子邮件的主题
MAILBODY="$(date +'%F %T'): VRRP STATUS CHANGED, $(hostname) change state to $1"
#邮件的正文
echo "$MAILBODY" | mail -s "$MAILSUBJECT" "$CONTACT"
}
#case $1的$1指的是执行脚本时后面给定的参数
#notify()函数的$1指的是case引用notify()函数执行命令时notify后面的参数
case $1 in
master)
notify MASTER
;;
backup)
notify BACKUP
;;
fault)
notify FAULT
;;
*)
echo "Usage: $(basename $0) { master | backup | fault}"
;;
esac
示例：两台调度器都要完成相同的操作
keepalived]# vim notify.sh
keepalived]# chmod +x notify.sh
keepalived]# vim keepalived.conf
VRRP的配置段：
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 36
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
测试：
1) 两台调度器重启keepalived服务；若此前keepalived主配置文件已添加notify的配置，无需重启服务；
2) 两台调度器都重启后，利用mail查看
查看调度器1(drct1)的邮箱：
keepalived]# mail
>N 11 root Fri May 3 23:58 18/698 "drct1 changed state to MASTER, VIP reflact."
//调度器1(drct1)进入MASTER
查看调度器2(drct2)的邮箱：
keepalived]# mail
>N 11 root Fri May 3 23:58 18/698 "drct2 changed state to BACKUP, VIP reflact."
//调度器2(drct2)进入BACKUP
3) 停止调度器1(drct1)的keepalived，利用mail查看
查看调度器2(drct2)的邮箱：
keepalived]# mail
>N 12 root Fri May 3 23:58 18/698 "drct2 changed state to MASTER, VIP reflact."
//因为停止了调度器1(drct1)，所以调度器2(drct2)进入MASTER
三.利用外部辅助脚本，完成MASTER/BACKUP的手动切换；
keepalived允许调用外部辅助脚本，完成对于资源的监控，并根据健康的返回状态结果实现动态调整；
如下指令应该使用在vrrp实例定义之前
vrrp_script {
script ""
interval
weight -
}
如下述指令使用的vrrp示例上下文之中；
track_script {
SCRIPT_NAME
}
示例：在调度器1(drct1)上实现手动的MASTER/BACKUP切换
keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct1
vrrp_mcast_group4 224.0.100.18
}
vrrp_script check_state {
#script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
#用上述配置keepalived启动时会显示：Unable to access script `[[`；此条vrrp_script不可用，无法被执行启动；改为如下；
script "/etc/keepalived/keep_down.sh"
interval 1
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 36
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.254/32 dev eno16777736 brd 172.16.72.254 label eno16777736:0
}
track_script {
check_state
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
keepalived]# vim keep_down.sh
#!/bin/bash
#
if [ -f /etc/keepalived/down ];then
weight -20
fi
keepalived]# chmod +x keep_down.sh
keepalived]# touch down
//调度器1(drct1)的优先级从100降到80(weight -20)；此时调度器2(drct2)的优先级为90，
//调度器1(drct1)为BACKUP；调度器2(drct2)为MASTER；可用mail查看状态转换的信息；
keepalived]# rm -f down
//调度器1(drct1)的优先级从80回到100；此时调度器2(drct2)的优先级为90，调度器2(drct2)的优先级为90；
//调度器1(drct1)为MASTER；调度器2(drct2)为BACKUP；可用mail查看状态转换的信息；
四.利用外部辅助脚本，高可用其他的系统服务：
用两台调度器上的进行http服务的高可用实现
抢占式(默认)下的高可用http服务实现：
1.编辑两台调度器的keepalived的主配置文件
调度器1(drct1)：
keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct1
vrrp_mcast_group4 224.0.100.18
}
vrrp_script check_state {
#script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
#用上述配置keepalived启动时会显示：Unable to access script `[[`；此条vrrp_script不可用，无法被执行启动；改为如下；
script "/etc/keepalived/keep_down.sh"
interval 1
}
vrrp_script check_httpd {
script "/usr/bin/killall -0 httpd && exit 0 || exit 1"
interval 1
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 36
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.100
}
track_script {
check_state
check_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
调度器2(drct2)：
keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct2
vrrp_mcast_group4 224.0.100.18
}
vrrp_script check_state {
#script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
#用上述配置keepalived启动时会显示：Unable to access script `[[`；此条vrrp_script不可用，无法被执行启动；改为如下；
script "/etc/keepalived/keep_down.sh"
interval 1
}
vrrp_script check_httpd {
script "/usr/bin/killall -0 httpd && exit 0 || exit 1"
interval 1
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 36
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.100
}
track_script {
check_state
check_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
//两台调度器的主配置文件中主要是加入了vrrp_script check_httpd和track_script这两个配置
3.配置两个调度器的http服务页面
4.测试
a.先开启调度器2(drct2)的keepalived和http服务：
keepalived]# systemctl start keepalived.service
keepalived]# ip addr list
//查看是否有172.16.72.100这个VIP
b.用客户端测试：
~]# curl http://172.16.72.100
sorry page for 2222222222222222222
c.再打开调度器1(drct1)的keepalived和http服务；
keepalived]# systemctl start keepalived.service
keepalived]# ip addr list
//查看是否有172.16.72.100这个VIP，相应的调度器2(drct2)上的VIP就没有了；因为此时调度器1(drct1)为MASTER，调度器2(drct2)为BACKUP；
d.用客户端测试：
~]# curl http://172.16.72.100
sorry page for 1
//此时访问到的页面已经是调度器1(drct1)上的http配置的页面
非抢占式下的高可用http服务实现：
与抢占式不同的是：
1.当A调度器的http服务故障时，B调度器马上就会变成BACKUP状态，后来A调度器的http服务修复恢复后，不会立即抢占为MASTER状态，依然处于BACKUP状态；即使A比B优先级高；
2.直到B调度器的http服务故障时，A才会从BACKUP状态回到MASTER
drct1的配置文件：/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct1
vrrp_mcast_group4 224.0.100.18
}
vrrp_script check_state {
script "/etc/keepalived/keep_down.sh"
interval 1
}
vrrp_script check_httpd {
script "/usr/bin/killall -0 httpd && exit 0 || exit 1"
interval 1
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 36
nopreempt
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.100
}
track_script {
check_state
check_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
//注意：
1.killall命令写进脚本的时候用绝对命令的绝对路径，否则可能会引起vrrp_script执行失败
2.drct1的初始状态为MASTER，需要填写在vrrp_instance VI_1填写nopreempt
3.vrrp_script check_httpd{}中需要写weight
drct2的配置文件：/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id drct2
vrrp_mcast_group4 224.0.100.18
}
vrrp_script check_state {
script "/etc/keepalived/keep_down.sh"
interval 1
}
vrrp_script check_httpd {
script "/usr/bin/killall -0 httpd && exit 0 || exit 1"
interval 1
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 36
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.16.72.100
}
track_script {
check_state
check_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
//注意：
1.killall命令写进脚本的时候用绝对命令的绝对路径，否则可能会引起vrrp_script执行失败;
2.drct2作为初始状态为BACKUP，所以不需要填写vrrp_instance VI_1中的nopreempt
3.vrrp_script check_httpd{}中不需要写weight
状态通知和转变情况脚本：
keepalived]# vim notify.sh
#!/bin/bash
#
CONTACT='root@localhost'
notify() {
MAILSUBJECT="$(hostname) changed state to $1, VIP reflact."
#发送电子邮件的主题
MAILBODY="$(date +'%F %T'): VRRP STATUS CHANGED, $(hostname) change state to $1"
#邮件的正文
echo "$MAILBODY" | mail -s "$MAILSUBJECT" "$CONTACT"
}
#case $1的$1指的是执行脚本时后面给定的参数
#notify()函数的$1指的是case引用notify()函数执行命令时notify后面的参数
case $1 in
master)
notify MASTER
;;
backup)
notify BACKUP
systemctl start httpd.service
;;
fault)
notify FAULT
systemctl start httpd.service
;;
*)
echo "Usage: $(basename $0) { master | backup | fault}"
;;
esac
keepalived]# chmox +x notify.sh
//两台调度器上都需要此脚本
测试：
a.先开启调度器2(drct2)的keepalived和http服务，客户端测试
~]# curl http://172.16.72.100
sorry page for 2222222222222222222
b.然后开启调度器1(drct1)的keepalived服务但不开启http服务，客户端测试：
~]# curl http://172.16.72.100
sorry page for 2222222222222222222
//此时表明访问到的依然是drct2上http服务，而不是drct1；
//因为如果访问到的是drct1，由于没有开启http服务，客户端得到的返回结果将会是：
curl: (7) Failed connect to 172.16.72.100:80; Connection refused
//这也间接证明了非抢占式的实现
c.开启调度器1(drct1)的http服务，客户端再测试：
~]# curl http://172.16.72.100
sorry page for 2222222222222222222
//返回结果依然是drct2的http服务配置的页面信息；
d.关闭调度器2(drct2)的http服务，客户端测试：
~]# curl http://172.16.72.100
sorry page for 1
f.打开关闭调度器2(drct2)的http服务，客户端测试：
~]# curl http://172.16.72.100
sorry page for 1
//根据e，f可以看出，调度器2(drct2)的http服务从故障到恢复正常后，依然访问调度器1(drct1)的http服务配置的页面信息；
测试总结：
1.当调度器2的keepalived、http第一开启，调度器1还没有开启keepalived、http服务，调度器2先处于MASTER状态；
2.直到调度2的http服务下线，调度器1的状态才能从BACKUP变成MASTER状态，调度器1开启的http服务才能被应用；
3.只要调度器1的http、keepalived服务一直正常工作，MASTER的状态就会一直不变，返回的信息依然是调度器1的http服务配置的内容信息；
设计非抢占式下的高可用http服务实现的要点：
1.killall命令写进脚本的时候用绝对命令的绝对路径，否则可能会引起vrrp_script执行失败;
2.drct1的初始状态为MASTER，需要填写在vrrp_instance VI_1填写nopreempt
3.vrrp_script check_httpd{}中需要写weight
4.drct2作为初始状态为BACKUP，所以不需要填写vrrp_instance VI_1中的nopreempt
5.vrrp_script check_httpd{}中不需要写weight
6.状态通知和转变情况脚本(/etc/keepalived/notify.sh)必须根据状态的改变进行对应的编写 个人心得总结：
1.抢占式下的高可用http服务(其他服务)实现属于强制性地选用某个调度器作为MASTER(在没有其他约束的条件下)；因此我觉得配合外部辅助脚本：MASTER/BACKUP的手动切换的脚本使用比较好；
抢占式 + MASTER/BACKUP的手动切换的脚本
2.非抢占式下的高可用http服务(其他服务)实现可以根据调度器自身的情况进行MASTER/BACKUP状态转换；
3.两者比较：
a.抢占式的设计来得易懂些，逻辑简单；自动化较非抢占式低一些；
b.非抢占式的设计逻辑比抢占式复杂一点，自动化程度更高些；