HA Cluster的实现方案
vrrp协议的实现：keepalived
VRRP协议的目的就是为了解决静态路由单点故障问题；VRRP通过竞选(election)协议来动态的将路由任务交给LAN中虚拟路由器中的某台VRRP路由器。
vrrp协议
（Virtual Redundant Routing Protocol）：虚拟冗余路由协议
| vrrp中的术语 | |
|---|---|
| VR | 虚拟路由器 |
| VRID | 虚拟路由器的标识号（0-255） |
| Master（Active） | 主路由器（活动节点） |
| Backup（Passive） | 备份路由器（被动节点） |
| VIP | 虚拟IP地址；流动（浮动）IP |
| VMAC | VIP对应的虚拟MAC地址（48位）；00-00-5E-00-01-{VRID} |
| 优先级 | 初始化过程中就决定了优先级 |
| gracious arp | 免费arp广播 |
vrrp工作方式
- 抢占式：如果 Backup 路由器工作在抢占方式下，当它收到 VRRP 报文后会将自己的优先级与通告报文中的优先级进行比较。如果自己的优先级比当前的 Master 路由器的优先级高就会主动抢占成为 Master 路由器否则将保持 Backup 状态。
- 非抢占式：如果 Backup 路由器工作在非抢占方式下则只要 Master 路由器没有出现故障Backup 路由器即使随后被配置了更高的优先级也不会成为Master 路由器。
vrrp认证方式与工作模式
认证方式
- 无认证
- 简单字符串认证：预共享密钥
- md5认证
工作模式
- master-backup模式
- master-master模式
vrrp工作原理
一个VRRP路由器有唯一的标识：VRID，范围为0-255，该路由器对外表现为唯一的虚拟MAC地址，地址的格式为00-00-5E- 00-01-[VRID]，主控路由器负责对ARP请求用该MAC地址应答，保证给终端设备的是唯一一致的IP和MAC地址，减少了切换对终端设备的影响。
VRRP控制报文只有一种：
VRRP通告(advertisement)使用IP多播数据包进行封装，组地址为224.0.0.18，发布范围只限于同一局域网内。这保证了VRID在不同网络中可以重复使用。在VRRP路由器组中按优先级选举主控路由器
为了保证VRRP协议的安全性，提供了两种安全认证措施：明文认证和IP头认证
vrrp优势
- 负载共享：允许来自LAN客户端的流量由多个路由器设备所共享；
- 多VRRP组：在一个路由器物理接口上可配置多达255个VRRP组；
- 抢占：在master故障时允许优先级更高的backup成为master；
- 通告协议：使用IANA所指定的组播地址224.0.0.18进行VRRP通告；
- VRRP追踪：基于接口状态来改变其VRRP优先级来确定最佳的VRRP路由器成为master；
- 冗余：可以使用多个路由器设备作为LAN客户端的默认网关，大大降低了默认网关成为单点故障的可能性；
- 多IP地址：基于接口别名在同一个物理接口上配置多个IP地址，从而支持在同一个物理接口上接入多个子网；
keepalived
Keepalived是基于VRRP协议实现的保证集群高可用的一个服务软件，运行在LVS之上，它的主要功能是实现真机的故障隔离及负载均衡器间的失败切换FailOver，可以防止单点故障。LVS结合keepalived，就实现了3层、4层、5/7层交换的功能
ka能够根据配置文件中定义自动生成ipvs规则（增、删），并能够对各RS的健康状态进行检测；
支持vrrp_script接口，vrrp_track；
通过编写vrrp脚本，通过vrrp_track跟踪各种服务；
keepalived组件
keepalived架构
控制面板（组件）：配置文件分析器，内存管理，IO复用
核心组件：vrrp stack，checker，ipvs wrapper，watch dog，smtp接口
只支持简单字符串认证
| 组件 | |
|---|---|
| vrrp stack | 实现vrrp协议 |
| smtp | 发送通知邮件；可在地址流动时发邮件，还可根据checkers检查后把宕机从从ipvs规则中移除，发出此邮件 |
| checkers | 检测后端RS健康状态，可基于tcp、htp、ssl、misc检测；发现宕机就从ipvs规则中移除 |
| watchdog | 实现监控vrrp stack和checkers的健康 |
| ipvs wrapper | checkers通过调用该包装器，来实现在ipvs中添加、删除或修改规则 |
HA Cluster配置的前提：（各调度器）
（1）各节点时间同步
基于ntp协议同步，chrony（centos 7）
（2）确保iptables及selinux不会阻碍
心跳信息传递：3种方式
- 单播
- 广播
- 组播：最佳使用方式；组播地址：224.0.0.0-239.0.0.0
（3）各节点之间可通过主机名相互通信（对ka并非必须，ka是无所谓的）
名称解析服务的解析结果必须与“uname -n”命令的结果一致；
如果是主机名通信，dns解析名必须与真实主机名一致；
（4）各节点之间的root用户可以基于密钥认证的sh通信，（对ka并非必须，ka无所谓）
配置示例
准备两台centos7主机，先做时间同步：
[root@node1 ~]# ntpdate 172.18.0.1
创建自动计划任务，向时间服务器每5分钟同步一次时间
[root@node1 ~]# crontab -e
*/5 * * * * /usr/sbin/ntpdate 172.18.0.1 &> /dev/null
安装keepalived并配置
[root@node1 ~]# yum -y install keepalived
...
Complete!
[root@node1 ~]# rpm -ql keepalived
/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/lib/systemd/system/keepalived.service
/usr/libexec/keepalived
/usr/sbin/keepalived
...
[root@node1 ~]# cd /etc/keepalived/
[root@node1 keepalived]# cp keepalived.conf{,.bak}
[root@node1 keepalived]# vim /etc/keepalived/keepalived.conf
:set nohlsearch
:.,$s/^/#/g
#先只配置全局段和虚拟路由实例段，其它都注释；
global_defs { #全局配置段
notification_email { #接收通告邮件的地址，收件人
root@localhost
}
notification_email_from [email protected] #发件人
vrrp_iptables #不自动生产iptables规则
smtp_server 127.0.0.1 #配置邮件服务器，要求能在互联网上正反解析；
smtp_connect_timeout 30 #邮件服务器的超时时长
router_id node1 #定义当前路由器设备的ID号，一般为主机名；
vrrp_mcast_group4 224.0.11.18 #组播地址，默认为224.0.0.18，但是集群里的所有节点都有使用此地址；表示同一个组播域；
}
vrrp_instance VI_1 { #虚拟路由器配置段，实例名称为VI_1，也可自定义；
state MASTER #当前节点在此虚拟路由器中的初始状态；表示为主节点；取值为：MASTER|BACKUP；
interface eno16777736 #表示vrrp发送通告的接口；
virtual_router_id 11 #虚拟路由器ID，范围0-255；
priority 100 #当前物理节点在此虚拟路由器中的优先级；
advert_int 1 #表示vrrp的通告的时间间隔；为1秒；
authentication { #认证方式
auth_type PASS #使用简单密码认证
auth_pass vHFHLlTA #随机随机字符串，可使用openssl rand -base64 10生成，前8位字符有效；；
}
virtual_ipaddress { #定义虚拟IP，不指明则为前面的interface指明的接口地址；
172.18.11.66 #网卡别名地址，可写为172.18.11.66 dev eno16777736 label eno16777736:0
}
}
配置文件复制到另一调度器172.18.11.121上
[root@node1 ~]# scp keepalived.conf [email protected]:/etc/keepalived
在172.18.11.121编辑该配置文件
[root@node2 ~]# vim /etc/keepalived/keepalived.conf
修改：
state BACKUP #修改为备用
priority 50 #修改小优先级；
#密码、router_id、vrrp_mcast_group4要保存一致；
在两个调度器上开启keepalived服务
[root@node1 ~]# systemctl start keepalived.service
[root@node2 ~]# systemctl start keepalived.service
分别查看两个调度器，VIP地址是否自动在主服务器的接口别名上：
[root@node1 ~]# ip addr l
[root@node2 ~]# ip addr l
[root@node1 ~]# tail /var/log/messages
[root@node2 ~]# tail /var/log/messages
手动使172.18.11.111主用服务器停止keepalived服务，再查看
[root@node1 ~]# ip addr l]# systemctl stop keepalived.service
用另一终端查看日志信息不退出
[root@centos7 ~]# tail -f /var/log/messages
显示部分内容：
May 10 05:25:01 localhost systemd: Stopping LVS and VRRP High Availability Monitor...
May 10 05:25:01 localhost Keepalived_vrrp[4258]: VRRP_Instance(VI_1) sending 0 priority 发送优先级为0的报文；
May 10 05:25:01 localhost Keepalived_vrrp[4258]: VRRP_Instance(VI_1) removing protocol VIPs. 移除VIP的协议
May 10 05:25:01 localhost Keepalived_healthcheckers[4257]: Netlink reflector reports IP 172.18.11.66 removed 移除VIP
May 10 05:25:01 localhost systemd: Stopped LVS and VRRP High Availability Monitor
在另一调度器172.18.11.121上，查看日志
[root@node2 ~]# tail /var/log/messages
May 9 08:09:21 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Transition to MASTER STATE 称为主用服务器
May 9 08:09:22 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Entering MASTER STATE 确认主用状态
May 9 08:09:22 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) setting protocol VIPs. 设置
May 9 08:09:22 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eno16777736 for 172.18.11.66 发送免费ARP报文
May 9 08:09:22 localhost Keepalived_healthcheckers[3880]: Netlink reflector reports IP 172.18.11.66 added 获得VIP地址
May 9 08:09:27 localhost Keepalived_vrrp[3881]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eno16777736 for 172.18.11.66
May 9 08:10:01 localhost systemd: Started Session 58 of user root.
May 9 08:10:01 localhost systemd: Starting Session 58 of user root.
May 9 08:10:01 localhost systemd: Started Session 59 of user root.
May 9 08:10:01 localhost systemd: Starting Session 59 of user root.
再手动使172.18.11.111主用服务器上线：
[root@node1 ~]# systemctl start keepalived.service
#默认工作在抢占模式；优先级高的会自动成为主用服务器；
这就是keepalived自动完成地址浮动；
修改两个调度器的VIP别名：
[root@node1 ~]# vim /etc/keepalived/keepalived.conf
virtual_ipaddress{
172.18.11.66 dev eno16777736 label eno16777736:0
}
把两个调度器停止keepalived服务：
先启动优先级低的服务器：172.18.11.121：
[root@node2 ~]# systemctl start keepalived.service
此时使用ifconfig命令，engine查看网卡别名，已经自动加上了；
再启动优先级高的服务器：172.18.11.111：
[root@node1 ~]# systemctl start keepalived.service
vrrp有两种工作模型：主/备，主/主
为了提高利用率，就使用主/主工作模型；
即A主B从，B主A从；这样AB都能发挥作用；在外部都称为主时，可在前端dns配置A记录把两个ip都配置在一个主机名上即可；
但在有些场景中，AB都称为主时不是配置在同一网卡上的两个不同的别名，而是分别配置在两块网卡上；
AB每个主机都有两块网卡，一块网卡面向外网，一块网卡面向内网通信；期望通信时，这两个地址都是VIP，只不过，前面外网是虚拟路由器1，内网是虚拟路由器2；
里面的别名配置在内网网卡上，外面的别名配置在外网网卡上；当外网其中一个线路掉线时，可自动切换到另一个外网网卡的服务器上，要求而也要内网同样要切换；要求内外通信使用VIP，所以没有必要切换的内网也要切换；
这种把两个虚拟路由器，配置为只能对一个设备是主服务器，同进同退，这种就叫做一个同步组；
安装keepalived
Centos6.4+，程序包已经在base源中提供；
centos 7
主配置文件：/etc/keepalived/keepalived.conf
Unit file：/usr/lib/systemd/system/keepalived.service
配置文件：/etc/sysconfig/keepalived
配置文件内容块：
GLOBAL CONFIGURATION
#全局配置段，对所有vvrp实例虚拟服务器都是生效；
global_defs {
...
}
VRRPD CONFIGURATION #虚拟路由器同步组配置段
vrrp_sync_group VG_1 {
...
}
VRRP instance(s) #虚拟路由器实例配置段
vrrp_instance INST_NAME {
...
}
LVS CONFIGURATION #集群配置段
virtual_server_group { #虚拟服务器组配置段
...
}
Virtual server(s) #虚拟服务器配置段
virtual_server IP port |
virtual_server fwmark int { #十进制的整数，（实际最终用的是十六进制数）
protocol TCP #注意：keepalived只支持tcp；
...
real_server { #真实后端服务器配置段
...
}
}
全局配置段示例
global_defs {
notification_email { #用于指定报警邮件发往的邮箱地址
root@localhost
}
notification_email_from keepalived@localhost #用于指定报警邮件的发件人
smtp_server 127.0.0.1 #用于指定邮件服务器的地址
smtp_connect_timeout 30 #用于指定邮件服务器的连接超时时长
router_id node1 #路由器的标识
vrrp_mcast_group4 224.0.100.19 #用于设置vrrp的广播地址，在同一个HA cluster中的机器，要确保其广播地址一致才能接受到相应的vrrp报文
vrrp_strict #执行严格的vrrp协议检查，某些情况下会禁止到vip的访问。
}
Static routes/address/rules：用于配置keepalived中不会被VRRP移除的静态地址、路由或者规则，一般不会使用。
VRRP相关配置段示例
vrrp_instance {
state MASTER|BACKUP #用于指定此虚拟路由器在vrrp组的角色状态
interface eno16777736 #用于绑定当前虚拟路由器所使用的物理接口
virtual_router_id 14 #用于指定当前虚拟路由器在vrrp组的唯一标识id，范围为0-255
priority 100 #用于设置当前虚拟路由器在vrrp组里面的优先级，范围为1-254；
advert_int 1 #用于设置虚拟路由器发送vrrp通告的时间间隔
nopreempt|preempt #定义工作模式为非抢占或抢占模式；
preempt_delay 300 #定义在抢占式模式下，节点上线后触发新选举操作的延迟时长；
authentication { #用于设置vrrp组协商的方式及密码
auth_type PASS #定义认证类型为简单密码认证
auth_pass 571f97b2 #定义密码串，最长不超过8个字符
}
virtual_ipaddress { #用于指定需要在绑定的物理接口上添加的虚拟ip地址
#/ brd dev scope label VRRP script(s)：用于定义周期性执行的脚本，可定义调用用于检查相应的服务或Ip的状态的脚本。
vrrp_script { #定义周期执行的脚本，此脚本的退出码决定了当前监控的vrrp组的运行状态
script | #定义执行脚本的存放路径
interval INT #定义调用执行脚本的周期，默认为1s。
timeout # 脚本执行超时时间，脚本执行超时后，则被认为失败
rise # 定义脚本检查成功多少次，才认可当前的状态为正常
fall #定义检查失败多少次，才认为当前状态为失败
}
LVS配置段示例
Virtual server(s)：用于定义虚拟服务器的设置，虚拟服务器可以用Ip端口、fwmark或virtual server group来定义。
virtual_server IP port | virtual_server fwmark |virtual_server group string
{
delay_loop #=服务轮询的时间间隔；
lb_algo rr|wrr|lc|wlc|lblc|sh|dh #定义调度方法；
lb_kind NAT|DR|TUN #集群的类型；
persistence_timeout #持久连接时长；
protocol TCP|UDP|SCTP #服务协议；
sorry_server #备用服务器地址；
real_server {
weight #设置real server的权重
notify_up | #当出现匹配字符串时，就认为服务是up的
notify_down | #当出现匹配字符串时，就认为服务是down的
HTTP_GET|SSL_GET { #对real server作应用层检测
url {
path #定义要监控的URL；
status_code #判断上述检测机制为健康状态的响应码；
digest #判断上述检测机制为健康状态的响应的内容的校验码；
}
nb_get_retry #重试次数；
delay_before_retry #重试之前的延迟时长；
connect_ip #向当前RS的哪个IP地址发起健康状态检测请求
connect_port #向当前RS的哪个PORT发起健康状态检测请求
bindto #发出健康状态检测请求时使用的源地址；
bind_port #发出健康状态检测请求时使用的源端口；
connect_timeout #连接请求的超时时长；
}
TCP_CHECK {
connect_ip #向当前RS的哪个IP地址发起健康状态检测请求
connect_port #向当前RS的哪个PORT发起健康状态检测请求
bindto #发出健康状态检测请求时使用的源地址；
bind_port #发出健康状态检测请求时使用的源端口；
connect_timeout #连接请求的超时时长；
}
}
}
高可用的ipvs集群
! Configuration File for keepalived
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 14
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 571f97b2
}
virtual_ipaddress {
10.1.0.93/16 dev eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 10.1.0.93 80 {
delay_loop 3
lb_algo rr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 10.1.0.69 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 1
nb_get_retry 3
delay_before_retry 1
}
}
real_server 10.1.0.71 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
TCP_CHECK {
nb_get_retry 3
delay_before_retry 2
connect_timeout 3
}
}
}
双主模型
主机配置：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node1
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 14
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 571f97b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
vrrp_instance VI_2 {
state BACKUP
interface eno16777736
virtual_router_id 15
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 578f07b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
备机配置：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node2
vrrp_mcast_group4 224.0.100.19
}
vrrp_instance VI_1 {
state BACKUP
interface eno16777736
virtual_router_id 16
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass 571f97b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
vrrp_instance VI_2 {
state MASTER
interface eno16777736
virtual_router_id 17
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 578f07b2
}
virtual_ipaddress {
172.18.11.66/16 dev eno16777736
}
}
通告脚本
可定义短信网关、微信网关、邮件服务器等方式发送；
此处，仅以本地邮件服务编写脚本；
[root@node1 ~]# vim notify.sh
#!/bin/bash
#
contact='root@localhost'
notify(){
mailsuject="$(hostname) to be $1:vip floating"
mailbody="$(date +'%F %T'):vrrp transition,$(hostname) change to be $1"
echo $mailbody |mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify bakcup
;;
fault)
notify fault
;;
*)
echo "Usage :$(basename $0) {master|backup|fault}"
;;
esac
[root@node1 ~]# chmod +x notify.sh
[root@node1 ~]# ./notify.sh master
[root@node1 ~]# mail
显示：
Heirloom Mail version 12.5 7/5/10. Type ? for help.
"/var/spool/mail/root": 1 message 1 new
>N 1 root Tue May 10 05:55 18/682 ""
& 1 输入1查看邮件1
部分内容：
2016-05-10 05:55:33:vrrp transition,localhost.localdomain change to be master
exit 退出邮件
把通告脚本复制给另一调度器上
[root@node1 ~]# scp notify.sh [email protected]:/etc/keepalived
在两个调度器上配置通告脚本
[root@node2 ~]# vim /etc/keepalived/keepalived.conf
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
把两个调度器停止keepalived服务器
[root@node1 ~]# systemctl stop keepalived.service
[root@node2 ~]# systemctl stop keepalived.service
[root@node1 ~]# mail
[root@node2 ~]# mail
删除已有邮件，为后续清空环境；
d 1
d 2
再先后启动两个调度器keepalived服务器
先启动优先级低的备用调度器：
[root@node2 ~]# systemctl start keepalived.service
[root@node2 ~]# mail
显示：刚启动发一个邮件为backup的，因为只有一个调度器，过一会又收到一个邮件成为master；
再启动优先级高的调度器：
[root@node1 ~]# mail
则查看，显示也收到2条邮件，均为master信息；
通告脚本示例：
#!/bin/bash
#
contact='root@localhost'
notify(){
mailsuject="$(hostname) to be $1:vip floating"
mailbody="$(date +'%F %T'):vrrp transition,$(hostname) change to be $1"
echo $mailbody |mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify bakcup
;;
fault)
notify fault
;;
*)
echo "Usage :$(basename $0) {master|backup|fault}"
;;
esac
调用方法：
vrrp_instanace {
...
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
注意：脚本路径要使用双引号；
示例：keepalived的主从架构
搭建RS1和RS2
首先安装nginx程序：
[root@RS1 ~]# yum install -y epel-release
[root@RS1 ~]# yum install -y nginx
然后编辑/etc/hosts文件：
[root@RS1 ~]# vim /etc/hosts
192.168.0.81 DR1.ilinux.io DR1
192.168.0.87 DR2.ilinux.io DR2
192.168.0.83 RS1.ilinux.io RS1
192.168.0.84 RS2.ilinux.io RS2
随后修改/usr/share/nginx/html/index.html 内容为如下：
[root@RS1 ~]# vim /usr/share/nginx/html/index.html
This is RS1 192.168.0.83
启动nginx：
[root@RS1 ~]# systemctl start nginx
关闭firewalld和修改selinux的状态为permissive：
[root@RS1 ~]# systemctl stop firewalld
[root@RS1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@RS1 ~]# setenforce 0
接着在RS1上配置lvs-dr的配置，首先创建rs脚本：
[root@RS1 ~]# vim RS.sh
#/bin/bash
vip=192.168.0.99
mask='255.255.255.255'
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig lo:0 $vip netmask $mask broadcast $vip up
route add -host $vip dev lo:0
;;
stop)
ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage $(basename $0) start|stop "
exit 1
;;
esac
随后执行RS脚本：
[root@RS1 ~]# bash -x RS.sh start
+ vip=192.168.0.99
+ mask=255.255.255.255
+ case $1 in
+ echo 1
+ echo 1
+ echo 2
+ echo 2
+ ifconfig lo:0 192.168.0.99 netmask 255.255.255.255 broadcast 192.168.0.99 up
+ route add -host 192.168.0.99 dev lo:0
重复以上步骤并结合相应的信息搭建RS2。
搭建DR1
首先yum安装keepalived和ipvsadm程序包：
[root@DR1 ~]# yum install -y ipvsadm keepalived
然后编辑/etc/hosts文件：
[root@DR1 ~]# vim /etc/hosts
192.168.0.81 DR1.ilinux.io DR1
192.168.0.87 DR2.ilinux.io DR2
192.168.0.83 RS1.ilinux.io RS1
192.168.0.84 RS2.ilinux.io RS2
随后编辑/etc/keepalived/keepalived.conf文件：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DR1
vrrp_mcast_group4 224.0.0.0.20
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state MASTER
interface eno16777736
virtual_router_id 1
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass ^&IUYH*&
}
virtual_ipaddress {
192.168.0.99/24 dev eno16777736 label eno16777736:0
}
}
virtual_server 192.168.0.99 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
启动keepalived，查看ipvsadm和接口的状态：
[root@DR1 ~]# systemctl start keepalived
[root@DR1 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.99:http rr
-> 192.168.0.83:http Route 1 1 4
-> 192.168.0.84:http Route 1 1 80
[root@DR1 ~]# ifconfig
eno16777736: flags=4163 mtu 1500
inet 192.168.0.81 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::20c:29ff:fe21:59b9 prefixlen 64 scopeid 0x20
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
RX packets 62277 bytes 72099132 (68.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21742 bytes 2744670 (2.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno16777736:0: flags=4163 mtu 1500
inet 192.168.0.99 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
关闭firewalld和修改selinux的状态为permissive：
[root@DR1 ~]# systemctl stop firewalld
[root@DR1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@DR1 ~]# setenforce 0
搭建DR2
首先yum安装keepalived和ipvsadm程序包：
[root@DR2 ~]# yum install -y ipvsadm keepalived
然后编辑/etc/hosts文件：
[root@DR2 ~]# vim /etc/hosts
192.168.0.81 DR1.ilinux.io DR1
192.168.0.87 DR2.ilinux.io DR2
192.168.0.83 RS1.ilinux.io RS1
192.168.0.84 RS2.ilinux.io RS2
随后编辑/etc/keepalived/keepalived.conf文件：
global_defs {
notification_email {
root@localhost
}
notification_email_from keepalived@localhost
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id DR2
vrrp_mcast_group4 224.0.0.0.20
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 1
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass ^&IUYH*&
}
virtual_ipaddress {
192.168.0.99/24 dev ens33 label ens33:0
}
}
virtual_server 192.168.0.99 80 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
最后启动keepalived并查看ipvsadm的状态：
[root@DR2 ~]# systemctl start keepalived
[root@DR2 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.99:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
[root@DR2 ~]# ifconfig #因为DR2的角色为BACKUP，因此不会创建Ip为192.168.0.99的子接口
ens33: flags=4163 mtu 1500
inet 192.168.0.87 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::5e4b:719d:3781:43a0 prefixlen 64 scopeid 0x20
inet6 fe80::4c6e:8b7e:2dcd:665d prefixlen 64 scopeid 0x20
ether 00:0c:29:26:a3:20 txqueuelen 1000 (Ethernet)
RX packets 34041 bytes 27376635 (26.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 14131 bytes 2169836 (2.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1 (Local Loopback)
RX packets 101 bytes 8902 (8.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 101 bytes 8902 (8.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
关闭firewalld和修改selinux的状态为permissive：
[root@DR2 ~]# systemctl stop firewalld
[root@DR2 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@DR2 ~]# setenforce 0
测试访问
在客户端上测试访问vip所提供的服务：
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.99; done
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
此时DR1为主机，DR2为备机，两者都工作正常，因此请求连接是由DR1来处理。
那么我们模拟DR1宕机，把DR1的服务停用后，再来观察DR2的状态和客户端的访问情况。
DR2的keepalived状态：
[root@DR2 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2018-05-29 14:34:08 CST; 1h 29min ago
Process: 11808 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 11809 (keepalived)
CGroup: /system.slice/keepalived.service
├─11809 /usr/sbin/keepalived -D
├─11810 /usr/sbin/keepalived -D
└─11811 /usr/sbin/keepalived -D
May 29 16:03:06 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) removing protocol VIPs.
May 29 16:03:47 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) Entering MASTER STATE
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) setting protocol VIPs.
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
May 29 16:03:48 DR2 Keepalived_vrrp[11811]: Sending gratuitous ARP on ens33 for 192.168.0.99
DR2检测到DR1的宕机，主动变成了MASTER状态。
在客户端的访问情况：
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.99; done
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
此时DR1模拟为宕机状态，DR2为MASTER，客户端的访问不受影响。
示例：Keepalived的主主架构
此处以上面主从架构的拓扑为例，将主从架构更改为主主架构，首先我们需更改DR1和DR2的keepalived的配置，然后要在RS1和RS2上添加lvs-dr中与192.168.0.98虚拟IP相关的配置。
DR模式开启ip转发
[root@DR1 ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
修改DR1
编辑/etc/keepalived/keepalived.conf文件
[root@DR1 ~]# vim /etc/keepalived/keepalived.conf
#添加如下配置
vrrp_instance VI_2 {
state BACKUP
interface eno16777736
virtual_router_id 2
priority 98
advert_int 1
authentication {
auth_type PASS
auth_pass POM123(*
}
virtual_ipaddress {
192.168.0.98/24 dev eno16777736 label eno16777736:1
}
}
#添加虚拟服务器组backend
virtual_server_group backend {
192.168.0.99 80
192.168.0.98 80
}
#修改虚拟服务器调用虚拟服务器组
virtual_server group backend {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
停用再启动keepalived：
[root@DR1 ~]# systemctl stop keepalived
[root@DR1 ~]# systemctl start keepalived
此时ipvsadm和接口的状态为：
[root@DR2 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.98:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
TCP 192.168.0.99:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
[root@DR1 ~]# ifconfig
eno16777736: flags=4163 mtu 1500
inet 192.168.0.81 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::20c:29ff:fe21:59b9 prefixlen 64 scopeid 0x20
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
RX packets 63844 bytes 72282542 (68.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23654 bytes 2934901 (2.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno16777736:0: flags=4163 mtu 1500
inet 192.168.0.99 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:21:59:b9 txqueuelen 1000 (Ethernet)
修改DR2
编辑/etc/keepalived/keepalived.conf文件：
[root@DR2 ~]# vim /etc/keepalived/keepalived.conf
#添加如下配置
vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 2
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass POM123(*
}
virtual_ipaddress {
192.168.0.98/24 dev ens33 label ens33:1
}
}
#添加虚拟服务器组backend
virtual_server_group backend {
192.168.0.99 80
192.168.0.98 80
}
#修改虚拟服务器调用虚拟服务器组
virtual_server group backend {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.0.83 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
real_server 192.168.0.84 80 {
weight 1
HTTP_GET {
url {
path /index.html
status_code 200
}
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
停用再启动keepalived：
[root@DR2 ~]# systemctl stop keepalived
[root@DR2 ~]# systemctl start keepalived
此时ipvsadm和接口的状态为：
[root@DR2 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.98:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
TCP 192.168.0.99:80 rr
-> 192.168.0.83:80 Route 1 0 0
-> 192.168.0.84:80 Route 1 0 0
[root@DR2 ~]# ifconfig
ens33: flags=4163 mtu 1500
inet 192.168.0.87 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::5e4b:719d:3781:43a0 prefixlen 64 scopeid 0x20
inet6 fe80::4c6e:8b7e:2dcd:665d prefixlen 64 scopeid 0x20
ether 00:0c:29:26:a3:20 txqueuelen 1000 (Ethernet)
RX packets 39989 bytes 28047325 (26.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20816 bytes 2894556 (2.7 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33:0: flags=4163 mtu 1500
inet 192.168.0.98 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:26:a3:20 txqueuelen 1000 (Ethernet)
配置RS1和RS2
复制编辑RS脚本：
[root@RS1 ~]# cp RS.sh RS_new.sh
#/bin/bash
#修改vip为192.168.0.98
vip=192.168.0.98
mask='255.255.255.255'
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig lo:1 $vip netmask $mask broadcast $vip up #修改接口为lo:1
route add -host $vip dev lo:1 #修改接口为lo:1
;;
stop)
ifconfig lo:1 down #修改接口为lo:1
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage $(basename $0) start|stop "
exit 1
;;
esac
运行脚本：
[root@RS1 ~]# bash -x RS_new.sh start
+ vip=192.168.0.98
+ mask=255.255.255.255
+ case $1 in
+ echo 1
+ echo 1
+ echo 2
+ echo 2
+ ifconfig lo:1 192.168.0.98 netmask 255.255.255.255 broadcast 192.168.0.98 up
+ route add -host 192.168.0.98 dev lo:1
在RS2 上也按照如上步骤执行操作即可
测试访问
此时在客户端通过192.168.0.99和192.168.0.98均能访问到后端RS所提供的web服务：
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.98; done
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
[root@client ~]# for i in {1..10} ; do curl http://192.168.0.99; done
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
在客户端上编辑/etc/hosts，添加域名解析到99和98：
[root@client ~]# vim /etc/hosts
192.168.0.99 www.ilinux.io
192.168.0.98 www.ilinux.io
此时通过域名解析能使得只要有99和98有一个正常工作，客户端均能正常访问服务。
[root@client ~]# for i in {1..10} ; do curl http://www.ilinux.io; done
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
This is RS2 192.168.0.84
This is RS1 192.168.0.83
参考文档：https://www.jianshu.com/p/eefa8afabb09