LVS+keepalived高可用实现

Keepalived介绍

Keepalived软件起初是专为LVS负载均衡软件设计的，用来管理并监控LVS集群系统中各个服务节点的状态，后来又加入了可以实现高可用的VRRP功能。因此，keepalived除了能够管理LVS软件外，还可以作为其他服务的高可用解决方案软件。

    keepalived软件主要是通过VRRP协议实现高可用功能的。VRRP是Virtual  Router  Redundancy Protocol（虚拟路由冗余协议）的缩写，VRRP出现的目的就是为了解决静态路由的单点故障问题的，它能保证当个别节点宕机时，整个网络可以不间断地运行。所以，keepalived一方面具有配置管理LVS的功能，同时还具有对LVS下面节点进行健康检查的功能，另一方面也可以实现系统网络服务的高可用功能。

Keepalived高可用故障切换转移原理

    Keepalived高可用服务对之间的故障切换转移，是通过VRRP(Virtual Router Redundancy Protocol ,中文名为虚拟路由冗余协议)来实现的。在keepalived服务工作时，主Master节点会不断地向备节点发送（多播的方式）心跳消息，用来告诉备Backup节点自己还活着。当主节点发生故障时，就无法发送心跳的消息了，备节点也因此无法继续检测到来自主节点的心跳了。于是就会调用自身的接管程序，接管主节点的IP资源和服务。当主节点恢复时，备节点又会释放主节点故障时自身接管的IP资源和服务，恢复到原来的备用角色。

VRRP介绍

VRRP（Virtual Router Redundancy Protocol，虚拟路由器冗余协议），VRRP是为了解决静态路由的高可用。VRRP的基本架构
虚拟路由器由多个路由器组成，每个路由器都有各自的IP和共同的VRID(0-255)，其中一个VRRP路由器通过竞选成为MASTER，占有VIP，对外提供路由服务，其他成为BACKUP，MASTER以IP组播（组播地址：224.0.0.18）形式发送VRRP协议包，与BACKUP保持心跳连接，若MASTER不可用（或BACKUP接收不到VRRP协议包），则BACKUP通过竞选产生新的MASTER并继续对外提供路由服务，从而实现高可用。

集群环境说明

Keepalived软件使用

安装keepalived

yum install keepalived -y

配置文件说明

配置文件在/etc/keepalived/keepalived.conf

global_defs {    #全局配置    

 notification_email {   定义报警邮件地址

      [email protected]

      [email protected]

      [email protected]

    }

    notification_email_from [email protected]  #定义发送邮件的地址

    smtp_server 192.168.200.1   #邮箱服务器

    smtp_connect_timeout 30      #定义超时时间

    router_id LVS_DEVEL        #定义路由标识信息，相同局域网唯一

 } 

vrrp_instance VI_1 {   #定义实例

    state MASTER         #状态参数 master/backup 只是说明，没有实用

    interface eth0       #虚IP地址放置的网卡位置

    virtual_router_id 51 #同一家族要一致，同一个集群id一致

    priority 100         # 优先级决定是主还是备    越大越优先

    advert_int 1        #主备通讯时间间隔

    authentication {     #

        auth_type PASS    #

        auth_pass 1111    #认证

    }                     

    virtual_ipaddress { 

        192.168.200.16    设备之间使用的虚拟ip地址

        192.168.200.17

        192.168.200.18

    }

}

LVS（DR）+keepalived

（LVS（DR）搭建可参考https://blog.csdn.net/yuanfangPOET/article/details/83716457），下面是LVS+keepalived高可用的实现及一步步优化。

keepalived日志设置

vim /etc/sysconfig/keepalived

KEEPALIVED_OPTIONS="-D -S 3"  //-S 3 添加日志

vim /etc/rsyslog.conf

local3.*                                       /var/log/keepalived.log

//找到对应位置进行添加

重新启动keepalived会产生日志文件

keepalived单点故障测试

修改lb1和lb2中的keepalived.conf文件

! Configuration File for keepalived

global_defs {
   notification_email {
   root@localhost
   }
   notification_email_from kaadmin@localhost//随意设置
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb1 #备用lb2可改为lb2
}
vrrp_instance VI_1 {
    state MASTER #lb2可改为BACKUP
    interface eth0
    virtual_router_id 51
    priority 100 #lb2修改为99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111  #主备lb认证密码需要相同
    }
    virtual_ipaddress {
    192.168.231.154 #VIP
    }
}

测试：

停止keepalived服务

[root@director keepalived]# service keepalived stop

Stopping keepalived:                                       [  OK  ]

[root@director keepalived]# tail /var/log/keepalived.log

Nov  2 09:39:52 director Keepalived[26520]: Stopping Keepalived v1.2.13 (03/19,2015)

Nov  2 09:39:52 director Keepalived_vrrp[26523]: VRRP_Instance(VI_1) sending 0 priority

Nov  2 09:39:52 director Keepalived_vrrp[26523]: VRRP_Instance(VI_1) removing protocol VIPs.

Nov  2 09:39:52 director Keepalived_healthcheckers[26522]: Netlink reflector reports IP 192.168.231.154 removed

查看VIP地址是否存在，不存在，说明漂移到lb2备用节点

lb1:

[root@director keepalived]# ip addr list

1: lo: mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:42:7b:e8 brd ff:ff:ff:ff:ff:ff

    inet 192.168.231.150/24 brd 192.168.231.255 scope global eth0

    inet6 fe80::20c:29ff:fe42:7be8/64 scope link

       valid_lft forever preferred_lft forever

lb2:漂移到lb2

[root@lb2 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:16:b9:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.151/24 brd 192.168.231.255 scope global eth0
    inet 192.168.231.154/32 scope global eth0
    inet6 fe80::20c:29ff:fe16:b920/64 scope link
       valid_lft forever preferred_lft forever

Keepalived手动调度期望地址能够在二者不停服务的情况下转换

修改lb1和lb2的keepalived.conf，lb2中也添加到响应的位置

! Configuration File for keepalived

global_defs {
   notification_email {
   root@localhost
   }
   notification_email_from kaadmin@localhost
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb1
}
vrrp_script chk_maintenance {   #chk_maintenance是自己定义的脚本名字
        script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"  #script 后边编写脚本，本脚本判断目录中是否存在down文件，存在则执行下边语句，否则，exit0
        interval 1
        weight -20 #有down文件时，权重-20
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    192.168.231.154
    }
    track_script { #追踪检测脚本
        chk_maintenance
        }
}

添加后重启keepalived,再进行测试：

创建down文件，发现VIP移除了，漂移到lb2中

[root@lb1 keepalived]# touch down
[root@lb1 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:42:7b:e8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.150/24 brd 192.168.231.255 scope global eth0
    inet6 fe80::20c:29ff:fe42:7be8/64 scope link
       valid_lft forever preferred_lft forever

[root@lb2 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:16:b9:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.151/24 brd 192.168.231.255 scope global eth0
    inet 192.168.231.154/32 scope global eth0
    inet6 fe80::20c:29ff:fe16:b920/64 scope link
       valid_lft forever preferred_lft forever

删除down文件，VIP就会重新出现在lb1中

双主虚拟路由

创建两个vrrp_instance实例：

在lb1和lb2的配置文件keepalived.conf中再添加一个vrrp_instance实例：

因为192.168.231.154在lb1中为MASTER（主节点），在lb2中为BACKUP（备用节点）,

那么就设置1

lb1:

vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 52 #不能与其他实例相同
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 2222 #不能与其他实例相同
    }
    virtual_ipaddress {
    192.168.231.155
    }
    track_script {
        chk_tenance
        }
}

lb2:

vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 2222
    }
    virtual_ipaddress {
    192.168.231.155
    }
    track_script {
        chk_maintenance
        }
}

测试：

重启keepalived后，会添加192.168.231.155到eth0

[root@lb2 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:16:b9:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.151/24 brd 192.168.231.255 scope global eth0
    inet 192.168.231.155/32 scope global eth0
    inet6 fe80::20c:29ff:fe16:b920/64 scope link
       valid_lft forever preferred_lft forever

也可以测试192.168.231.155，在lb2中创建down文件，看VIP是否会漂移到lb1中

[root@lb1 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:42:7b:e8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.150/24 brd 192.168.231.255 scope global eth0
    inet 192.168.231.154/32 scope global eth0
    inet 192.168.231.155/32 scope global eth0
    inet6 fe80::20c:29ff:fe42:7be8/64 scope link
       valid_lft forever preferred_lft forever

脚本实现VIP漂移和RS(down,up)发送通知

在lb1和lb2的/etc/keepalived/目录中创建脚本notify.sh,根据情况修改

#!/bin/bash

# Author: MageEdu

# description: An example of notify script

#

vip=192.168.231.150

contact='root@localhost'

notify() {

mailsubject="`hostname` to be $1: $vip floating"

mailbody="`date '+%F %H:%M:%S'`: vrrp transition, `hostname` changed to be $1"

echo $mailbody | mail -s "$mailsubject" $contact

}

case "$1" in

master)

notify master

/etc/rc.d/init.d/haproxy start

exit 0 ;; backup)

notify backup

/etc/rc.d/init.d/haproxy stop

exit 0 ;; fault)

notify fault

/etc/rc.d/init.d/haproxy stop

exit 0 ;; *)

echo 'Usage: `basename $0` {master|backup|fault}'

exit 1 ;;

esac

然后在keepalived.conf中的vrrp_instance {} 实例中添加调用脚本：

    notify_master "/etc/keepalived/notify.sh  master"

    notify_backup "/etc/keepalived/notify.sh  backup"

    notify_fault "/etc/keepalived/notify.sh    fault"

重启keepalived后进行测试：

创建down文件,使192.168.231.154漂移到lb2

[root@lb1 keepalived]# touch down

[root@lb1 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:42:7b:e8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.150/24 brd 192.168.231.255 scope global eth0
    inet6 fe80::20c:29ff:fe42:7be8/64 scope link
       valid_lft forever preferred_lft forever
[root@lb1 keepalived]# mail
Heirloom Mail version 12.4 7/29/08.  Type ? for help.
"/var/spool/mail/root": 2 messages 2 new
>N  1 root                 Sun Nov  4 22:59  18/676   "lb1 to be backup: 192.168.231.154 "

把rs1中的httpd停掉，会受到邮件：

N 14 [email protected]  Sun Nov  4 23:36  17/626   "[lb1] Realserver [192.168.231.153]:80 - DOWN"

添加virtual_server实现不用手动创建LVS规则

首先清空在LB中LVS规则和绑定在eth0上的VIP，在对rs进行修改

lb1,lb2

ifconfig  eth0:0 down

ip  addr del 192.168.231.154 dev eth0:0

ipvsadm -C

rs1,rs2

修改arp_ignore和arp_announce

        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
不懂就参考一下https://blog.csdn.net/yuanfangPOET/article/details/83716457

博文中arp_ignore,arp_announce进行了解

在keepalived.conf中添加入下内容：

virtual_server 192.168.231.154 80 {
    delay_loop 6
    lb_algo wrr #调度算法
    lb_kind DR #LVS类型
    nat_mask 255.255.255.0
    protocol TCP
    real_server 192.168.231.152 80 { #rs1服务器IP
        weight 1
        HTTP_GET {
            url {                             #检测rs服务器是否挂掉
              path /
              status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
    real_server 192.168.231.153 80 {#rs2服务器IP
        weight 2
        HTTP_GET {
            url {
              path /
              status_code 200
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

重启查看是否有VIP和ipvsadm规则：

[root@lb1 keepalived]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:42:7b:e8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.231.150/24 brd 192.168.231.255 scope global eth0
    inet 192.168.231.154/32 scope global eth0
    inet6 fe80::20c:29ff:fe42:7be8/64 scope link
       valid_lft forever preferred_lft forever

[root@lb1 keepalived]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.231.154:80 wrr
  -> 192.168.231.152:80           Route   1      0          63        
  -> 192.168.231.153:80           Route   2      0          0        

那如果rs1和rs2都挂掉了，怎么让用户知道服务挂掉了呢？

在keepalived.conf中添加sorry_server:

virtual_server 192.168.231.154 80 {

    delay_loop 6

    lb_algo wrr

    lb_kind DR

    nat_mask 255.255.255.0

    protocol TCP

    sorry_server 127.0.0.1 80

    real_server 192.168.231.152 80 {

        weight 1

        HTTP_GET {

 url {

              path /

              status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

    real_server 192.168.231.153 80 {

        weight 2

        HTTP_GET {

            url {

              path /

              status_code 200

            }

            connect_timeout 3

            nb_get_retry 3

            delay_before_retry 3

        }

    }

}

lb1和lb2中yum install httpd -y，在启动httpd,并在/var/www/html/中添加响应的index.html提醒页面

停掉rs1和rs2的httpd服务测试：