Keepalivd+Haproxy高可用负载均衡

一、Keepalivd+Haproxy介绍

---

1、简介

虽然keepalived+LVS是最好的搭配，但是其配置相对比较复杂，对于小型的集群可以采用Keepalivd+Haproxy搭配，而且自带健康检查，配置起来相对比较简单。通过Keepalived实现对Haproxy的高可用(通过脚本对Haproxy的状态进行检查)，Haproxy实现负载均衡。

2、keepalived原理图：

二、实验环境

---

1、安装环境

软件网盘链接： https://pan.baidu.com/s/15i6LvJ1H6A1JrNvvr9mSRw 密码: vp73

selinux iptables off
Operating System: Red Hat Enterprise Linux Server 7.0 (Maipo)
Keepalived官方下载链接：http://www.keepalived.org/download.html
Haproxy官方下载链接：https://www.haproxy.org/download/

2、功能说明

主机名	IP	搭建服务	功能说明
server1(Master)	10.10.10.1	Keepalived+Haproxy	通过keepavlied实现对Haproxy(负载均衡)高可用
server2(Slave)	10.10.10.2	Keepalived+Haproxy	通过keepavlied实现对Haproxy(负载均衡)高可用

三、Haproxy安装及配置

---

1、编译安装Haproxy

[root@server1 ~]# useradd -u 1001 yy
[root@server1 ~]# echo "1"|passwd --stdin yy

[root@server1 ~]# vim /etc/security/limits.conf
yy              soft    nofile          65535
yy              hard    nofile          65535

[root@server1 ~]# su - yy
[yy@server1 ~]$ wget https://www.haproxy.org/download/1.5/src/haproxy-1.5.3.tar.gz
[yy@server1 ~]$ tar xf haproxy-1.5.3.tar.gz  
[yy@server1 ~]$ cd haproxy-1.5.3/
[yy@server1 haproxy-1.5.3]$ make TARGET=linux2628 ARCH=x86_64 PREFIX=/home/yy/haproxy      
[yy@server1 haproxy-1.5.3]$ make install PREFIX=/home/yy/haproxy 
[yy@server1 haproxy-1.5.3]$ cp /home/yy/haproxy-1.5.3/examples/haproxy.cfg /home/yy/haproxy
[yy@server1 haproxy-1.5.3]$ cp -r /home/yy/haproxy-1.5.3/examples/errorfiles/ /home/yy/haproxy

2、参数说明：

ARGET=linux26：内核版本，使用uname -r查看内核，如：2.6.18-371.el5，此时该参数就为linux26；内核大于2.6.28的用：TARGET=linux2628 ARCH=x86_64：系统位数

[yy@server1 haproxy-1.5.3]$ vim /home/yy/haproxy-1.5.3/README 

3、配置haproxy.cfg ：

[yy@server1 haproxy-1.5.3]$ vim /home/yy/haproxy/haproxy.cfg 
global
       log 127.0.0.1   local0
       log 127.0.0.1   local1 notice
       #log loghost    local0 info
       maxconn 4096
       #chroot /usr/share/haproxy              ###注释掉
       uid yy
       gid yy
       daemon
       pidfile /home/yy/haproxy/haproxy.pid    ###加入pid
       #debug
       #quiet

defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
       retries 3
       #redispatch
       maxconn 2000
       timeout connect  5000           ###下面3个地方都要修改，默认的不支持                                          
       timeout client  50000
       timeout server  50000

listen  appli1-rewrite 0.0.0.0:10001
       cookie  SERVERID rewrite
       balance roundrobin

listen  appli2-insert 0.0.0.0:10002
       option  httpchk
       balance roundrobin
       cookie  SERVERID insert indirect nocache
       server  inst1 192.168.114.56:80 cookie server01 check inter 2000 fall 3
       server  inst2 192.168.114.56:81 cookie server02 check inter 2000 fall 3
       capture cookie vgnvisitor= len 32

       option  httpclose               # disable keep-alive
       rspidel ^Set-cookie:\ IP=       # do not let this cookie tell our internal IP address

listen  appli3-relais 0.0.0.0:10003
       dispatch 192.168.135.17:80

listen  appli4-backup 0.0.0.0:10004
       option  httpchk /index.html
       option  persist
       balance roundrobin
       server  inst1 192.168.114.56:80 check inter 2000 fall 3
       server  inst2 192.168.114.56:81 check inter 2000 fall 3 backup
listen  ssl-relay 0.0.0.0:8443
       option  ssl-hello-chk
       balance source
       server  inst1 192.168.110.56:443 check inter 2000 fall 3
       server  inst2 192.168.110.57:443 check inter 2000 fall 3
       server  back1 192.168.120.58:443 backup

listen  appli5-backup 0.0.0.0:10005
       option  httpchk *
       balance roundrobin
       cookie  SERVERID insert indirect nocache
       server  inst1 192.168.114.56:80 cookie server01 check inter 2000 fall 3
       server  inst2 192.168.114.56:81 cookie server02 check inter 2000 fall 3
       server  inst3 192.168.114.57:80 backup check inter 2000 fall 3
       capture cookie ASPSESSION len 32
       timeout server 20000                   ###修改

       option  httpclose               # disable keep-alive
       option  checkcache              # block response if set-cookie & cacheable

       rspidel ^Set-cookie:\ IP=       # do not let this cookie tell our internal IP address

       errorloc        502     http://192.168.114.58/error502.html
       errorfile       503     /home/yy/haproxy/errorfiles/503.http                    ###修改为上面复制过去的路径

4、启动Haproxy

[yy@server1 haproxy-1.5.3]$ /home/yy/haproxy/sbin/haproxy -f /home/yy/haproxy/haproxy.cfg

5、查看是否启动成功

[yy@server1 ~]$ ps -aux|grep haproxy

yy         2070  0.0  0.2  12896  1368 ?        Ss   23:33   0:00 /home/yy/haproxy/sbin/haproxy -f /home/yy/haproxy/haproxy.cfg

yy         2269  0.0  0.2 112640   984 pts/0    R+   23:49   0:00 grep --color=auto haproxy

6、报错解决

（1）报错：

[yy@server1 haproxy-1.5.3]$ /home/yy/haproxy/sbin/haproxy -f /home/yy/haproxy/haproxy.cfg 
[WARNING] 183/232300 (2455) : [/home/yy/haproxy/sbin/haproxy.main()] Cannot raise FD limit to 8217.
[WARNING] 183/232300 (2455) : [/home/yy/haproxy/sbin/haproxy.main()] FD limit (1024) too low for maxconn=4096/maxsock=8217.  Please raise 'ulimit-n' to 8217 or more to avoid any trouble.

（2）解决方案：

[root@server1 ~]# vim /etc/security/limits.conf
yy              soft    nofile          65535
yy              hard    nofile          65535

重新登陆即可启动成功！！！

7、配置server2：

[root@server2 ~]# useradd -u 1001 yy
[root@server2 ~]# echo "1"|passwd --stdin yy
[yy@server1 ~]$ scp -r haproxy [email protected]:             ###server1中发送到server2中

[root@server1 ~]# vim /etc/security/limits.conf
yy              soft    nofile          65535
yy              hard    nofile          65535

[yy@server2 ~]$ /home/yy/haproxy/sbin/haproxy -f /home/yy/haproxy/haproxy.cfg   ###启动Haproxy

四、Keepalived安装

---

1、安装依赖包：

[root@server1 ~]# yum install -y openssl-devel

2、编译安装：

[root@server1 ~]# tar xf keepalived-2.0.4.tar.gz 
[root@server1 ~]#  cd keepalived-2.0.4/
[root@server1 keepalived-2.0.4]# ./configure --prefix=/home/yy/keepalived --with-init=SYSV
[root@server1 keepalived-2.0.4]# make && make install

3、可以看到所支持的：

4、查看是否安装成功

[root@server1 keepalived-2.0.4]# cd /usr/local/keepalived/
[root@server1 keepalived]# ll
total 0
drwxr-xr-x 2 root root 20 Jul  4 00:48 bin
drwxr-xr-x 5 root root 50 Jul  4 00:48 etc
drwxr-xr-x 2 root root 23 Jul  4 00:48 sbin
drwxr-xr-x 5 root root 37 Jul  4 00:48 share

5、进行链接：

[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/
[root@server1 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server1 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@server1 keepalived]# chmod +x /usr/local/keepalived/etc/rc.d/init.d/keepalived

6、修改配置：

[root@server1 keepalived]# vim /etc/keepalived/keepalived.conf       ###后面的注释掉既可
global_defs {
  notification_email {
    [email protected]
    [email protected]
    [email protected]
  }
  notification_email_from [email protected]
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  #vrrp_strict                    ###严格执行VRRP协议规范，否则VIP不同
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}

vrrp_instance VI_1 {
   state MASTER                     
   interface eth0                 ###网卡名，根据自己的网卡名进行设置
   virtual_router_id 51           ###路由ID
   priority 100                   ###优先级
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 1111
   }
   virtual_ipaddress {
       10.10.10.100               ###VIP（虚拟IP）
   }
}

7、启动测试：

[root@server1 keepalived]# systemctl restart keepalived
[root@server1 keepalived]# ps aux |grep keepalived
root      14645  0.0  0.1  42376   676 ?        Ss   00:56   0:00 keepalived -D
root      14647  0.0  0.2  42376  1136 ?        S    00:56   0:00 keepalived -D
root      14659  0.0  0.2 112640   984 pts/0    R+   00:57   0:00 grep --color=auto keepalived

这样keepalived就搭建好了，但是我们可以发现Haproxy和Keepalived没有任何的联系，因此我们许需要加入脚本来读取Haproxy的状态，从而达到高可用的作用！！！
五、加入脚本检测Haproxy状态

---

1、检查Haproxy状态脚本：

[root@server1 ~]# vim /etc/keepalived/chk_haproxy.sh
#!/bin/bash
Haproxy_Status=`ps -C haproxy --no-header |wc -l`
if [ $Haproxy_Status -eq 0 ];then
/home/yy/haproxy/sbin/haproxy -f /home/yy/haproxy/haproxy.cfg
sleep 3
if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then
/etc/init.d/keepalived stop
fi
fi
 
[root@server1 ~]# chmod +x /etc/keepalived/chk_haproxy.sh

2、修改Keepalived.conf配置：

global_defs {
 notification_email {
   [email protected]
   [email protected]
   [email protected]
  }
  notification_email_from [email protected]
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  #vrrp_strict
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}

vrrp_script check_haproxy {                       ###指定监控脚本
       script "/etc/keepalived/chk_haproxy.sh"
       interval 2
       weight 2
}

vrrp_instance VI_1 {
   state MASTER
   interface eth0
   virtual_router_id 51
   priority 100
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 1111
   }
   virtual_ipaddress {
       10.10.10.100
   }
   track_script {                ###监控此模块
           check_haproxy
   }
}

[root@server1 ~]# chkconfig --add keepalived         
[root@server1 ~]# chkconfig keepalived on              ###设置为开机自动启动

3、安装server2

（1）发送脚本及配置文件

[root@server1 ~]# scp -r /usr/local/keepalived/ [email protected]:/usr/local/
[root@server1 ~]# scp /etc/keepalived/chk_haproxy.sh [email protected]:/etc/keepalived/
[root@server2 ~]# cd /usr/local/keepalived/
[root@server2 keepalived]# ll
total 0
drwxr-xr-x 2 root root 20 Jul  4 01:37 bin
drwxr-xr-x 5 root root 50 Jul  4 01:37 etc
drwxr-xr-x 2 root root 23 Jul  4 01:37 sbin
drwxr-xr-x 5 root root 37 Jul  4 01:37 share

[root@server2 keepalived]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@server2 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/
[root@server2 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server2 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@server2 keepalived]# chmod +x /usr/local/keepalived/etc/rc.d/init.d/keepalived

（2）Keepalived.conf中修改：

[root@server2 keepalived]# systemctl restart keepalived
[root@server2 keepalived]# chkconfig --add keepalived
[root@server2 keepalived]# chkconfig keepalived on

六、测试

---

1、在server1中我们把脚本的启动给禁了

2、关闭掉Haproxy

3、发现VIP飘移到server2

4、恢复server1后VIP恢复

七、VIP漂移设置

---

默认设置为：VIP会漂移回Master；另外一个设置：当Master挂掉后，VIP漂移到Slave上，当Master恢复后，VIP不漂移回来，在优先级高（即之前得Master）把MASTER修改为BACKUP（虽然2个都为BACKUP，但是根据优先级选择出MASTER即server1），并加入nopreempt参数！！！

1、配置keepalived.conf（server1）

修改配置Master（server1）配置，slave（server2）配置文件不做修改！！！

[root@server1 ~]# /etc/init.d/keepalived restart

2、测试结果

当Master（server1）的haproxy挂掉后，VIP漂移到Slave（server2）上，在恢复Master（server1）的haproxy后，VIP并不会漂移回Master（server1）！！！