Haproxy、Keepalived双主高可用负载均衡

通过Haproxy+Keepalived实现双主模式，以便充分利用了服务器资源。测试环境如下：
vip1=192.168.36.210（a.3evip.cn）
vip2=192.168.36.200（b.3evip.cn）
haproxy01=192.168.36.102
haproxy02=192.168.36.207
web1：192.168.36.54 部署a.3evip.cn和b.3evip.cn

web2：192.168.36.189 部署a.3evip.cn和b.3evip.cn

架构图：

一、安装配置web服务

为了模拟web服务，使用apache搭建web服务

1.安装apache

yum -y install httpd

2.配置：

vi /etc/httpd/conf/httpd.conf

#增加监听端口

Listen 8080
Listen 8081

#配置虚拟主机

<VirtualHost *:8080>
    ServerAdmin root@slave-a
    DocumentRoot /var/www/html/8080/
    ServerName a.3evip.cn
    ErrorLog logs/dummy-host.example.com-error_log
    CustomLog logs/dummy-host.example.com-access_log common
</VirtualHost>
<VirtualHost *:8081>
    ServerAdmin root@slave-a
    DocumentRoot /var/www/html/8081/
    ServerName b.3evip.cn
    ErrorLog logs/dummy-host.example.com-error_log
    CustomLog logs/dummy-host.example.com-access_log common
</VirtualHost>

在 /var/www/html/8080/和/var/www/html/8081/创建访问页面

echo "server:192.168.36.54:8080 ==> a.3evip.cn"  > index.html

echo "server:192.168.36.54:8081 ==> b.3evip.cn"  > index.html

另一台web：192.168.36.189配置类似

3.启动服务

service httpd start

若出现如下错误：

Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.36.54 for ServerName
(13)Permission denied: make_sock: could not bind to address [::]:8081
(13)Permission denied: make_sock: could not bind to address 0.0.0.0:8081
no listening sockets available, shutting down
Unable to open logs
[FAILED]

由于权限不足导致，selinux有3个模式
Enforcing：强制模式
Permissive：警告模式
Disabled：关闭模式

vi /etc/selinux/config
#SELINUX=enforcing #注释掉
#SELINUXTYPE=targeted #注释掉
SELINUX=disabled #增加
:wq!  #保存退出
setenforce 0 #使配置立即生效

关闭防火墙

启动后，访问看看是否正常：http://192.168.36.54:8080/

显示：server:192.168.36.54:8080 ==>a.3evip.cn

二、安装配置haproxy（haproxy用户）

1.安装忽略可以参考：http://blog.csdn.net/zhu_tianwei/article/details/41117323

2,.配置

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        maxconn 1000
        daemon

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        maxconn 1000
        timeout connect 5000
        timeout client  50000
        timeout  server 50000
        balance roundrobin

listen admin_stats  
        bind 0.0.0.0:1080               #设置Frontend和Backend的组合体，监控组的名称，按需要自定义名称  
        mode http                       #http的7层模式  
        option httplog                  #采用http日志格式  
        maxconn 10                                              #默认的最大连接数  
        stats refresh 30s               #统计页面自动刷新时间  
        stats uri /stats                #统计页面url  
        stats realm XingCloud\ Haproxy  #统计页面密码框上提示文本  
        stats auth admin:admin     #设置监控页面的用户和密码:admin,可以设置多个用户名  
        stats auth  Frank:Frank   #设置监控页面的用户和密码：Frank  
        stats hide-version              #隐藏统计页面上HAProxy的版本信息  
        stats  admin if TRUE       #设置手工启动/禁用，后端服务器(haproxy-1.4.9以后版本)  

frontend www
        bind *:8888
        acl aweb hdr_beg(host) -i a.3evip.cn #ACL规则若配置hdr(host) 则访问非80端口为503错误
        acl bweb hdr_beg(host) -i b.3evip.cn
        use_backend awebserver if aweb
        use_backend bwebserver if bweb
backend awebserver
        mode http
        balance   roundrobin
        option  httpchk /index.html
        server     web01 192.168.36.54:8080  check inter 2000 fall 3 weight 10
        server     web02 192.168.36.189:8080  check inter 2000 fall 3 weight 10

backend bwebserver
        mode http
        balance   roundrobin
        option  httpchk /index.html
        server     web01 192.168.36.54:8081  check inter 2000 fall 3 weight 10
        server     web02 192.168.36.189:8081  check inter 2000 fall 3 weight 10

2台机器配置一样

3.启动

./sbin/haproxy -f ./conf/haproxy.cfg

可以通关控制台查看配置。

三、安装配置keepalived(root用户)

1.下载keepalived，解压

wget http://www.keepalived.org/software/keepalived-1.2.13.tar.gz

tar -zxvf keepalived-1.2.13.tar.gz 

2.安装

cd keepalived-1.2.13

./configure  #配置，必须看到以下提示，说明配置正确，才能继续安装
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
Use VRRP Framework       : Yes

make 

make install

3.配置启动

cp /usr/local/etc/sysconfig/keepalived  /etc/sysconfig/
mkdir /etc/keepalived
cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/sbin/keepalived /usr/sbin/
cp /usr/local/etc/rc.d/init.d/keepalived  /etc/rc.d/init.d/
chmod +x /etc/rc.d/init.d/keepalived  
chkconfig keepalived on 

4.配置keepalived

haproxy01配置：

vi /etc/keepalived/keepalived.conf

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from keepalived@master-c
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script chk_http_port {
   script "/opt/check_haproxy.sh"
   interval 2
   weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
        chk_http_port #执行监控服务
    }
    virtual_ipaddress {
        192.168.36.210
    }
	notify_master "/opt/clean_arp.sh  192.168.36.210"  #更新虚拟服务器（VIP）地址的arp记录到网关
}


vrrp_instance VI_2 {
    state BACKUP
    interface eth0
    virtual_router_id 52
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    
   virtual_ipaddress {
        192.168.36.200
    }
	notify_master "/opt/clean_arp.sh  192.168.36.200"  #更新虚拟服务器（VIP）地址的arp记录到网关
}

haproxy02配置：

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from keepalived@master-c
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_script chk_http_port {
   script "/opt/check_haproxy.sh"
   interval 2
   weight 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    track_script {
        chk_http_port #执行监控服务
    }
    virtual_ipaddress {
        192.168.36.210
    }
	notify_master "/opt/clean_arp.sh  192.168.36.210"  #更新虚拟服务器（VIP）地址的arp记录到网关
}

vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    
   virtual_ipaddress {
        192.168.36.200
    }
	notify_master "/opt/clean_arp.sh  192.168.36.200"  #更新虚拟服务器（VIP）地址的arp记录到网关
}

5.检查脚本， 为了防止haproxy服务关闭导致keepalived不自动切换。

vi  /opt/check_haproxy.sh

#!/bin/bash
if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then
     su - haproxy -c "/home/haproxy/haproxy/sbin/haproxy -f /home/haproxy/haproxy/conf/haproxy.cfg"
fi
sleep 2
if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then
       /etc/init.d/keepalived stop
fi

chmod +x /opt/check_haproxy.sh

6.设置更新虚拟服务器（VIP）地址的arp记录到网关脚本

vi  /opt/clean_arp.sh

#!/bin/sh
VIP=$1
GATEWAY=192.168.21.2 #网关地址
/sbin/arping -I eth0 -c 5 -s $VIP $GATEWAY &>/dev/null

7.系统内核优化

在两台HAProxy服务器上分别执行以下命令

echo 1024 60999 > /proc/sys/net/ipv4/ip_local_port_range
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets
echo 262144 > /proc/sys/net/ipv4/tcp_max_orphans
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 0 > /proc/sys/net/ipv4/tcp_dsack

8，启动keepalived服务

service keepalived start  (stop/status)

9.查看vip生效情况

在两台HAProxy服务器：192.168.36.102、192.168.36.217上执行命令：ip addr

192.168.36.102：

[root@master-c keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:f2:66:46 brd ff:ff:ff:ff:ff:ff
    inet 192.168.36.102/24 brd 192.168.36.255 scope global eth0
    inet 192.168.36.210/32 scope global eth0
    inet6 fe80::20c:29ff:fef2:6646/64 scope link 
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 76:ad:3f:58:42:ce brd ff:ff:ff:ff:ff:ff

可以看出现在VIP：192.168.36.210指向192.168.36.102。

192.168.36.217：

[root@master-a home]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:31:ca:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.36.217/24 brd 192.168.36.255 scope global eth0
    inet 192.168.36.200/32 scope global eth0
    inet6 fe80::20c:29ff:fe31:ca70/64 scope link 
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 1e:0e:40:13:89:7b brd ff:ff:ff:ff:ff:ff

可以看出现在VIP：192.168.36.200指向192.168.36.217。

四、查看结果

访问http://a.3evip.cn:8888/ 和http://b.3evip.cn:8888/

关闭一台apache或haproxy测试一下。

关闭192.168.36.217的haproxy可以看出VIP都指向192.168.36.102

[root@master-c keepalived]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:0c:29:f2:66:46 brd ff:ff:ff:ff:ff:ff
   inet 192.168.36.102/24 brd 192.168.36.255 scope global eth0
    inet 192.168.36.210/32 scope global eth0
    inet 192.168.36.200/32 scope global eth0
    inet6 fe80::20c:29ff:fef2:6646/64 scope link 
       valid_lft forever preferred_lft forever
3: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 76:ad:3f:58:42:ce brd ff:ff:ff:ff:ff:ff