HAProxy双机高可用方案之HAProxy+Keepalived

 HAProxy双机高可用方案之HAProxy+Keepalived

环境：centos 5.4 x64
软件版本：haproxy-1.3.26.tar.gz 、keepalived-1.2.7.tar.gz
网络拓扑：
                  192.168.48.143
             +-----------VIP----------+  
             |           eth1         |
             |                        |
           Master                   Backup
        192.168.48.139            192.168.48.140
        +----------+             +----------+
        | HAProxy  |    eth0     | HAProxy  |
        |keepalived|-------------|keepalived|
        +----------+1.139   1.140+----------+
             | 
             v 
        +--------+
        |        |     
        |        |     
        v        v  eth1   
    +------+  +------+ 
    | WEB1 |  | WEB2 | 
    +------+  +------+ 
    48.141     48.142

haproxy安装
主页：http://haproxy.1wt.eu/
下载地址：http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.26.tar.gz
[root@localhost tools]# wget http://haproxy.1wt.eu/download/1.3/src/haproxy-1.3.26.tar.gz
[root@localhost tools]# tar zxf haproxy-1.3.26.tar.gz
[root@localhost tools]# cd haproxy-1.3.26
[root@localhost haproxy-1.3.26]# make TARGET=linux26 PREFIX=/usr/local/haproxy
[root@localhost haproxy-1.3.26]# make install PREFIX=/usr/local/haproxy
[root@localhost haproxy-1.3.26]# cd /usr/local/haproxy/
[root@localhost haproxy]# mkdir conf
主备配置一样
[root@localhost haproxy]# cat haproxy.cfg
global 
        log 127.0.0.1   local0 
        maxconn 65535 
        chroot /usr/local/haproxy 
        uid 99 
        gid 99 
        daemon 
        nbproc 2
        #pidfile /usr/local/haproxy/haproxy.pid 
        debug 

defaults 
         log     127.0.0.1       local3 
         mode   http 
         option httplog 
         option httpclose 
         option dontlognull 
         option forwardfor 
         option redispatch 
         retries 2 
         maxconn 2000 
#         balance roundrobin
#         stats   uri     /haproxy-stats 
         contimeout      5000 
         clitimeout      50000 
         srvtimeout      50000 

frontend web_proxy 
         bind *:80
         mode http
         option httplog 
         log global  
         default_backend phppool  

backend phppool
        mode http
        balance source
        option httpchk HEAD /
        server web1 192.168.48.141:80  check inter 2000
        server web2 192.168.48.142:80  check inter 2000       

listen  web_ha
        bind 0.0.0.0:8080
        mode http 
        stats   uri     /haproxy-stats 
        stats hide-version
#        stats realm Haproxy\statistics
        stats auth gao:gao   # 用户名/密码
         
haproxy启动脚本
[root@ha139 sbin]# cat haproxy.sh
#!/bin/sh 
# chkconfig 35 on 
# description: HAProxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments. 

# Source function library. 
if [ -f /etc/init.d/functions ]; then 
  . /etc/init.d/functions 
elif [ -f /etc/rc.d/init.d/functions ] ; then 
  . /etc/rc.d/init.d/functions 
else 
  exit 0 
fi 

# Source networking configuration. 
. /etc/sysconfig/network 

# Check that networking is up. 
[ ${NETWORKING} = "no" ] && exit 0 

[ -f /usr/local/haproxy/conf/haproxy.cfg ] || exit 1 

RETVAL=0

start() { 
  /usr/local/haproxy/sbin/haproxy -c -q -f /usr/local/haproxy/conf/haproxy.cfg 
  if [ $? -ne 0 ]; then 
    echo "Errors found in configuration file." 
    return 1 
  fi 

  echo -n "Starting HAproxy: " 
  daemon /usr/local/haproxy/sbin/haproxy -D -f /usr/local/haproxy/conf/haproxy.cfg -p /var/run/haproxy.pid 
  RETVAL=$? 
  echo 
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/haproxy 
  return $RETVAL 
} 

stop() { 
  echo -n "Shutting down HAproxy: " 
  killproc haproxy -USR1 
  RETVAL=$? 
  echo 
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/haproxy 
  [ $RETVAL -eq 0 ] && rm -f /var/run/haproxy.pid 
  return $RETVAL 
} 

restart() { 
  /usr/local/haproxy/sbin/haproxy -c -q -f /usr/local/haproxy/conf/haproxy.cfg 
  if [ $? -ne 0 ]; then 
    echo "Errors found in configuration file, check it with 'haproxy check'." 
    return 1 
  fi 
  stop 
  start 
} 

check() { 
  /usr/local/haproxy/sbin/haproxy -c -q -V -f /usr/local/haproxy/conf/haproxy.cfg 
} 

rhstatus() { 
  status haproxy 
} 

condrestart() { 
  [ -e /var/lock/subsys/haproxy ] && restart || : 
} 

# See how we were called. 
case "$1" in 
  start) 
    start 
    ;; 
  stop) 
    stop 
    ;; 
  restart) 
    restart 
    ;; 
  reload) 
    restart 
    ;; 
  condrestart) 
    condrestart 
    ;; 
  status) 
    rhstatus 
;; 
  check) 
    check 
    ;; 
  *) 
    echo $"Usage: haproxy {start|stop|restart|reload|condrestart|status|check}" 
    RETVAL=1
esac 
exit $RETVAL
         
haproxy加日志：
编辑/etc/syslog.conf文件， 添加内容如下：
local3.*        /var/log/haproxy.log
local0.*        /var/log/haproxy.log
编辑/etc/sysconfig/syslog文件，修改内容如下：
SYSLOGD_OPTIONS="-r -m 0"
然后重启syslog服务，命令如下：
service syslog restart

keepalived安装
主页:www.keepalived.org
[root@localhost tools]# wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
[root@ha tools]# tar zxf keepalived-1.2.7.tar.gz
[root@ha tools]# cd keepalived-1.2.7
[root@ha keepalived-1.2.7]# ./configure --prefix=/usr/local/keepalived
过程中出错：
configure: error:
  !!! OpenSSL is not properly installed on your system. !!!
  !!! Can not include OpenSSL headers files.            !!!
因为openssl没有安装好
yum install -y openssl-devel
[root@ha keepalived-1.2.7]# make
[root@ha keepalived-1.2.7]# make install

keepalived做成服务模式启动
[root@ha keepalived-1.2.7]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@ha keepalived-1.2.7]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@ha keepalived-1.2.7]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@ha keepalived-1.2.7]# mkdir /etc/keepalived
[root@ha keepalived-1.2.7]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
主的配置文件
[root@ha keepalived-1.2.7]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_1
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
!state MASTER
state BACKUP
priority 100
advert_int 1
virtual_router_id 50
garp_master_delay 1
interface eth1
nopreempt
authentication {
auth_type PASS
auth_pass Kxiaokk345Pix
}
track_interface {
eth0
eth1
}
virtual_ipaddress {
192.168.48.143
}
track_script {
chk_haproxy
}
! Configuration File for keepalived
global_defs {
router_id LVS_1
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 2
weight 2
}
vrrp_instance VI_1 {
!state MASTER
state BACKUP
priority 100
advert_int 1
virtual_router_id 50
garp_master_delay 1
interface eth1
nopreempt
authentication {
auth_type PASS
auth_pass Kxiaokk345Pix
}
track_interface {
eth0
eth1
}
virtual_ipaddress {
192.168.48.143
}
track_script {
chk_haproxy
}
!notify_master "/etc/keepalived/mailnotify.py master"
!notify_backup "/etc/keepalived/mailnotify.py backup"
!notify_fault "/etc/keepalived/mailnotify.py fault"
}
}

配置说明：
! Configuration File for keepalived
global_defs {
     notification_email { #发生事件切换的时候，发送的邮箱，可以有多个，每行一个
     [email protected]
     }
     notification_email_from [email protected]  #指定发件人
     smtp_server 127.0.0.1  #指定smtp服务器地址
     smtp_connect_timeout 30    #指定smtp连接超时时间
     router_id LVS_2  #运行keepalived机器的一个标识主备不能相同
     }

#vrrp_script区域定义脚本名字和脚本执行的间隔和脚本执行的优先级变更
vrrp_script check_haproxy {
#定义监控nginx的命令，命令的$?返回1就改变优先级。
        script "/etc/keepalived/check_haproxy.sh" 
        interval 5     #脚本执行间隔
        weight -10   #脚本结果1导致的优先级变更：10表示优先级+10；-10则表示优先级-10
        fall   2     # 检测几次返回1为失败
        rise   1     # 检测几次返回0成功为ok
        }

#VIP
vrrp_instance VI_1 {    #监控多个网段的实例
     state BACKUP     #全部服务器都设为BACKUP，让优先级高的自动为master
#    dont track primary     #忽略VRRP的interface错误
     track_interface {
     eth0                 #跟踪接口，设置额外的监控，里面任意一块网卡出现问题，都会进入故障(FAULT)状态
     eth1
     }
     interface eth0         #实例绑定的网卡
     mcast_src_ip 192.168.48.139 #指定主或备的eth0的Ip    
     virtual_router_id 51   #这里非常重要，相同的VRID为一个组
     priority 100           #优先级，
     advert_int 1           #检查间隔，默认1s
#     nopreempt            #不设置抢占，优先级高就做master，当主恢复时不抢占。nopreempt  //master 设置为BACKUP ，通过priority来判断哪成为主，设置nopreempt 是可以在MASTER 出现问题修复好后不抢用vip，直接充当slave的角色。这样，可以确保session不会丢失
     authentication {       #认证
     auth_type PASS         #认证的方式，支持PASS和AH
     auth_pass 123         #认证的密码
     }
    track_interface {      #设置监听本机的接口
    eth0
    eth1
    }     
     virtual_ipaddress {     #指定漂移地址（VIP)
     192.168.48.143         #如果有多个VIP，继续换行填写
     }
    track_script {     #执行定义的命令
    check_haproxy
    }
}

haproxy检测脚本
[root@ha139 keepalived]# cat check_haproxy.sh
#!/bin/bash
#
# desc: check haproxy service
#
A=`ip address show eth1 | grep 192.168.48.143 | wc -l`
B=`ps -C haproxy --no-heading | wc -l`

if [ $A -eq 1 ]; then
if [ $B -eq 0 ]; then
/usr/local/haproxy/sbin/haproxy.sh start
sleep 3
fi
fi

主或备服务要求抢占：不要配置nopreempt
track_interface配置监听没有配置时，如果是多网卡，只能监控之中的一张网卡。
track_interface配置同时监听本机的eth0,eth1时，两张网卡之中的一张有问题，vip就会漂移
track_interface配置分别监听本机的eth0或eth1时，只是监听指定的本机网卡有问题时，vip才会漂移

主不要求抢占：加nopreempt，做到主不抢占，优先级高就做master，当主恢复时不抢占。//master 设置为BACKUP ，通过priority来判断哪成为主，设置nopreempt 是可以在MASTER 出现问题修复好后不抢用vip，直接充当slave的角色。这样，可以确保session不会丢失
抢占问题处理：http://bbs.linuxtone.org/thread-6788-1-1.html

高可用（keepalived+haproxy）切换测试：（两台机）
主或备断eth0或eth1网卡，看能否漂移，是否抢占。
主或备关keepaviled，看能否漂移。

并发测试
webbench最多可以模拟3万个并发连接去测试网站的负载能力，个人感觉要比Apache自带的ab压力测试工具好，安装使用也特别方便。
wget http://blog.s135.com/soft/linux/webbench/webbench-1.5.tar.gz
tar zxvf webbench-1.5.tar.gz
cd webbench-1.5
make &&mkdir -p /usr/local/man&& make install
使用：
webbench -c 500 -t 30 http://192.168.48.143
参数说明：-c表示并发数，-t表示时间(秒)

haproxy参考资料
http://network.51cto.com/art/201110/295955.htm  HAProxy双机高可用方案之HAProxy+Keepalived(1)
http://dngood.blog.51cto.com/446195/738634 haproxy 安装与配置 （有配置说明）
http://os.51cto.com/art/201202/317741_2.htm HAProxy配置语法及实例(3）（有配置说明）
http://hi.baidu.com/gaolongquan/item/51a8f6c68f44ad61f7c95db0  HAProxy+Keepalived
http://xukaizijian.blog.163.com/blog/static/17043311920115283358709/   web] Haproxy配置（有配置说明）
http://blog.51cto.com/zt/232     HAProxy 能够做些什么？51cto专题

keepalived参考资料
http://xzregg.blog.51cto.com/3829250/976793 ngnix+keepalived  实现N主高可用负载均衡web群集 (有配置说明)
http://www.jb51.net/os/RedHat/55468.html  haproxy+keepalived  负载均衡之主备切换
http://bbs.linuxtone.org/thread-6788-1-1.html                 vip抢占问题处理
http://www.tumblr.com/tagged/keepalived                       HAProxy + KeepAlived构建高可用的反向代理系统(带并发测试)

http://blog.s135.com/post/288/                                并发测试