LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)+ldirector健康检查及转发)

LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)ldirector健康检查及转发)
        很久没有正儿八经的写技术博客了,希望各位看官海涵。接下来,直接进入主题。
为了使用一个名叫LCMC(好像叫Linux Cluster Managerment Console)的lvs图形化管理界面,我这次lvs的主角确定为heartbeat(基本停止开发,只进行维护)。之前在公司的生产环境进行试验,所以LCMC可以直接连接公网进行wget和yum安装,所以一切都显得非常便捷,基本一个主备模式的lvs只需要几分钟就轻松搞定。为了防止试验环境对生产网的干扰,只得把主机放到测试环境中去。由于测试环境无法直接访问公网,所以花了一天把所需的安装包都收集好。好了,开始安装。
1.系统安装
操作系统:RHEL5.8 X86_64 版本
内核版本: 2.6.18-308.el5

2.配置

借图一用

LVS环境搭建及图形化管理(Pacemaker管理+heartbeat心跳+ipvs(DR)+ldirector健康检查及转发)_第1张图片

服务器及IP为
虚拟IP(VIP) :                              eth0:192.168.29.100                  eth1:xxx.xxx.xxx.45/46
主LVS(ds1.lvs.stefentang):        eth0:192.168.29.45                  eth1:xxx.xxx.xxx.45
备LVS(ds2.lvs.stefentang):        eth0:192.168.29.46                  eth1:xxx.xxx.xxx.46
实机1(rs1.lvs.stefentang):       eth0:192.168.29.47(45)  eth0:无
实机2(rs2.lvs.stefentang):       eth0:192.168.29.48(46)  eth0:无

3.软件安装
为了便于日后集成到操作系统的安装包中去,都采用rpm包安装(所需安装包稍后上传)
在ds1和ds2上分别安装:
[root@ds1 ~]# rpm -ivh --force *.rpm
[root@ds2 ~]# rpm -ivh --force *.rpm
执行结果:
Preparing...                ########################################### [100%]
   1:libtool-ltdl           ########################################### [  2%]
   2:net-snmp-libs          ########################################### [  4%]
   3:libesmtp               ########################################### [  5%]
   4:openhpi-libs           ########################################### [  7%]
   5:OpenIPMI-libs          ########################################### [  9%]
   6:lm_sensors             ########################################### [ 11%]
   7:libqb                  ########################################### [ 13%]
   8:imake                  ########################################### [ 14%]
   9:drbd83                 ########################################### [ 16%]
  10:bzip2                  ########################################### [ 18%]
  11:pacemaker-debuginfo    ########################################### [ 20%]
  12:openais-debuginfo      ########################################### [ 21%]
  13:heartbeat-debuginfo    ########################################### [ 23%]
  14:glib2-devel            ########################################### [ 25%]
  15:cluster-glue-debuginfo ########################################### [ 27%]
  16:autoconf               ########################################### [ 29%]
  17:zlib-devel             ########################################### [ 30%]
  18:libxml2-devel          ########################################### [ 32%]
  19:automake               ########################################### [ 34%]
  20:libtool-ltdl-devel     ########################################### [ 36%]
  21:perl-TimeDate          ########################################### [ 38%]
  22:openib                 ########################################### [ 39%]
  23:libsepol-devel         ########################################### [ 41%]
  24:libselinux-devel       ########################################### [ 43%]
  25:libgpg-error-devel     ########################################### [ 45%]
  26:libgcrypt-devel        ########################################### [ 46%]
  27:keyutils-libs-devel    ########################################### [ 48%]
  28:e2fsprogs-devel        ########################################### [ 50%]
  29:krb5-devel             ########################################### [ 52%]
  30:openssl-devel          ########################################### [ 54%]
  31:libesmtp-devel         ########################################### [ 55%]
  32:libxslt-devel          ########################################### [ 57%]
  33:libtool                ########################################### [ 59%]
  34:bzip2-devel            ########################################### [ 61%]
  35:kmod-drbd83            ########################################### [ 63%]
  36:libqb-devel            ########################################### [ 64%]
  37:resource-agents-debugin########################################### [ 66%]
  38:cluster-glue-libs      ########################################### [ 68%]
  39:cluster-glue           ########################################### [ 70%]
  40:libibverbs             ########################################### [ 71%]
  41:resource-agents        ########################################### [ 73%]
  42:corosync               ########################################### [ 75%]
  43:librdmacm              ########################################### [ 77%]
  44:corosynclib            ########################################### [ 79%]
  45:cluster-glue-libs-devel########################################### [ 80%]
  46:libnes                 ########################################### [ 82%]
  47:corosynclib-devel      ########################################### [ 84%]
  48:heartbeat-libs         ########################################### [ 86%]
  49:pacemaker-libs         ########################################### [ 88%]
  50:openaislib             ########################################### [ 89%]
  51:heartbeat              ########################################### [ 91%]
  52:openais                ########################################### [ 93%]
  53:pacemaker              ########################################### [ 95%]
  54:heartbeat-devel        ########################################### [ 96%]
  55:openaislib-devel       ########################################### [ 98%]
  56:pacemaker-libs-devel   ########################################### [100%]
4.配置HB(/etc/ha.d/ha.cf)可以由LCMC来完成,也可以手动编辑。
#集群中的节点不会自动加入
autojoin    none

#heartbeat会记录debug日志,如果启用use_logd,则此选项会被忽略
debugfile   /var/log/ha-debug

#记录所有non-debug消息,如果启用use_logd,则此选项会被忽略
logfile    /var/log/ha-log

#告诉heartbeat记录那些syslog
logfacility   local0

#指定两个心跳检测包的时间间隔
keepalive 1

#多久以后心跳检测决定集群中的node已经挂掉
deadtime   30

#心跳包检测的延时事件,如果延时,只是往日志中记录warning日志,并不切换服务
warntime  10

#在heartbeat启动后,在多长时间内宣布node是dead状态,因为有时候系统启动后,网络还需要一段时间才能启动
initdead  120

#如果udpport指令在bcast ucast指令的前面,则使用哪个端口进行广播,否则使用默认端口
udpport   694

#设置使用哪个网络接口发送UDP广播包,可以设置多个网络接口
bcast  eth1 eth1

#设置在哪个网络接口进行多播心跳检测
mcast eth0 239.192.0.0 694 1 0

#设置使用哪个网络接口进行UDP单播心跳检测,在ds2上为10.0.0.45
#ucast  eth1 10.0.0.46

#在主节点的服务恢复后,是否把从节点的服务切换回来
auto_failback off

#告诉集群中有哪些节点,node名称必须是uname -n显示出来的名称,可以在一个node中设置多个节点,也可以多次设置node,每一个在集群中的node都必须被列出来
node ds1.lvs.stefentang ds2.lvs.stefentang

#设置ping节点,ping节点用来检测网络连接
#ping 10.0.0.254

#开启Pacemaker cluster manager,因为历史原因,次选项默认是off,但是应该保持该选项值为respawn。在设置为respawn默认自动使用以下配置
#pacemaker  respawn
crm respawn
compression bz2
compression_threshold 20
traditional_compression on
#默认配置文件中下面还有很多选项,由于暂时用不到所以暂时忽略

5.配置/etc/ha.d/authkeys(LCMC完成,authkeys文件属性必须是600,否则日志会报错。)

## generated by drbd-gui
auth 1
1 sha1 9AO5FmCl9RwBZZLIepNERwSbMMsmgeiF

6.配置ldirectord(/etc/ha.d/ldirectord.cf)可以手动编辑,也可以拷贝ldirectord的share目录下的文件

#
# Sample ldirectord configuration file to configure various virtual services.
#
# Ldirectord will connect to each real server once per second and request
# /index.html. If the data returned by the server does not contain the
# string "Test Message" then the test fails and the real server will be
# taken out of the available pool. The real server will be added back into
# the pool once the test succeeds. If all real servers are removed from the
# pool then localhost:80 is added to the pool as a fallback measure.

# Global Directives
checktimeout=3
checkinterval=1
#fallback=127.0.0.1:80
autoreload=yes
logfile="/var/log/ldirectord.log"
#logfile="local0"
emailalert="[email protected]"
emailalertfreq=3600
emailalertstatus=all
quiescent=yes

# Sample for an http virtual service
virtual=192.168.29.100:80
    real=192.168.29.47(45):80 gate
    real=192.168.29.48(46):80 gate
#    real=192.168.6.6:80 gate
#     fallback=127.0.0.1:80 gate
    service=http
    request="health/health.html"
    receive="running status: available"
#    virtualhost=some.domain.com.au
    scheduler=rr
    #persistent=600
    #netmask=255.255.255.255
    protocol=tcp
      checktype=negotiate
      checkport=80
     request="health/health.html"
     receive="running status: available"
     virtualhost=ds1/ds2.lvs.stefentang

测试心跳是否正常: tcpdump -i eth1 -n -p udp port 694

7.安装http服务器

lvs配置完后,配置真实服务器
[root@rs1 ~]# rpm -ivh /mnt/Server/apr-1.2.7-11.el5_6.5.x86_64.rpm
[root@rs1 ~]# rpm -ivh /mnt/Server/postgresql-libs-8.1.23-1.el5_7.3.x86_64.rpm
[root@rs1 ~]# rpm -ivh /mnt/Server/apr-util-1.2.7-11.el5_5.2.x86_64.rpm
[root@rs1 ~]# rpm -ivh /mnt/Server/mailcap-2.1.23-1.fc6.noarch.rpm
[root@rs1 ~]# rpm -ivh /mnt/Server/httpd-2.2.3-63.el5.x86_64.rpm
[root@rs1 ~]# service httpd start

http服务80端口根目录添加一个health/health.html,内容为

[47(45)/48(46).DS1/2]running status: available

然后浏览器输入http://RIP/health/health.html ,能正确看到内容。
配置实机环路地址 lo:0(/etc/init.d/lvs_rs)
#!/bin/bash
#chkconfig: 2345 80 90
#description:lvs real server set lo:0 VIP address
VIP=192.168.29.100  #virtual ip
    case "$1" in
    start)
    echo " start LVS of REALServer"
    /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    ;;
    stop)
    /sbin/ifconfig lo:0 down
    echo "close LVS Directorserver"
    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
    ;;
    *)
    echo "Usage: $0 {start|stop}"
    exit 1
    esac
然后
chmod u+x lvs_rs
/etc/inid.d/lvs_rs start

验证ldirectord是否做健康检查可以尝试查看httpd的access日志文件如(访问源从29.45变成29.46期间LVS做了一次切换):

192.168.29.45 - - [08/Aug/2013:07:32:16 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"
192.168.29.45 - - [08/Aug/2013:07:32:17 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"
192.168.29.46 - - [08/Aug/2013:07:33:12 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"
192.168.29.46 - - [08/Aug/2013:07:33:13 +0800] "GET /health/health.html HTTP/1.1" 200 372 "-" "libwww-perl/5.805"

开始怎么也想不通,为啥切换动作差不多花了整整55秒,后来发现DNS配置有问题,干脆把DNS解析注释掉(修改/etc/resolv.conf文件):

192.168.29.45 - - [17/Aug/2013:02:46:37 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"
192.168.29.45 - - [17/Aug/2013:02:46:38 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"
192.168.29.46 - - [17/Aug/2013:02:46:55 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"
192.168.29.46 - - [17/Aug/2013:02:46:59 +0800] "GET /health/health.html HTTP/1.1" 200 364 "-" "libwww-perl/5.805"

并且修改heartbeat的以下三个参数后,就缩短到17秒,估计也不能改得太小,造成误判。

warntime  5
deadtime  10
initdead 30


接下来都可以在LCMC端操作了,稍后配图。

8.查看配置结果

[root@ds1 ~]# crm_mon
Attempting connection to the cluster...
============
Last updated: Thu Aug  8 06:59:53 2013
Stack: Heartbeat
Current DC: ds1.lvs.stefentang (a4f7c50b-823d-4434-bb1d-850aa18918b2) - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
2 Resources configured.
============

Online: [ ds1.lvs.stefentang ds2.lvs.stefentang ]

res_IPaddr2_VIP (ocf::heartbeat:IPaddr2):       Started ds1.lvs.stefentang
res_ldirectord_1        (ocf::heartbeat:ldirectord):    Started ds1.lvs.stefentang

[root@ds2 ~]# crm_mon
Attempting connection to the cluster...
============
Last updated: Thu Aug  8 07:01:15 2013
Stack: Heartbeat
Current DC: ds1.lvs.stefentang (a4f7c50b-823d-4434-bb1d-850aa18918b2) - partition with quorum
Version: 1.0.12-unknown
2 Nodes configured, unknown expected votes
2 Resources configured.
============

Online: [ ds1.lvs.stefentang ds2.lvs.stefentang ]

res_IPaddr2_VIP (ocf::heartbeat:IPaddr2):       Started ds1.lvs.stefentang
res_ldirectord_1        (ocf::heartbeat:ldirectord):    Started ds1.lvs.stefentang

最后用ab并发1000,10000连接,压了一下,查看了分发情况如下:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.29.100:http rr
  -> 192.168.29.45:http           Local   1      3          5071
  -> 192.168.29.46:http           Route   1      4          4930
Server Software:        Apache/2.2.3
Server Hostname:        192.168.29.100
Server Port:            80

Document Path:          /is/checkNumber.do
Document Length:        1336 bytes

Concurrency Level:      1000
Time taken for tests:   68.166 seconds
Complete requests:      10000
Failed requests:        9883
   (Connect: 0, Receive: 0, Length: 9883, Exceptions: 0)
Write errors:           0
Total transferred:      16318053 bytes
HTML transferred:       13448053 bytes
Requests per second:    146.70 [#/sec] (mean)
Time per request:       6816.590 [ms] (mean)
Time per request:       6.817 [ms] (mean, across all concurrent requests)
Transfer rate:          233.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    7  42.7      5    3014
Processing:    81 6228 1490.0   6244   11462
Waiting:       33 3396 2218.5   3159   10526
Total:         86 6235 1490.7   6250   11467

心血来潮,测了一下ab的极限,呵呵:

==================================10000=====================================
InActConn连接数上不去了,保持在2600左右
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.29.100:http rr
  -> 192.168.29.45:http           Local   1      4996       2563
  -> 192.168.29.46:http           Route   1      5003       2557

==================================20000=====================================
ActiveConn连接数上不去了,保持在7500左右,ab压力机报错:apr_socket_connect(): 由于系统缓冲区空间不足或队列已满,不能执行套接字上的操作。(730055)
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.29.100:http rr
  -> 192.168.29.45:http           Local   1      7624       2188
  -> 192.168.29.46:http           Route   1      7571       2186



你可能感兴趣的:(IP,LVS,heartbeat,ipvsadm,pacemaker,addr)