一、概述

1.1、背景

随着网站业务量的增长网站的服务器压力越来越大,需要负载均衡方案,而商业的硬件如F5成本高,为了节省不必要的浪费,同时实现商业硬件一样的高性能高可用的功能,我们可以利用LVS+Keepalived基于完整开源软件的架构提供一个负载均衡及高可用的服务器。

1.2、LVS+Keepalived介绍

LVS: Linux Virtual Server的简写,意即Linux虚拟服务器,是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立,是中国国内最早出现的自由软件项目之一。目前有三种IP负载均衡技术(VS/NAT、VS/TUN和VS/DR):


目前有十种调度算法(rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq),上次实验是采用的VS/DR方法并带有主备模式LVS,调度算法用的wrr.两台WEB通vip经lvs(主备)调度实现业务分发负载均衡。

Keepalived在这里主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现.

以下实验是在只用两台的情况下,实现WEB负载均衡的方法

二、安装前环境和准备

2.1、实验说明

主LVS_master,WEB1: 192.168.6.14

备LVS_backeup,WEB2:192.168.6.247

对外VIP:192.168.6.201

2.2、系统环境

操作系统64位centos 6.0,

内核:2.6.32-220.el6.x86_64

4台操作系统均为centos 6.0,并配有yum环境.

2.3、安装前准备

2.3.1、下载popt-static

http://mirror.centos.org/centos/6/os/x86_64/Packages/popt-static-1.13-7.el6.x86_64.rpm

我在安装过程碰到些报错,下载该包问题解决,下载以下包时根据系统内核确定其版本。

2.3.2、下载keepalived

http://www.keepalived.org/software/keepalived-1.2.2.tar.gz

2.3.3、下载ipvsadm

http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.26.tar.gz

2.3.4、加载ip_vs模块(重要)

   查看内核uname–r,加载模块,modprobe ip_vs,检查lsmod | grep ip_vs

三、Director Server安装

3.1、LVS_Master安装(192.168.6.14)

3.1.1、安装依赖包

#yum -y install kernel-devel make gccopenssl-devel libnl* popt*

我这用的yum安装,大多情况实际下不一定有网络,手动去下载相关包或从安装包中找到相关。

3.1.2、安装popt-static

#rpm -ivh popt-static-1.13-7.el6.x86_64.rpm


3.1.3、安装ipvsadm

# ln -s /usr/src/kernels/2.6.32-71.el6.x86_64/usr/src/linux
# tar -zxvfipvsadm-1.26.tar.gz -C /usr/src/

# cd /usr/src/ipvsadm-1.26/
# make
# make install

3.1.3、安装keepalived

# tar -zxvfkeepalived-1.2.2.tar.gz -C /usr/src/
# cd /usr/src/keepalived-1.2.2/
# ./configure

# make
# make install
#cp /usr/local/etc/rc.d/init.d/keepalived/etc/rc.d/init.d/
#cp /usr/local/etc/sysconfig/keepalived/etc/sysconfig/
# mkdir /etc/keepalived
#cp /usr/local/etc/keepalived/keepalived.conf/etc/keepalived/
#cp /usr/local/sbin/keepalived /usr/sbin/

3.1.4、配置keepalived.conf

# vi /etc/keepalived/keepalived.conf//内容如下

! Configuration File for keepalived
global_defs {
notification_email {
     [email protected]
   }
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id linux01
}
vrrp_instance VI_1 {                               #定义一个实例
statemaster
interface eth0
virtual_router_id 51
     priority 100                   #在priority高的这台设置这个参数
nopreempt                                      #可以自动接管
advert_int 1                                 
authentication {
auth_type PASS
auth_pass 1111
    }
virtual_ipaddress {
        192.168.6.201 dev eth0 label eth0:1  #使用label标签启用eth0:1
     }
}
virtual_server 192.168.6.201 80 {             #这里往下是定义LVS
delay_loop 6
lb_algowrr
lb_kind DR
nat_mask 255.255.255.0
protocol TCP
real_server 127.0.0.1 80 {                 #其中一台是本地web服务
weight 1
        TCP_CHECK {
connect_timeout 3
        }
    }
real_server 192.168.6.247 80 {             #另一台的web服务
weight 1
        TCP_CHECK {
connect_timeout 3
        }
    }  
}

3.1.5开启负载服务器路由机制(IP转发)

vi /etc/sysctl.conf 保证有如下内容
 net.ipv4.ip_forward = 1


3.1.6建立负载服务器启动脚本

vi /usr/local/bin/lvsdr.sh 加入如下内容

#!/bin/bash
VIP=192.168.6.72
RIP1=192.168.6.14
RIP2=192.168.7.144
/sbin/rc.d/init.d/functions
case "$1" in
start)
  echo "start LVS of DirectorServer"
   #Set the Virtual IP Address
/sbin/ifconfig eth0:1 $VIP broadcast $VIP netmask 255.255.255.255 up
      /sbin/route add -host $VIP dev eth0:1
       #Clear IPVS Table
       /sbin/ipvsadm -C
       #Set Lvs算法wrr,-g dr模式
       /sbin/ipvsadm -A -t $VIP:80 -s wrr
       /sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -g
       /sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -g
       #Run Lvs
       /sbin/ipvsadm
;;
stop)
echo "Close LVS Directorserver"
/sbin/ifconfig eth0:1 down
/sbin/ipvsadm -C
;;
*)
echo "Usage0{start|stop}"
exit 1
esac

3.1.7、测试负载服务器启动脚本

#Chmod +x /usr/local/bin/lvsdr.sh

#Chomd 775 /usr/local/bin/lvsdr.sh

执行测试
   #/usr/local/bin/lvsdr.shstart
   查看ifconfig是否有ifcfg-eth0:1   (有就对了)
   查看route -n 路由表是否多了eth0:1路由(有就对了)
   [root@host1 ~]/usr/local/bin/lvsdr.sh stop
   查看ifconfig是否有ifcfg-eth0:1  (无就对了)
   查看route -n 路由表是否多了eth0:1路由(无就对了)
   [root@host1 ~]/usr/local/bin/lvsdr.sh  adsa
   是否提示参数错误,只能使用{start|stop}。

3.1.8、安装启动回环地址脚本

要是用lvsdr模式,必须在realserver上启动回环地址,所以,搭建两台机器的负载均衡,当然就得在本机也启动相应的回环地址,才能实现所要的功能,脚本如下

vi /usr/local/bin/startlo

#!/bin/sh
VIP=192.168.6.201
case "$1" in
stop)
        # close lo:0 interface
echo $"Close lo:0 interface"
        /sbin/route del -host $VIP dev lo:0
        /sbin/ifconfig lo:0 down
echo "0">/proc/sys/net/ipv4/conf/all/arp_announce
echo "0">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0">/proc/sys/net/ipv4/conf/lo/arp_ignore
        ;;
start)
        # start lo:0 interface
echo $"Start lo:0 interface"
        /sbin/ifconfig lo:0 $VIP/32 broadcast $VIP up
        /sbin/route add -host $VIP dev lo:0
echo "2">/proc/sys/net/ipv4/conf/all/arp_announce
echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2">/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore
        ;;
  *)
echo $"Usage: $0 (start|stop)"
exit 1
        ;;
esac

给其执行chmod+x/usr/local/bin/startlo

3.1.9、安装检测启动回环地址脚本

但由于keepalived不像heartbeat里有ldirectord来控制回环地址的启动和停止,所以我们需要人为的判断应该在哪台机器上启动回环地址,所以就得使用下面的一个脚本判断,

# vi /usr/local/bin/check_interface

#!/bin/bash
#
#脚本说明:
#本脚本用于判断当前server是否是master 只用于只用两台机做lvs+keepalived
#如果当前server不是master,则启动回环地址
#否则停止回环地址
#
#VIP_interface请根据自己的情况自行设定,这个是VIP接口
#lo_interface请根据自己的情况自行设定,是回环地址接口
VIP_interface="eth0:1"
lo_interface="lo:0"
i=10
while [ $i != 0 ]
do
    /sbin/ifconfig |grep $VIP_interface&>/dev/null
retval=$?
if [ ! $retval -eq0 ];then
        /sbin/ifconfig |grep $lo_interface&>/dev/null
retval=$?
if [ ! $retval -eq0 ];then
            /usr/local/bin/startlo start
fi
else
        /sbin/ifconfig |grep $lo_interface&>/dev/null
retval=$?
if [ $retval -eq 0 ];then
            /usr/local/bin/startle stop
fi
fi
    i=10
sleep 10
done

给其执行权限chmod +x /usr/local/bin/check_interface

3.2LVS_Backup安装

LVS_Backup(192.168.6.247)安装与LVS_MASTER安装一致,不同的只是修改keepalived配置文件,确定其主备关系,当主的不在线时,备的顶上。

[root@host1 ~]# vi/etc/keepalived/keepalived.conf//内容如下

! Configuration File for keepalived
global_defs {
notification_email {
     [email protected]
   }
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id linux01
}
vrrp_instance VI_1 {                                                       
interface eth0
virtual_router_id 51
priority 50
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
    }
virtual_ipaddress {
        192.168.6.201 dev eth0 label eth0:1              
     }
}
virtual_server 192.168.6.201 80 {                       
delay_loop 6
lb_algowrr
lb_kind DR
nat_mask 255.255.255.0
protocol TCP
real_server 127.0.0.1 80 {                         
weight 1
        TCP_CHECK {
connect_timeout 3
        }
    }
real_server 192.168.6.14 80 {           
weight 1
        TCP_CHECK {
connect_timeout 3
        }
    }
}

四、最后安装测试

Httpd安装这里就不说了,启动每台机器的脚本,和HTTPD服务,keepalived。

[root@host1~]# /usr/local/bin/lvsdr.sh start

[root@host1~]# /usr/local/bin/check_interface

[root@host1~]# chkconfigkeepalived on
[root@host1 ~]# /etc/init.d/keepalived start
[root@host1 ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
IP Virtual Server version 1.2.1 (size=4096)

ProtLocalAddress:PortScheduler Flags

->RemoteAddress:PortForward Weight ActiveConnInActConn

TCP192.168.6.201:httpwrr

->localhost.localdomain:httpLocal100

-> 192.168.6.247:httpRoute100

在局域网中浏览器不同主机上同时打开http://192.168.6.201查看web,测试时进入的页面不一样则成功,做测试,两台机器做了两个不同页面,实际环境中为一致的。

关闭其中一台/etc/init.d/keepalived stop查看,有页面则正常。

[root@node1 keepalived]# vi realserver

#!/bin/bash

#description : start realserver

VIP=192.168.1.250

/etc/rc.d/init.d/functions

case "$1" in

start)

echo " start LVS ofREALServer"

/sbin/ifconfig lo:0 $VIP broadcast $VIPnetmask 255.255.255.255 up

echo "1">/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1">/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2">/proc/sys/net/ipv4/conf/all/arp_announce

;;

stop)

/sbin/ifconfig lo:0 down

echo "close LVSDirectorserver"

echo "0">/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "0">/proc/sys/net/ipv4/conf/lo/arp_announce

echo "0">/proc/sys/net/ipv4/conf/all/arp_ignore

echo "0">/proc/sys/net/ipv4/conf/all/arp_announce

;;

*)

echo "Usage: $0 {start|stop}"

exit 1

esac

[root@node1 keepalived]# chmod +xrealserver

[root@node1 keepalived]# ./realserverstart