LVS+keepalived(DR+TUN+NAT）
LVS(DR)+keepalived
u 拓扑描述
后端web服务器1：192.168.15.233
后端web服务器2：192.168.15.234
负载服务器master：192.168.15.235
负载服务器backup：192.168.15.236
Lvs负载虚拟服务器：192.168.15.253
u 设定安装环境
[root@localhost keepalived-1.1.20]#yum install kernel-devel kernel-headers openssl-devel
注：保证内核版本一致 若不一致可以采用yum方式更新
[root@hbchen ~]# rpm -qa|grep kernel
kernel-2.6.18-164.el5
kernel-headers-2.6.18-164.el5
kernel-devel-2.6.18-164.el5
u 软件安装配置
ü Ipvsadm安装配置
wget http://www.linuxvirtualserver.org/software/kernel-2.6/ipvsadm-1.24.tar.gz
#tar -zxvf ipvsadm-1.24.tar.gz
#cd ipvsadm-1.24
#cd ipvsadm-1.24
#建立编译时必须的一个软链接
#ln -s /usr/src/kernels/2.6.9-42.EL-i686/ /usr/src/linux
#ln -s /usr/src/kernels/2.6.9-42.EL-i686/ /usr/src/linux
#编译安装
#make && make install
#make && make install
#确认安装成功
#whereis ipvsadm
#whereis ipvsadm
注：可以采用yum方式安装
配置直接编写shell文件即可，如下
cat /root/ipvsadm.sh
#!/bin/sh
VIP=192.168.15.253
RIP1=192.168.15.233
RIP2=192.168.15.234
case "$1" in
start)
echo " start LVS "
#
set the Virtual IP Address
#
/sbin/modprobe ipip
/sbin/ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev eth0:0
/sbin/ipvsadm -C
/sbin/ipvsadm -A -t $VIP:80 -s rr
/sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -g -w 1
/sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -g -w 1
;;
stop)
echo "close LVS Director"
/sbin/ipvsadm -C
/sbin/ifconfig eth0:0 down
#
/sbin/modprobe -r ipip
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
ü Keepalived安装配置
wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz
注：可以到 http://www.keepalived.org/中更新最新版本
#yum install kernel-devel kernel-headers openssl-devel
#wget http://www.keepalived.org/software/keepalived-1.1.20.tar.gz
#tar -zxvf keepalived-1.1.15.tar.gz
#cd keepalived-1.1.15
#./configure --with-kernel-dir=/usr/src/kernels/2.6.18-194.32.1.el5-i686
#make && make install
# cp /usr/local/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/
# cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/
# mkdir /etc/keepalived
# cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/
# cp /usr/local/sbin/keepalived /usr/sbin/
#chkconfig –-add keepalived（添加至服务）
注：切记./configure添加kernel指向，如果make出现乱码错误请按如下错误汇总处理
Keepalived配置如下：
主调度器（192.168.15.235）
[root@localhost ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from [email protected]
smtp_server smtp.163.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.15.253
}
}
virtual_server 192.168.15.253 80 {
delay_loop 10
lb_algo wlc
lb_kind DR
persistence_timeout 50
protocol TCP
#
sorry_server 127.0.0.1 80
real_server 192.168.15.233 80 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.15.234 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
备用调度（192.168.15.236）
[root@localhost ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from [email protected]
smtp_server smtp.163.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.15.253
}
}
virtual_server 192.168.15.253 80 {
delay_loop 10
lb_algo wlc
lb_kind DR
persistence_timeout 50
protocol TCP
#
sorry_server 127.0.0.1 80
real_server 192.168.15.233 80 {
weight 3
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.15.234 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
u Real机配置
（192.168.15.233/234,写一shell即可）
[root@localhost ~]# vi /root/real.sh
#!/bin/bash
#description : start realserver
VIP=192.168.15.253
/etc/rc.d/init.d/functions
case "$1" in
start)
echo " start LVS of REALServer"
#/sbin/modprobe ipip
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
;;
stop)
#echo " stop LVS of REALServer"
echo "close LVS Directorserver"
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
#/sbin/ifconfig lo down
#/sbin/modprobe -r ipip
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
u 测试负载切换
ü 验证ipvsadm
启动两台web机器real.sh（./real.sh start）启动调度器上的ipvsadm （/root/ipvsadm start）
测试web服务器是否可以负载切换
ü 验证keepalived高可用性
关闭ipvsadm（/root/ipvsadm stop）启动keepalived 查看日志验证相关信息
1. 关闭web（192.168.15.234）
[root@localhost keepalived]# tail -f /var/log/messages
Apr 11 23:08:21 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 11 23:08:21 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.15.253
Apr 11 23:08:21 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.15.253 added
Apr 11 23:08:21 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.15.253 added
Apr 11 23:08:21 localhost avahi-daemon[2915]: Registering new address record for 192.168.15.253 on eth0.
Apr 11 23:08:27 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.15.253
Apr 11 23:11:40 localhost Keepalived_healthcheckers: TCP connection to [192.168.15.234:80] failed !!!
Apr 11 23:11:40 localhost Keepalived_healthcheckers: Removing service [192.168.15.234:80] from VS [192.168.15.253:80]
Apr 11 23:11:40 localhost Keepalived_healthcheckers: Remote SMTP server [127.0.0.1:25] connected.
Apr 11 23:11:40 localhost Keepalived_healthcheckers: SMTP alert successfully sent.
关闭主调度（192.168.15.235 service keepalived stop）
Apr 12 00:52:33 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.15.253 removed
Apr 12 00:52:33 localhost avahi-daemon[2915]: Withdrawing address record for 192.168.15.253 on eth0.
Apr 12 00:53:57 localhost Keepalived_vrrp: VRRP_Instance(VI_1)
Transition to MASTER STATE
Apr 12 00:54:02 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 12 00:54:02 localhost Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 12 00:54:02 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.15.253
Apr 12 00:54:02 localhost Keepalived_vrrp: Netlink reflector reports IP 192.168.15.253 added
Apr 12 00:54:02 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.15.253 added
Apr 12 00:54:02 localhost avahi-daemon[2915]: Registering new address record for 192.168.15.253 on eth0.
Apr 12 00:54:07 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.15.253
开启主调度（service keepalived start）
Apr 12 00:57:36 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert
Apr 12 00:57:36 localhost Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Apr 12 00:57:36 localhost Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.
Apr 12 00:57:36 localhost Keepalived_vrrp: Netlink reflector reports
IP 192.168.15.253 removed
Apr 12 00:57:36 localhost Keepalived_healthcheckers: Netlink reflector reports IP 192.168.15.253 removed
Apr 12 00:57:36 localhost avahi-daemon[2915]: Withdrawing address record for 192.168.15.253 on eth0.
u Lvs+Keepalive问题汇总：
ü 编译安装问题：
1../configure
Keepalived configuration
------------------------
Keepalived version
: 1.1.15
Compiler
: gcc
Compiler flags
: -g -O2
Extra Lib
: -lpopt -lssl -lcrypto
Use IPVS Framework
: Yes
IPVS sync daemon support : Yes
Use VRRP Framework
: Yes
Use LinkWatch
: No
Use Debug flags
: No
如果Use IPVS Framework 为No 则使用keepalived启用后将无法条用ipvsadm，所以
安装时需要指定kernel：
./configure --with-kernel-dir=/usr/src/kernels/2.6.18-194.32.1.el5-i686
ü 2. Can not include OpenSSL
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
提示open-ssl未安装：yum openssl-devel
ü .make时出现如下乱码错误：
check_http.c:459: 警告：对指针赋值时目标与指针有/无符号不一致
check_http.c:461: 警告：传递参数 1 (属于 ‘sprintf’) 给指针时目标与指针有/无符号不一致
check_http.c:463: 警告：传递参数 1 (属于 ‘strlen’) 给指针时目标与指针有/无符号不一致
check_http.c:463: 警告：传递参数 2 (属于 ‘__builtin_strcmp’) 给指针时目标与指针有/无符号不一致
check_http.c:463: 警告：传递参数 2 (属于 ‘__builtin_strcmp’) 给指针时目标与指针有/无符号不一致
check_http.c:463: 警告：传递参数 1 (属于 ‘strlen’) 给指针时目标与指针有/无符号不一致
check_http.c:463: 警告：传递参数 2 (属于 ‘__builtin_strcmp’) 给指针时目标与指针有/无符号不一致
check_http.c:463: 警告：传递参数 2 (属于 ‘__builtin_strcmp’) 给指针时目标与指针有/无符号不一致
尽量保持kernel版本一致，如：
[root@hbchen ~]# rpm -qa|grep kernel
kernel-2.6.18-164.el5
kernel-headers-2.6.18-164.el5
kernel-devel-2.6.18-164.el5
(keepalived对于一些kernel不兼容，所以kernel都要保持一致，Version 1.1.19以下的版本都兼容性比较好，
所以安装1.1.19以上的版本make时会出现很多乱七八糟的错误)
注意下这里2.6.18-194.32.1.el5-i686要替换为你自己系统的核心，用uname -r 查询
解决方法：yum install kernel-devel
ln -s /usr/src/kernels/2.6.9-22.EL-i686/ /usr/src/linux
ü ip_vs文件错误
/usr/include/sys/types.h:62: 错误：与 ‘dev_t’ 类型冲突
/usr/src/kernels/2.6.18-194.3.1.el5-i686/include/linux/types.h:22: 错误：‘dev_t’ 的上一个声明在此
/usr/include/sys/types.h:67: 错误：与 ‘gid_t’ 类型冲突
/usr/src/kernels/2.6.18-194.3.1.el5-i686/include/linux/types.h:54: 错误：‘gid_t’ 的上一个声明在此
/usr/include/sys/types.h:72: 错误：与 ‘mode_t’ 类型冲突
/usr/src/kernels/2.6.18-194.3.1.el5-i686/include/linux/types.h:24: 错误：‘mode_t’ 的上一个声明在此
/usr/include/sys/types.h:77: 错误：与 ‘nlink_t’ 类型冲突
/usr/src/kernels/2.6.18-194.3.1.el5-i686/include/linux/types.h:25: 错误：‘nlink_t’ 的上一个声明在此
/usr/include/sys/types.h:82: 错误：与 ‘uid_t’ 类型冲突
/usr/src/kernels/2.6.18-194.3.1.el5-i686/include/linux/types.h:53: 错误：‘uid_t’ 的上一个声明
.找到ip_vs.h文件.
find / -name ip_vs.h copy to /usr/include/net下
[root@hbchen /]# cp /usr/src/kernels/2.6.18-164.el5-i686/include/net/ip_vs.h /usr/include/net/
/usr/src/kernels/2.6.18-194.32.1.el5-i686/include/net/ip_vs.h
这个是我系统上的路径，
2.6.18-194.32.1.el5-i686要替换为你自己系统的核心没有这个路径的话用find命令查找
find / -name ip_vs.h （若找不到这个文件，你先要先把kernel-devel 安装好）
ü 加载ipvs模块
3.实现lvs群集的两个重要部件是ipvs内核模块和ipvsadm工具包。当前内核版本的系统已经包含ipvs内核模块，
但默认并没有加载到内核中，可以手工加载或安装ipvsadm之后会被加载。使用modprobe命令手工加载ipvs模块
并查询模块是否加
[root@hbchen ~]# modprobe ip_vs
[root@hbchen ~]# lsmod |grep ip_vs
ip_vs_wlc
6080 1
ü Keepalived_vrrp不生效
在/var/log/messages中Keepalived_vrrp不生效，即无vrrp日志
Nov 23 17:46:41 SN2008-06-070 Keepalived_vrrp: receive an invalid ip number count associated with VRID!
Nov 23 17:46:41 SN2008-06-070 Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
Nov 23 17:46:41 SN2008-06-070 Keepalived_vrrp: VRRP_Instance(VI_1) Dropping received VRRP packet...
Nov 23 17:46:42 SN2008-06-070 Keepalived_vrrp: receive an invalid ip number count associated with VRID!
Nov 23 17:46:42 SN2008-06-070 Keepalived_vrrp: bogus VRRP packet received on eth0 !!!
Nov 23 17:46:42 SN2008-06-070 Keepalived_vrrp: VRRP_Instance(VI_1) Dropping received VRRP packet...
重新编译源码包，可能是由于修改types.h后才configure
ü Ipvsadm启动问题
Ipvsadm可以通过脚本直接启动不用启动服务（service ipvsadm start）
若要直接启动，报错如下：
Applying IPVS configuration: /etc/init.d/ipvsadm: line 62: /etc/sysconfig/ipvsadm: 没有那个文件或目录
[失败]
[root@localhost init.d]# service ipvsadm save
Saving IPVS table to /etc/sysconfig/ipvsadm:
[确定]
[root@localhost init.d]# service ipvsadm start
Clearing the current IPVS table:
[确定]
Applying IPVS configuration:
[确定]
此时只需要保存ipvsadm表即可！（service ipvsadm save）
LVS(TUN)+keepalived
u Real机配置
[root@localhost ~]# vi /root/real.sh
#!/bin/bash
#description : start realserver
VIP=192.168.15.253
/etc/rc.d/init.d/functions
case "$1" in
start)
echo " start LVS of REALServer"
/sbin/modprobe ipip
/sbin/ifconfig tunl0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev tunl0
echo "1" >/proc/sys/net/ipv4/conf/tunl0/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/tunl0/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
;;
stop)
#echo " stop LVS of REALServer"
echo "close LVS Directorserver"
echo "0" >/proc/sys/net/ipv4/conf/tunl0/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/tunl0/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
/sbin/ifconfig tunl0 down
/sbin/modprobe -r ipip
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
u 调度器ipvsadm配置
#!/bin/sh
VIP=192.168.15.253
RIP1=192.168.15.233
RIP2=192.168.15.234
case "$1" in
start)
echo " start LVS "
#
set the Virtual IP Address
/sbin/modprobe ipip
/sbin/ifconfig tunl0 $VIP broadcast $VIP netmask 255.255.255.255 up
/sbin/route add -host $VIP dev tunl0
/sbin/ipvsadm -C
/sbin/ipvsadm -A -t $VIP:80 -s rr
/sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -i
/sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -i
;;
stop)
echo "close LVS Director"
/sbin/ipvsadm -C
/sbin/ifconfig tunl0 down
/sbin/modprobe -r ipip
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
u Mster keepalived配置
只需要将更改lb_kind TUN，backup机对应修改即可！
[root@localhost ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from [email protected]
smtp_server smtp.163.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.15.253
}
}
virtual_server 192.168.15.253 80 {
#delay_loop 10
lb_algo wlc
lb_kind TUN
#
persistence_timeout 1
protocol TCP
#
sorry_server 127.0.0.1 80
real_server 192.168.15.233 80 {
weight 1
TCP_CHECK {
connect_timeout 1
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.15.234 80 {
weight 1
TCP_CHECK {
connect_timeout 1
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
注：关闭ipvsadm—> /root/ipvsadm.sh stop
测试时将keepavlived里配置信息（weight、connect_timeout等）可自行设置。若要查看
LVS(NAT)+keepalived
u 拓扑描述
根据实际网络情况可在
后端web服务器1：192.168.15.233
后端web服务器2：192.168.15.234
负载服务器master：wlan-192.168.15.253
Lan-10.0.0.1
负载服务器backup：192.168.15.236
u Real机配置
需根据具体网络环境设置（1或2）
1. 设置网关
/etc/sysconfig/network-scripts/ifcfg-eth0 添加
GATEWAY=10.0.0.1
2. 添加默认网关
route add default gw 10.0.0.1 （提示：删除则用delete）
u 调度器ipvsadm设置
#echo 1 > /proc/sys/net/ipv4/ip_forward （开启路由机制）
[root@localhost ~]# cat /root/ipvsadm.sh
#!/bin/sh
VIP=192.168.15.253
VIP_LAN=10.0.0.1
RIP1=10.0.0.233
RIP2=10.0.0.234
case "$1" in
start)
echo " start LVS "
#
set the Virtual IP Address
#
/sbin/modprobe ipip
#
/sbin/ifconfig eth0:0 $VIP broadcast $VIP netmask 255.255.255.255 up
#
/sbin/route add -host $VIP dev eth0:0
/sbin/ifconfig eth0:2 $VIP_LAN netmask 255.255.255.0 broadcast 10.0.0.255 up
/sbin/ipvsadm -C
/sbin/ipvsadm -A -t $VIP:80 -s wlc
/sbin/ipvsadm -a -t $VIP:80 -r $RIP1:80 -m -w 1
/sbin/ipvsadm -a -t $VIP:80 -r $RIP2:80 -m -w 1
;;
stop)
echo "close LVS Director"
/sbin/ipvsadm -C
/sbin/ifconfig eth0:2 down
#
/sbin/modprobe -r ipip
;;
*)
echo "Usage: $0 {start|stop}"
exit 1
esac
测试验证正常
u Master Keepalived配置
[root@localhost ~]# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
}
notification_email_from [email protected]
smtp_server smtp.163.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 5
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.15.253
}
}
virtual_server 192.168.15.253 80 {
#delay_loop 10
lb_algo wlc
lb_kind NAT
#
persistence_timeout 1
protocol TCP
#
sorry_server 127.0.0.1 80
real_server 10.0.0.233 80 {
weight 1
TCP_CHECK {
connect_timeout 1
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 10.0.0.234 80 {
weight 1
TCP_CHECK {
connect_timeout 1
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
backup机对应修改即可
u 验证配置
关闭ipvsadm—> /root/ipvsadm.sh stop
添加IP /sbin/ifconfig eth0:2 10.0.0.1 netmask 255.255.255.0 broadcast 10.0.0.255 up
启动keepalived验证 service keepalived start