部署Keepalived+LVS

部署keepalived高可用

部署环境：

KEEP1 IP :10.0.0.5

KEEP2 IP : 10.0.0.6

两台服务器的环境都是如下

[root@KEEP2 ~]# cat /etc/redhat-release 

CentOS release 6.6 (Final)

[root@KEEP2 ~]# uname -v

#1 SMP Wed Jul 15 10:13:09 UTC 2015

[root@KEEP2 ~]# uname -a

Linux KEEP2 2.6.32-504.30.3.el6.x86_64 #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@KEEP2 ~]# uname -r

2.6.32-504.30.3.el6.x86_64

下面是正式开始部署Keepalived，在KEEP1和KEEP2做相同的操作，知道第四部，配置配置文件才有所差别

#######################################################################

#######################################################################

#######################################################################

1. 做相关内核软连接

安装keepalived之前要先做软连接

ls /usr/src/kernels/`uname -r` -ld

ln -s /usr/src/kernels/`uname -r` /usr/src/linux

ls /usr/src/ -l

=========================================================================

[root@KEEP2 ~]# ls /usr/src/kernels/`uname -r` -ld

drwxr-xr-x. 22 root root 4096 Jul 20 10:47 /usr/src/kernels/2.6.32-431.el6.x86_64

[root@KEEP2 ~]# ln -s /usr/src/kernels/`uname -r` /usr/src/linux

[root@KEEP2 ~]# ls /usr/src/ -l

total 8

drwxr-xr-x. 2 root root 4096 Sep 23  2011 debug

drwxr-xr-x. 4 root root 4096 Jul 21 09:16 kernels

lrwxrwxrwx  1 root root   38 Aug 18 08:29 linux -> /usr/src/kernels/2.6.32-431.el6.x86_64

[root@KEEP2 ~]#

#######################################################################

#######################################################################

#######################################################################

操作步骤2之前可以需要安装下面依赖库

yum install openssl* -y

yum install libnl* -y 

2.下载并安装keepalived软件

[ ! -d /home/lvnian/tools ] && mkdir /home/lvnian/tools -p

cd /home/lvnian/tools 

wget http://www.keepalived.org/software/keepalived-1.2.10.tar.gz

tar xf keepalived-1.2.10.tar.gz 

cd keepalived-1.2.10

./configure 

make && make install

 ======================================================

安装简略过程

[root@KEEP2 keepalived-1.2.10]# ./configure

config.status: creating keepalived/libipvs-2.6/Makefile

Keepalived configuration

------------------------

Keepalived version       : 1.2.10

Keepalived configuration

------------------------

Keepalived version       : 1.2.10

Compiler                 : gcc

Compiler flags           : -g -O2 -DFALLBACK_LIBNL1

Extra Lib                : -lssl -lcrypto -lcrypt  -lnl

Use IPVS Framework       : Yes

IPVS sync daemon support : Yes

IPVS use libnl           : Yes  (如果这个为no 可以选择yum install libnl* -y 在编译安装)

Use VRRP Framework       : Yes

Use VRRP VMAC            : Yes

SNMP support             : No

SHA1 support             : No

Use Debug flags          : No

[root@KEEP2 keepalived-1.2.10]# make && make install

make[1]: Leaving directory `/home/deng/tools/keepalived-1.2.10/keepalived'

make -C genhash install

make[1]: Entering directory `/home/deng/tools/keepalived-1.2.10/genhash'

install -d /usr/local/bin

install -m 755 ../bin/genhash /usr/local/bin/

install -d /usr/local/share/man/man1

install -m 644 ../doc/man/man1/genhash.1 /usr/local/share/man/man1

make[1]: Leaving directory `/home/deng/tools/keepalived-1.2.10/genhash'

[root@KEEP2 keepalived-1.2.10]#

编译参数说明：

--with-kernel-dir=DIR 这个参数很重要，但是这个参数并不是要把keepalived编译到内核中，而是指定使用的内核源码里面的头文件，就是include目录，如果使用LVS，才需要这个参数，但是就算使用了LVS也可以不需要这个参数,但需要做软连接：

ln -s /usr/src/kernels/`uname -r` /usr/src/linux

#######################################################################

#######################################################################

#######################################################################

3.配置启动规范

/bin/cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/

/bin/cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived 

/bin/cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

/bin/cp /usr/local/sbin/keepalived /usr/sbin/

/etc/init.d/keepalived start

 ps -ef |grep keepalived

 =================================

[root@KEEP2 keepalived-1.2.10]# /bin/cp /usr/local/etc/rc.d/init.d/keepalived /etc/init.d/

[root@KEEP2 keepalived-1.2.10]# /bin/cp /usr/local/etc/sysconfig/keepalived /etc/sysconfig/

/etc/init.d/keepalived start

[root@KEEP2 keepalived-1.2.10]# mkdir /etc/keepalived 

 ps -ef |grep keepalived

[root@KEEP2 keepalived-1.2.10]# /bin/cp /usr/local/etc/keepalived/keepalived.conf /etc/keepalived/

[root@KEEP2 keepalived-1.2.10]# /bin/cp /usr/local/sbin/keepalived /usr/sbin/

[root@KEEP2 keepalived-1.2.10]# /etc/init.d/keepalived start

Starting keepalived:                                       [  OK  ]

[root@KEEP2 keepalived-1.2.10]#  ps -ef |grep keepalived

root      2765     1  0 06:04 ?        00:00:00 keepalived -D

root      2768  1348  0 06:04 pts/0    00:00:00 grep keepalived

root      2769  2765  0 06:04 ?        00:00:00 keepalived -D

root      2770  2765  0 06:04 ?        00:00:00 keepalived -D

[root@KEEP2 keepalived-1.2.10]#

安装配置成功

上面的配置需要在KEEP1和KEEP2中都执行

#######################################################################

#######################################################################

#######################################################################

4.配置keepalived单实例

[root@KEEP1 keepalived]# pwd

/etc/keepalived

主配置文件如下MASTER：

[root@KEEP1 keepalived]# cat keepalived.conf

! Configuration File for keepalived

global_defs {

   notification_email {

   [email protected]

   }

   notification_email_from [email protected]

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_1                       ##路由ID，主备必须要不相同

}

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 55                 ##虚拟路由ID，主备必须要相同

    priority 150                         ##主的优先级必须比备的大

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.0.0.100/24

        10.0.0.200/25

    }

}

[root@KEEP1 keepalived]#

备机配置文件BACKUP:

[root@KEEP2 keepalived]# cat keepalived.conf

! Configuration File for keepalived

global_defs {

   notification_email {

   [email protected]

   }

   notification_email_from [email protected]

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_2

}

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 55

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.0.0.100/24

        10.0.0.200/25

    }

}

[root@KEEP2 keepalived]#

查看VIP情况

KEEP1：

KEEP2

VIP会跟着主备服务器的切换而漂移

把KEEP1的keepalived服务停止

[root@KEEP1 ~]# /etc/init.d/keepalived stop

Stopping keepalived:                                       [  OK  ]

KEEP1

KEEP2

#######################################################################

#######################################################################

#######################################################################

5．keepalived的日志信息

默认情况下keepalived的日志文件在/var/log/message

[root@KEEP1 ~]# tail -f /var/log/messages  

Aug 18 09:06:49 LVS1 Keepalived[6046]: Starting Keepalived v1.2.10 (08/18,2015)

Aug 18 09:06:49 LVS1 Keepalived[6047]: Starting Healthcheck child process, pid=6049

Aug 18 09:06:49 LVS1 Keepalived[6047]: Starting VRRP child process, pid=6050

Aug 18 09:06:49 LVS1 Keepalived_vrrp[6050]: Netlink reflector reports IP 10.0.0.3 added

Aug 18 09:06:49 LVS1 Keepalived_healthcheckers[6049]: Netlink reflector reports IP 10.0.0.3 added

Aug 18 09:06:49 LVS1 Keepalived_vrrp[6050]: Netlink reflector reports IP fe80::20c:29ff:fea7:2f91 added

Aug 18 09:06:49 LVS1 Keepalived_vrrp[6050]: Registering Kernel netlink reflector

Aug 18 09:06:49 LVS1 Keepalived_vrrp[6050]: Registering Kernel netlink command channel

Aug 18 09:06:49 LVS1 Keepalived_vrrp[6050]: Registering gratuitous ARP shared channel

Aug 18 09:06:49 LVS1 Keepalived_healthcheckers[6049]: Netlink reflector reports IP fe80::20c:29ff:fea7:2f91 added

Aug 18 09:06:49 LVS1 Keepalived_healthcheckers[6049]: Registering Kernel netlink reflector

Aug 18 09:06:49 LVS1 Keepalived_healthcheckers[6049]: Registering Kernel netlink command channel

#######################################################################

配置keepalived日志信息到单独文件/var/log/keepalived.log

默认情况如下

[root@KEEP1 keepalived]# cat /etc/sysconfig/keepalived 

# Options for keepalived. See `keepalived --help' output and keepalived(8) and

# keepalived.conf(5) man pages for a list of all options. Here are the most

# common ones :

#

# --vrrp               -P    Only run with VRRP subsystem.

# --check              -C    Only run with Health-checker subsystem.

# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.

# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.

# --dump-conf          -d    Dump the configuration data.

# --log-detail         -D    Detailed log messages.

# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)

#

KEEPALIVED_OPTIONS="-D"（把这个注销换成下面内容）

[root@KEEP1 keepalived]#

#######################################################################

修改后的内容

[root@KEEP1 keepalived]# cat /etc/sysconfig/keepalived 

# Options for keepalived. See `keepalived --help' output and keepalived(8) and

# keepalived.conf(5) man pages for a list of all options. Here are the most

# common ones :

#

# --vrrp               -P    Only run with VRRP subsystem.

# --check              -C    Only run with Health-checker subsystem.

# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.

# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.

# --dump-conf          -d    Dump the configuration data.

# --log-detail         -D    Detailed log messages.

# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)

#

#KEEPALIVED_OPTIONS="-D"

KEEPALIVED_OPTIONS="-D -d -S 0"

 

[root@KEEP1 keepalived]#

#######################################################################

配置/etc/rsyslog.conf 

cp /etc/rsyslog.conf /etc/rsyslog.conf.ori

echo "local0.*    /var/log/keepalived.log" >>/etc/rsyslog.conf

tail -4 /etc/rsyslog.conf 

=================================

[root@KEEP1 etc]# tail -4 /etc/rsyslog.conf 

#*.* @@remote-host:514

# ### end of the forwarding rule ###

#keepalived

local0.*    /var/log/keepalived.log

[root@KEEP1 etc]#

#######################################################################

#######################################################################

#######################################################################

重启日志服务，检查配置结果

/etc/init.d/rsyslog restart

tail /var/log/keepalived.log  -f

==================

[root@KEEP1 ~]# /etc/init.d/rsyslog restart

Shutting down system logger:                               [  OK  ]

Starting system logger:                                    [  OK  ]

[root@KEEP1 ~]# tail /var/log/keepalived.log  -f

Aug 18 09:36:07 KEEP1 Keepalived_healthcheckers[14792]: Netlink reflector reports IP 10.0.0.3 added

Aug 18 09:36:07 KEEP1 Keepalived_healthcheckers[14792]: Netlink reflector reports IP fe80::20c:29ff:fea7:2f91 added

Aug 18 09:36:07 KEEP1 Keepalived_healthcheckers[14792]: Registering Kernel netlink reflector

Aug 18 09:36:07 KEEP1 Keepalived_healthcheckers[14792]: Registering Kernel netlink command channel

Aug 18 09:36:07 KEEP1 Keepalived_vrrp[14793]: Netlink reflector reports IP 10.0.0.3 added

Aug 18 09:36:07 KEEP1 Keepalived_vrrp[14793]: Netlink reflector reports IP fe80::20c:29ff:fea7:2f91 added

Aug 18 09:36:07 KEEP1 Keepalived_vrrp[14793]: Registering Kernel netlink reflector

Aug 18 09:36:07 KEEP1 Keepalived_vrrp[14793]: Registering Kernel netlink command channel

Aug 18 09:36:07 KEEP1 Keepalived_vrrp[14793]: Registering gratuitous ARP shared channel

Aug 18 09:36:07 KEEP1 Keepalived_vrrp[14793]: Initializing ipvs 2.6

如果没看到日志内容，但也不提示错误，那你重启一下你的keepalived服务就能看到日志内容了

###################################################################

###################################################################

如果要keepalived配合LVS使用，使用下面的配置文件即可。master即KEEP1，BACKCP即KEEP2.

MASTER:

! Configuration File for keepalived

global_defs {

   notification_email {

   [email protected]

   }

   notification_email_from [email protected]

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_1

}

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    virtual_router_id 55

    priority 150

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.0.0.100/24

     #   10.0.0.200/25

    }

}

####################################

virtual_server 10.0.0.100 80{  

    delay_loop 3  

    lb_algo rr  

    lb_kind DR  

    persistence_timeout 1  

    protocol TCP  

    real_server 10.0.0.3 80 {  

        weight 1                       

        TCP_CHECK {                    

            connect_timeout 10         

            nb_get_retry 3             

            delay_before_retry 3       

            connect_port 80            

        }                              

    }                                  

    real_server 10.0.0.4 80  {         

        weight 1                       

        TCP_CHECK {                    

            connect_timeout 10         

            nb_get_retry 3             

            delay_before_retry 3       

            connect_port 80            

        }                              

    }                                  

}  

###################################################################

BACKUP:

! Configuration File for keepalived

global_defs {

   notification_email {

   [email protected]

   }

   notification_email_from [email protected]

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_2

}

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 55

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.0.0.100/24

 #       10.0.0.200/25

    }

}

################################################

virtual_server 10.0.0.100 80{  

    delay_loop 3  

    lb_algo rr  

    lb_kind DR  

    persistence_timeout 1  

    protocol TCP  

    real_server 10.0.0.3 80 {  

        weight 1                       

        TCP_CHECK {                    

            connect_timeout 10         

            nb_get_retry 3             

            delay_before_retry 3       

            connect_port 80            

        }                              

    }                                  

    real_server 10.0.0.4 80  {         

        weight 1                       

        TCP_CHECK {                    

            connect_timeout 10         

            nb_get_retry 3             

            delay_before_retry 3       

            connect_port 80            

        }                              

    }                                  

}  

###################################################################

 后端的RS服务配置好web服务之后再执行下面命令即可：

每台RS配置VIP

ifconfig lo:100 10.0.0.100 netmask 255.255.255.255 up

ifconfig lo:100

ifconfig

route add -host 10.0.0.100 dev lo

抑制ARP响应

echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore

echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

或者用下面脚本执行：

cat /etc/init.d/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

 

VIP=10.0.0.100

 

/etc/rc.d/init.d/functions

 

case "$1" in

start)

       ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP

       /sbin/route add -host $VIP dev lo:0

       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

       sysctl -p >/dev/null 2>&1

       echo "RealServer Start OK"

 

       ;;

stop)

       ifconfig lo:0 down

       route del $VIP >/dev/null 2>&1

       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

       echo "RealServer Stoped"

       ;;

*)

       echo "Usage: $0 {start|stop}"

       exit 1

esac

 

exit 0

 

###################################################################

###################################################################

 问题解决

故障1：

[root@KEEP1 keepalived-1.2.10]# ./configure

checking for sys/time.h... yes

checking openssl/ssl.h usability... no

checking openssl/ssl.h presence... no

checking for openssl/ssl.h... no

configure: error: 

  !!! OpenSSL is not properly installed on your system. !!!

  !!! Can not include OpenSSL headers files.

yum install openssl* -y

故障2

[root@KEEP1 keepalived-1.2.10]# ./configure

libipvs.c:73: error: ‘NLE_PERM’ undeclared (first use in this function)

libipvs.c:74: error: ‘NLE_BUSY’ undeclared (first use in this function)

libipvs.c:75: error: ‘NLE_RANGE’ undeclared (first use in this function)

libipvs.c:76: error: ‘NLE_NODEV’ undeclared (first use in this function)

make[2]: *** [libipvs.o] Error 1

make[2]: Leaving directory `/home/lvnian/tools/keepalived-1.2.10/keepalived/libipvs-2.6'

make[1]: *** [all] Error 1

make[1]: Leaving directory `/home/lvnian/tools/keepalived-1.2.10/keepalived'

make: *** [all] Error 2

解决办法

[root@KEEP1 keepalived-1.2.10]# yum install libnl* -y