Lvs + keepalived + nginx 高可用负载均衡

https://www.cnblogs.com/liuyisai/p/5990645.html

lvs+keepalived+nginx实现高性能负载均衡集群

一、为什么要使用负载均衡技术?

1、系统高可用性

2、  系统可扩展性

3、  负载均衡能力

LVS+keepalived能很好的实现以上的要求,LVS提供负载均衡,keepalived提供健康检查,故障转移,提高系统的可用性!采用这样的架构以后很容易对现有系统进行扩展,只要在后端添加或者减少realserver,只要更改lvs的配置文件,并能实现无缝配置变更!

二、LVS+Keepalived介绍

1、  LVS

LVS是一个开源的软件,可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写,意思是Linux虚拟服务器。目前有三种IP负载均衡技术(VS/NAT、VS/TUN和VS/DR);八种调度算法(rr,wrr,lc,wlc,lblc,lblcr,dh,sh)。

2、  keepalived

Keepalived 是运行在lvs 之上,它的主要功能是实现真实机的故障隔离及负载均衡器间的失败切换,提高系统的可用性

三、环境:

四台服务器,系统全为CentOS6.8:

192.168.2.203 master lvs+keepalived

192.168.2.202 backup lvs+keepalived

192.168.2.204 web1(nginx)

192.168.2.205 web2 (nginx)

vip:192.168.2.13

其中nginx已预装好,这里不再写搭建过程


四、搭建并配置

1、分别在backup lvs和master lvs上安装lvs

1root@bogon src]# yum -y install ipvsadm 2已加载插件:fastestmirror 3设置安装进程 4Determining fastest mirrors 5epel/metalink                                                                                  |5.4kB00:00 6*base: mirror.lzu.edu.cn 7... ... 8已安装: 9ipvsadm.x86_640:1.26-4.el6                                                                                          1011作为依赖被安装:12libnl.x86_640:1.1.4-2.el6                                                                                          1314完毕!

2、把ipvsadm模块加载进系统

1[root@bogon src]# ipvsadm2IP Virtual Server version1.2.1(size=4096)3Prot LocalAddress:Port Scheduler Flags4-> RemoteAddress:Port          Forward Weight ActiveConn InActConn5[root@bogon src]#lsmod|grep ip_vs6ip_vs12689707libcrc32c12461 ip_vs8ipv6336282270ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

3、分别在backup lvs和master lvs上安装keepalived(keepalived官网:http://www.keepalived.org/)

[root@bogon src]#tarzxf keepalived-1.2.24.tar.gz

[root@bogon src]# cd keepalived-1.2.24[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/报错:

configure: error:

  !!! OpenSSL is not properly installed on your system. !!!  !!! Can not include OpenSSL headers files.            !!!

系统缺少openssl-devel包所致

安装openssl-devel

root@bogon keepalived-1.2.24]#yum-yinstallopenssl-devel

再次编辑安装

[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/Keepalived configuration------------------------Keepalived version      : 1.2.24Compiler                : gccPreprocessor flags      : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//includeCompiler flags          : -Wall -Wunused -Wstrict-prototypes

Linker flags            :

Extra Lib                : -ldl -lssl -lcrypto

Use IPVS Framework      : Yes

IPVS use libnl          : No

IPVS syncd attributes    : No

IPVS 64 bit stats        : No

fwmark socket support    : Yes

Use VRRP Framework      : Yes

Use VRRP VMAC            : Yes

Use VRRP authentication  : Yes

With ip rules/routes    : Yes

SNMP keepalived support  : No

SNMP checker support    : No

SNMP RFCv2 support      : No

SNMP RFCv3 support      : No

SHA1 support            : No

Use Debug flags          : No

Stacktrace support      : No

Memory alloc check      : No

libnl version            : None

Use IPv4 devconf        : No

Use libiptc              : No

Use libipset            : No

Build genhash            : Yes

Build documentation      : No

[root@bogon keepalived-1.2.24]#make&&makeinstall

[root@bogon keepalived-1.2.24]#ln-s /usr/local/sbin/keepalived /sbin/[root@bogon keepalived-1.2.24]# chkconfig --add keepalived

[root@bogon keepalived-1.2.24]# chkconfig --level35keepalived on

4、配置keepalived

lvs-master的配置文件如下

[root@bogon keepalived-1.2.24]#cat/etc/keepalived/keepalived.conf! Configuration Fileforkeepalivedglobal_defs {      #全局配置部分#  notification_email {#email 通知,基本不用此处所以注释掉#    [email protected]#    [email protected]#    [email protected]#  }#  notification_email_from [email protected]#  smtp_server192.168.200.1#  smtp_connect_timeout 30router_id LVS_DEVEL#设置lvs的id,在一个网络内应该是唯一的vrrp_skip_check_adv_addr    vrrp_strict  vrrp_garp_interval0  vrrp_gna_interval 0}vrrp_instance VI_1 {    #vrrp实例定义部分    state MASTER#设置lvs的状态,报错MASTER和BACKUP两种,必须大写interface eth1#设置对外服务的接口virtual_router_id60  #设置虚拟路由标示,这个标示是一个数字,同一个vrrp实例使用唯一标示    priority 100    #定义优先级,数字越大优先级越高,在一个vrrp——instance下,master的优先级必须大于backup    advert_int 1    #设定master与backup负载均衡器之间同步检查的时间间隔,单位是秒    authentication {  #设置验证类型和密码

        auth_type PASS  #主要有PASS和AH两种

        auth_pass 1111  #验证密码,同一个vrrp_instance下MASTER和BACKUP密码必须相同    }

    virtual_ipaddress {  #设置虚拟ip地址,可以设置多个,每行一个

        192.168.2.13    }

}

virtual_server 192.168.2.1380 {  #设置虚拟服务器,需要指定虚拟ip和服务端口

    delay_loop 3#健康检查时间间隔lb_algo rr #负载均衡调度算法lb_kind DR#负载均衡转发规则persistence_timeout 50#设置会话保持时间,对动态网页非常有用    protocol TCP    #指定转发协议类型,有TCP和UDP两种

    real_server 192.168.2.20480 {  #配置服务器节点1,需要指定real server的真实IP地址和端口

        weight 1    #设置权重,数字越大权重越高    TCP_CHECK {     #realserver的状态监测设置部分单位秒

            connect_timeout 3    #超时时间            nb_get_retry 3      #重试次数            delay_before_retry 3    #重试间隔        connect_port 80      #监测端口

    }

}

    real_server 192.168.2.20580 {

    weight 1    TCP_CHECK {

    connect_timeout 3    nb_get_retry 3    delay_before_retry 3    connect_port 80        }

    }

}


LVS-backup的配置文件如下

[root@bogon keepalived-1.2.24]#cat/etc/keepalived/keepalived.conf! Configuration Filefor keepalived

global_defs {

#  notification_email {

#    [email protected]

#    [email protected]

#    [email protected]

#  }

#  notification_email_from [email protected]

#  smtp_server 192.168.200.1  smtp_connect_timeout 30  router_id LVS_DEVEL

  vrrp_skip_check_adv_addr

  vrrp_strict

  vrrp_garp_interval 0  vrrp_gna_interval 0}

vrrp_instance VI_1 {

    state BACKUP

    interface eth1

    virtual_router_id 60    priority 80    advert_int 1    authentication {

        auth_type PASS

        auth_pass 1111    }

    virtual_ipaddress {

        192.168.2.13    }

}

virtual_server 192.168.2.1380 {

    delay_loop 3    lb_algo rr

    lb_kind DR

    persistence_timeout 3    protocol TCP

    real_server 192.168.2.20480 {

        weight 1    TCP_CHECK {

            connect_timeout 3            nb_get_retry 3            delay_before_retry 3        connect_port 80   

    }

}

    real_server 192.168.2.20580 {

    weight 1    TCP_CHECK {

    connect_timeout 3    nb_get_retry 3    delay_before_retry 3    connect_port 80        }

    }

}


5、realserver的配置

两台web服务器都要执行下面脚本

[root@bogon www]#cat/etc/rc.d/init.d/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

SNS_VIP=192.168.2.13/etc/rc.d/init.d/functions

case"$1"instart)

      ifconfiglo:0$SNS_VIP netmask255.255.255.255 broadcast $SNS_VIP

      /sbin/route add -host $SNS_VIP dev lo:0echo"1">/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo"2">/proc/sys/net/ipv4/conf/lo/arp_announce

      echo"1">/proc/sys/net/ipv4/conf/all/arp_ignore

      echo"2">/proc/sys/net/ipv4/conf/all/arp_announce

      sysctl -p >/dev/null2>&1echo"RealServer Start OK"

      ;;

stop)

      ifconfiglo:0 down

      route del $SNS_VIP >/dev/null2>&1echo"0">/proc/sys/net/ipv4/conf/lo/arp_ignore

      echo"0">/proc/sys/net/ipv4/conf/lo/arp_announce

      echo"0">/proc/sys/net/ipv4/conf/all/arp_ignore

      echo"0">/proc/sys/net/ipv4/conf/all/arp_announce

      echo"RealServer Stoped"      ;;*)

      echo"Usage: $0 {start|stop}"      exit 1esac

exit 0

[root@bogon www]# /etc/rc.d/init.d/realserver.sh start/etc/rc.d/init.d/realserver.sh: line6: /etc/rc.d/init.d/functions: 权限不够

RealServer Start OK

[root@bogon www]# ifconfigeth0      Link encap:Ethernet  HWaddr 00:0C:29:41:71:DF 

          inet addr:192.168.12.129Bcast:192.168.12.255Mask:255.255.255.0          inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1          RX packets:728errors:0dropped:0overruns:0frame:0          TX packets:98errors:0dropped:0overruns:0carrier:0          collisions:0txqueuelen:1000

          RX bytes:137311(134.0KiB)  TX bytes:7369(7.1 KiB)

eth1      Link encap:Ethernet  HWaddr 00:0C:29:41:71:E9 

          inet addr:192.168.2.204Bcast:192.168.2.255Mask:255.255.255.0          inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1          RX packets:119838errors:0dropped:0overruns:0frame:0          TX packets:31612errors:0dropped:0overruns:0carrier:0          collisions:0txqueuelen:1000

          RX bytes:23411786(22.3MiB)  TX bytes:2119106(2.0 MiB)

lo        Link encap:Local Loopback 

          inet addr:127.0.0.1Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:65536Metric:1          RX packets:2errors:0dropped:0overruns:0frame:0          TX packets:2errors:0dropped:0overruns:0carrier:0          collisions:0txqueuelen:0

          RX bytes:182(182.0b)  TX bytes:182(182.0 b)

lo:0      Link encap:Local Loopback 

          inet addr:192.168.2.13Mask:255.255.255.255          UP LOOPBACK RUNNING  MTU:65536Metric:1

6、启动keepalived并进行测试

[root@bogon keepalived-1.2.24]# service keepalived start

正在启动 keepalived:                                      [确定]

lvs-master

[root@bogon keepalived-1.2.24]#tail-f /var/log/messages

Oct 2101:19:46bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:46bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:46bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP192.168.2.13 added

Oct 2101:19:46bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1(size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn

TCP  192.168.2.13:80rr persistent3->192.168.2.204:80Route100->192.168.2.205:80Route100

访问curl http://192.168.2.13/test.txt

[root@www etc]# curl http://192.168.2.13/test.txtit is web2

[root@www etc]# curl http://192.168.2.13/test.txtit is web2

关掉web2再次测试

[root@www etc]# curl http://192.168.2.13/test.txtit is web1

[root@www etc]# curl http://192.168.2.13/test.txtit is web1

查看lvs-master

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1(size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn

TCP  192.168.2.13:80rr persistent3->192.168.2.204:80Route102       

[root@bogon keepalived-1.2.24]#tail-f /var/log/messages

Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:28:58bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80failed after1 retry.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80from VS [192.168.2.13]:80Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 2101:29:31bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

已经自动把web2剔除

打开web2再次查看

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1(size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port          Forward Weight ActiveConn InActConn

TCP  192.168.2.13:80rr persistent3->192.168.2.204:80Route100->192.168.2.205:80Route100       

[root@bogon keepalived-1.2.24]#tail-f /var/log/messages

Oct 2101:19:51bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2101:28:58bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80failed after1 retry.

Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80from VS [192.168.2.13]:80Oct 2101:29:01bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 2101:29:31bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

Oct 2101:31:01bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success.

Oct 2101:31:01bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80to VS [192.168.2.13]:80Oct 2101:31:01bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25connected.

恢复后已自动添加进来

关掉lvs master的keepalived

[root@bogon keepalived-1.2.24]# service keepalived stop

停止 keepalived:                                          [确定]

访问web并查看lvs backup

[root@www etc]# curl http://192.168.2.13/test.txtit is web2

[root@www etc]# curl http://192.168.2.13/test.txtit is web2

[root@lys2 src]#tail-f /var/log/messages

Oct 2319:03:26lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE

Oct 2319:03:27lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE

Oct 2319:03:27lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs.

Oct 2319:03:27lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP192.168.2.13 added

Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:27lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13Oct 2319:03:32lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1for192.168.2.13

[root@lys2 src]# ip addr1: lo: mtu65536 qdisc noqueue state UNKNOWN

    link/loopback00:00:00:00:00:00brd00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever2: eth0: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff

    inet 192.168.144.101/24brd192.168.144.255 scope global eth0

    inet6 fe80::20c:29ff:fe89:fe3/64 scope link

      valid_lft forever preferred_lft forever3: eth1: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.202/24brd192.168.2.255 scope global eth1

    inet 192.168.2.13/32 scope global eth1

    inet6 fe80::20c:29ff:fe89:fed/64 scope link

      valid_lft forever preferred_lft forever


可以看到lvs backup已自动切换成master状态并自动绑定了vip

查看lvs master vip

[root@bogon keepalived-1.2.24]# ip addr1: lo: mtu65536 qdisc noqueue state UNKNOWN

    link/loopback00:00:00:00:00:00brd00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

      valid_lft forever preferred_lft forever2: eth0: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff

    inet 192.168.12.128/24brd192.168.12.255 scope global eth0

    inet6 fe80::20c:29ff:fe55:4d7a/64 scope link

      valid_lft forever preferred_lft forever3: eth1: mtu1500qdisc pfifo_fast state UP qlen1000    link/ether00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff

    inet 192.168.2.203/24brd192.168.2.255 scope global eth1

    inet6 fe80::20c:29ff:fe55:4d84/64 scope link

      valid_lft forever preferred_lft forever

已自动解除vip


到处全部结束

你可能感兴趣的:(Lvs + keepalived + nginx 高可用负载均衡)