LVS-DR模式负载均衡构建配置

过程

client->VS->RS->client(VS只做调度,RS为虚拟服务器)
DR模式：直接路由，通过改写请求报文的目的MAC地址,将请求发送给真实服务器,服务器将响应消息直接发送给客户端。

原理详情

1、通过在调度器VS上修改数据包的目的MAC地址实现转发,注意源地址仍然是CIP,目的地址仍然是VIP地址;（cip(cm，m代表mac地址）->vip(vm->RM,在2层链路层改变了mac地址来定位)）
2、请求的报文经过调度器,而服务器响应处理后的报文无需经过调度器,因此并发访问量大时使用效率很高(和NAT模式比)
3、因为DR模式是通过MAC地址改写机制实现转发,因此所有RS节点和调度器只能在一个局域网里面
4、服务器主机需要绑定VIP地址在LO接口上,且需要配置ARP抑制。
5、服务器节点的默认网关不需要配置成LB,而是直接配置为上级路由的网关,能让服务器直接出网就可以。
6、因为DR模式的调度器仅做MAC地址的改写,所以调度器就不能改写目标端口,那么服务器就得使用和VIP相同的端口提供服务。

实验环境

iptables和selinux关闭
redhat6.5
VS：server1 172.25.35.4
RS：server2 172.25.35.5
RS：server3 172.25.35.6
vip：172.25.35.100

VS：

[root@server1 ~]# ls
bansys.zip  varnish-3.0.5-1.el6.x86_64.rpm  varnish-libs-3.0.5-1.el6.x86_64.rpm
[root@server1 ~]# rm -fr *
[root@server1 ~]# ls
[root@server1 ~]# /etc/init.d/varnish start //开启服务
[root@server1 ~]# vim /etc/yum.repos.d/rhel-source.repo  
//配置yum源，内容如下：
//其中：高可用HighAvailability,负载均衡LoadBalancer，分布式存储ResilientStorage,大的文件系统ScalableFileSystem
[rhel-source]
name=Red Hat Enterprise Linux $releasever - $basearch - Source
baseurl=http://172.25.35.250/rhel6.5
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
[HighAvailability]
name=HighAvailability
baseurl=http://172.25.35.250/rhel6.5/HighAvailability
gpgcheck=0
[LoadBalancer]
name=LoadBalancer
baseurl=http://172.25.35.250/rhel6.5/LoadBalancer
gpgcheck=0

[ResilientStorage]
name=ResilientStorage
baseurl=http://172.25.35.250/rhel6.5/ResilientStorage
gpgcheck=0
[ScalableFileSystem]
name=ScalableFileSystem
baseurl=http://172.25.35.250/rhel6.5/ScalableFileSystem
gpgcheck=0
[root@server1 ~]# yum install -y ipvsadm
[root@server1 ~]# iptables -L   //查看iptables是否关闭
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
[root@server1 ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@server1 ~]# ipvsadm -A -t 172.25.35.100:80 -s rr  
//-s调度算法，默认为wlc加权调度算法
[root@server1 ~]# ipvsadm -a -t 172.25.35.100:80 -r 172.25.35.5:80 -g
//-a表示在添加虚拟服务中添加,-g表示使用直连模式
[root@server1 ~]# ipvsadm -a -t 172.25.35.100:80 -r 172.25.35.6:80 -g
[root@server1 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.35.100:80 rr
  -> 172.25.35.5:80               Route   1      0          0         
  -> 172.25.35.6:80               Route   1      0          0    
[root@server1 ~]# ip addr add 172.25.35.100/24 dev eth0  //添加虚拟ip
[root@server1 ~]# ip addr
link/ether 52:54:00:9f:e0:90     
inet 172.25.35.4/24
inet 172.25.35.100/24

RS：

注意： RS上必须有vip才能建立连接

Server2:
[root@server2 ~]# /etc/init.d/httpd start
[root@server2 html]# cat index.html 
www.westos.org-Server2
[root@server2 ~]# ip addr add 172.25.29.100/24 dev eth0
//添加虚拟ip，这里是临时添加
[root@server2 ~]# ip addr
link/ether 52:54:00:74:05:fc 
inet 172.25.35.5/24 brd 
inet 172.25.35.100/24 	

Server3:
[root@server3 ~]# /etc/init.d/httpd start
[root@server3 html]# cat index.html 
bbs.westos.org
[root@server3 ~]# ip addr add 172.25.29.100/24 dev eth0
[root@server3 ~]# ip addr
link/ether 52:54:00:10:6f:56
inet 172.25.35.6/24
inet 172.25.35.100/24

客户端测试：

发现连接到的ip（VS和RS的ip都一样）是随机的，因为三台server在同一VLAN下具有相同的vip，故不能保证每次都会访问调度器server1

[root@foundation35 rhel6.5]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 rhel6.5]# curl 172.25.35.100
www.westos.org -server2
[root@foundation35 rhel6.5]# arp -an | grep 100
//mac地址是VS的，所以访问内容有变化
? (172.25.35.100) at 52:54:00:9f:e0:90 [ether] on br0
[root@foundation35 rhel6.5]# arp -d 172.25.35.100  //down掉连接
[root@foundation35 rhel6.5]# ping 172.25.35.100  //重新连接
PING 172.25.35.100 (172.25.35.100) 56(84) bytes of data.
64 bytes from 172.25.35.100: icmp_seq=1 ttl=64 time=0.193 ms
^C
--- 172.25.35.100 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.193/0.193/0.193/0.000 ms
[root@foundation35 rhel6.5]# arp -an | grep 100  
//查看到mac地址有变化，且变化成服务器的，所以访问内容不会变化
? (172.25.35.100) at 52:54:00:74:05:fc [ether] on br0
[root@foundation35 rhel6.5]# curl 172.25.35.100
www.westos.org -server2
[root@foundation35 rhel6.5]# curl 172.25.35.100
www.westos.org -server2

为了解决这个问题，需要设置禁止访问连接RS

RS：

server2：
[root@server2 ~]# yum install arptables_jf -y
[root@server2 ~]# arptables -A IN -d 172.25.35.100 -j DROP  
//-A控制arp协议,IN添加策略,mangle转换
[root@server2 ~]# arptables -A OUT -s 172.25.35.100 -j mangle --mangle-ip-s 172.25.35.5
[root@server2 ~]# /etc/init.d/arptables_jf save
[root@server2 ~]# cat /etc/sysconfig/arptables  
//此文件存有arptables的记录，关掉之后，重启依旧存在
# Generated by arptables-save v0.0.8 on Sun Jul 29 04:15:06 2018
*filter
:IN ACCEPT [570:15960]
:OUT ACCEPT [15:420]
:FORWARD ACCEPT [0:0]
[0:0] -A IN -d 172.25.35.100 -j DROP 
[0:0] -A OUT -s 172.25.35.100 -j mangle --mangle-ip-s 172.25.35.5 
COMMIT
# Completed on Sun Jul 29 04:15:06 2018

server3：
[root@server3 ~]# yum install arptables_jf -y
[root@server3 ~]# arptables -A IN -d 172.25.35.100 -j DROP
[root@server3 ~]# arptables -A OUT -s 172.25.35.100 -j mangle --mangle-ip-s 172.25.35.6
[root@server3 ~]# /etc/init.d/arptables_jf save

客户端测试：

再次测试时ip的mac地址只有VS的

[root@foundation35 rhel6.5]# arp -an | grep 100
? (172.25.35.100) at 52:54:00:9f:e0:90 [ether] on br0
[root@foundation35 rhel6.5]# arp -d 172.25.35.100  //多次down掉后查看是否会依旧访问调度器
[root@foundation35 rhel6.5]# arp -an | grep 100
? (172.25.35.100) at  on br0
[root@foundation35 rhel6.5]# ping 172.25.35.100
PING 172.25.35.100 (172.25.35.100) 56(84) bytes of data.
64 bytes from 172.25.35.100: icmp_seq=1 ttl=64 time=0.183 ms
64 bytes from 172.25.35.100: icmp_seq=2 ttl=64 time=0.293 ms
^C
--- 172.25.35.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.183/0.238/0.293/0.055 ms
[root@foundation35 rhel6.5]# arp -an | grep 100   //查看依旧是vs的
? (172.25.35.100) at 52:54:00:9f:e0:90 [ether] on br0
[root@foundation35 rhel6.5]# curl 172.25.35.100
www.westos.org -server2
[root@foundation35 rhel6.5]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 rhel6.5]# vim /etc/hosts
172.25.35.100  server1 www.westos.org bbs.westos.org westos.org

浏览器查看为：(查看内容不同是因为server3中的域名和ip查看的发布目录不同）

Vs对后端没有健康检查

方式一：用ldirectord解决此问题

VS：

[root@server1 ~]# yum install ldirectord-3.9.5-3.1.x86_64.rpm -y
[root@server1 ~]# rpm -ql ldirectord   //查看配置文件
/usr/share/doc/ldirectord-3.9.5/ldirectord.cf
[root@server1 ~]# cp /usr/share/doc/ldirectord-3.9.5/ldirectord.cf /etc/ha.d/
[root@server1 ~]# cd /etc/ha.d
[root@server1 ha.d]# ls
ldirectord.cf  resource.d  shellfuncs
[root@server1 ha.d]# vim ldirectord.cf  //文件内容如下图

[root@server1 ha.d]# ipvsadm -C  //清理规则
[root@server1 ha.d]# ipvsadm -l  //查看是否清除
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@server1 ha.d]# /etc/init.d/ldirectord start //再次打开服务又可以加载出规则
[root@server1 ha.d]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.25.35.100:80 rr
  -> 172.25.35.5:80               Route   1      0          0         
  -> 172.25.35.6:80               Route   1      0          0    
[root@server1 ha.d]# cd /var/www/html
[root@server1 html]# ls
bansys            config.php  index.php         static
class_socket.php  index.html  purge_action.php
[root@server1 html]# rm -f *
rm: cannot remove `bansys': Is a directory
rm: cannot remove `static': Is a directory
[root@server1 html]# vim index.html
系统维护中......
//在客户端curl 172.25.35.4测试时，RS轮询，当关闭server1时，只访问server2，RS都关闭时会访问本地，而显示“系统维护中......”

客户端测试：

关闭RS的httpd

server2：
[root@server2 ~]# /etc/init.d/httpd stop

server3：
[root@server3 ~]# /etc/init.d/httpd stop

客户端测试：访问的是本地httpd的发布目录，表示可以健康监测

[root@foundation35 rhel6.5]# curl 172.25.35.100
系统维护中......

方式二：用keepalived软件解决

下载官网：http://www.keepalived.org/download.html
再建立一个子盘server4：172.25.35.7

VS:

安装keepalived：

server1:
[root@server1 ~]# tar zxf keepalived-2.0.6.tar.gz
[root@server1 ~]# cd keepalived-2.0.6
[root@server1 keepalived-2.0.6]# ./configure --prefix=/usr/local/keepalived --with-init=SYSV
[root@server1 keepalived-2.0.6]# yum install openssl-devel 
[root@server1 keepalived-2.0.6]# ./configure --prefix=/usr/local/keepalived --with-init=SYSV
[root@server1 keepalived-2.0.6]# make  //编译
[root@server1 keepalived-2.0.6]# make install
[root@server1 keepalived-2.0.6]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@server1 keepalived-2.0.6]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server1 keepalived-2.0.6]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@server1 keepalived-2.0.6]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@server1 keepalived-2.0.6]# cd /usr/local/ 
[root@server1 local]# scp -r keepalived/ server4:/usr/local/  将keepalived传给server4
[root@server1 local]# cd /etc/init.d/
[root@server1 init.d]# chmod +x keepalived 
[root@server1 init.d]# /etc/init.d/keepalived start  //开启服务

server4:
[root@server4 ~]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@server4 ~]# ln -s /usr/local/keepalived/etc/keepalived/ /etc/
[root@server4 ~]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@server4 ~]# ln -s /usr/local/keepalived/sbin/keepalived /sbin/
[root@server4 ~]# chmod +x /usr/local/keepalived/etc/init.d/keepalived
[root@server4 local]# /etc/init.d/keepalived start

配置keepalived：

server1:
[root@server1 ~]# cd /etc/keepalived/
[root@server1 keepalived]# yum install mailx -y
[root@server1 keepalived]# vim keepalived.conf //内容如下
//Virtual_router_id虚拟路由id，Delay_loop后端的健康检查，Persistence_timeout持续连接，一直保持协议磋商
global_defs {
   notification_email {
        root@localhost
        }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   #vrrp_strict				#注释以放其修改防火墙规则
   vrrp_garp_interval 0
   vrrp_gna_interval 0

vrrp_instance VI_1 {
    state MASTER					
    interface eth0
    virtual_router_id 35
    priority 100		#数值越大，优先级越高
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        172.25.35.100
        }
}

virtual_server 172.25.35.100 80 {		#VS的vip，服务启动生效时自动添加
    delay_loop 3			#对后端的健康检查时间
    lb_algo rr				#调度算法
    lb_kind DR				#模式为DR
    #persistence_timeout 50		#注释持续连接
    protocol TCP

real_server 172.25.35.5 80{		#RS
        weight 1
        TCP_CHECK{
            connect_timeout 3
            retry 3
            delay_before_retry 3
        }
    }
    real_server 172.25.35.6 80{
        weight 1
        TCP_CHECK{
            connect_timeout 3
            retry 3
            delay_before_retry 3
       }
    }
}
[root@server1 keepalived]# scp keepalived.conf server4:/etc/keepalived/
[root@server1 keepalived]# ip addr del 172.25.35.100/24 dev eth0 
[root@server1 keepalived]# /etc/init.d/keepalived restart

server4:
[root@server4 keepalived]# cd /etc/keepalived/
[root@server4 keepalived]# yum install mailx -y
[root@server4 keepalived]# vim keepalived.conf //只修改如下图内容

[root@server4 keepalived]# /etc/init.d/keepalived restart
[root@server4 keepalived]# cat /var/log/messages
Jul 29 09:25:41 server1 Keepalived_vrrp[1875]: (VI_1) Entering BACKUP STATE

负载均衡：

客户端测试：

[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 Desktop]# curl 172.25.35.100
www.westos.org -server2
[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 Desktop]# curl 172.25.35.100
www.westos.org -server2

高可用

VS：

[root@server1 keepalived]# /etc/init.d/keepalived stop
//关掉server1(也可使用echo c > /proc/sysrq-trigger破坏server1的内核使其不能使用)，客户端查看内容不会变化，因为server4接替了server1
[root@server4 keepalived]# cat /var/log/messages  //查看server4的日志
Jul 29 09:37:17 server1 Keepalived_vrrp[1875]: (VI_1) Entering MASTER STATE

RS：

[root@server2 ~]# /etc/init.d/httpd stop  //关闭server2的httpd

客户端测试：

server1关掉之后server4代替其工作，客户端依旧可以正常访问
[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 Desktop]# curl 172.25.35.100
curl: (7) Failed connect to 172.25.35.100:80; Connection refused  
//出现failed是因为设置了时间为3秒
[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org 
[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org
[root@foundation35 Desktop]# curl 172.25.35.100
bbs.westos.org

RS：

[root@server3 ~]# /etc/init.d/httpd stop  
//将两个RS都关掉，则不能正常访问，与ldirectord不同的是本地不会接替让VS访问

客户端测试：

[root@foundation35 Desktop]# curl 172.25.35.100
curl: (7) Failed connect to 172.25.35.100:80; Connection refused
[root@foundation35 Desktop]# curl 172.25.35.100
curl: (7) Failed connect to 172.25.35.100:80; Connection refused