lvs-dr 示例详解
lvs-dr 示例详解
- lvs-dr 场景介绍
- 1、lvs-dr 注意事项
- 2、lvs-dr 的问题及解决办法
- 3、lvs-dr 解决MAC地址冲突
- 4、lvs-dr RS 预配置脚本
- 5、lvs-dr VS 预配置脚本
- lvs-dr 场景示例
- 1、绘制拓扑
- 2、各个虚拟机的配置
- 2.1 RS1 和 RS2（配置 Nginx 服务）
- 2.2 RS1 和 RS2（配置默认路由，指向 R）
- 2.3 VS 进行验证，看能否正常访问 RS1 和 RS2
- 2.4 VS 安装 ipvsadm
- 2.5 VS IP地址配置
- 2.6 RS IP地址 和 ARP响应 、路由 配置
- 3、调度算法-RR
- 3.1 VS 进行集群设置和RS设置
- 3.2 用户A 进行web访问
- 3.3 用户B 进行web访问
- 3.4 VS 的统计信息查看
- 4、调度算法-WRR
- 4.1 VS 进行集群设置和RS设置
- 4.2 用户A 进行web访问
- 4.3 用户B 进行web访问
- 4.4 VS 的统计信息查看（符合权重比 3:1）
lvs-dr 场景介绍
通过为请求报文重新封装一个MAC首部进行转发，源MAC是DIP所在的接口的MAC，目标MAC是某挑选出的RS的RIP所在接口的MAC地址；源IP/PORT，以及目标IP/PORT均保持不变。
1、lvs-dr 注意事项
- Director和各RS都得配置使用VIP
- 确保前端路由器将目标IP为VIP的请求报文发往Director
- RS的RIP可以使用私网地址，也可以是公网地址；RIP与DIP在同一IP网络；RIP的网关不能指向DIP，以确保响应报文不会经由Director
- RS跟Director要在同一个物理网络
- 请求报文要经由Director，但响应不能经由Director，而是由RS直接发往Client
- 不支持端口映射
2、lvs-dr 的问题及解决办法
- Director和各RS都得配置使用VIP，如何保证ARP请求报文和响应都只有Director的MAC地址能够响应？（解决办法：修改Liunx内核中关于ARP的内核参数：arp_announce/arp_ignore）
- 由于RS只有一个网络接口，而Linux内核默认情况下包封装时是从哪个网口出去那个网口就是源地址，如何确保RS转发出来的报文源IP是VIP，而不是RIP？（解决办法：在数据报封装后指定路由经lo别名上的接口发送出去然后再通过物理网卡转发出去）
- Director和各RS都得配置使用VIP，如何避免个主机的IP广播，造成无法通信？（解决办法：在RS上进行配置时，设定VIP的广播地址为它自己即可）
3、lvs-dr 解决MAC地址冲突
dr模型中，各主机上均需要配置VIP，解决地址冲突的方式有三种：
-
在前端网关做静态绑定
-
在各RS使用arptables
-
在各RS修改内核参数，来限制arp响应和通告的级别
限制响应级别：arp_ignore 0 # 默认值，表示可使用本地任意接口上配置的任意地址进行响应 1 # 仅在请求的目标IP配置在本地主机的接收到请求报文接口上时，才给予响应 限制通告级别：arp_announce 0 # 默认值，把本机上的所有接口的所有信息向每个接口上的网络进行通告 1 # 尽量避免向非直接连接网络进行通告 2 # 必须避免向非本网络通告
4、lvs-dr RS 预配置脚本
#!/bin/bash
#
vip=10.1.0.5
mask='255.255.255.255'
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig lo:0 $vip netmask $mask broadcast $vip up
route add -host $vip dev lo:0
;;
stop)
ifconfig lo:0 down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
;;
esac
5、lvs-dr VS 预配置脚本
#!/bin/bash
#
vip='10.1.0.5'
iface='eno16777736:0'
mask='255.255.255.255'
port='80'
rs1='10.1.0.7'
rs2='10.1.0.8'
scheduler='wrr'
type='-g'
case $1 in
start)
ifconfig $iface $vip netmask $mask broadcast $vip up
iptables -F
ipvsadm -A -t ${vip}:${port} -s $scheduler
ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
;;
stop)
ipvsadm -C
ifconfig $iface down
;;
*)
echo "Usage $(basename $0) start|stop"
exit 1
;;
esac
lvs-dr 场景示例
1、绘制拓扑
2、各个虚拟机的配置
2.1 RS1 和 RS2（配置 Nginx 服务）
-
RS1
[root@neo-neo ~]# ifconfig | grep "inet 192" inet 192.168.1.12 netmask 255.255.255.0 broadcast 192.168.10.255 [root@neo-neo ~]# cat /var/www/html/index.html <h1>RS1 web-server</h1> [root@neo-neo ~]# ss -tnl | grep 80 LISTEN 0 128 :::80 :::* -
RS2
[root@Neo_Tang ~]# ifconfig | grep "inet 192" inet 192.168.1.13 netmask 255.255.255.0 broadcast 192.168.10.255 [root@Neo_Tang ~]# cat /var/www/html/index.html <h1>RS2 web-server</h1> [root@Neo_Tang ~]# ss -tnl | grep 80 LISTEN 0 128 :::80 :::*
2.2 RS1 和 RS2（配置默认路由，指向 R）
-
RS1
[root@neo-neo ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 102 0 0 ens33 192.168.1.0 0.0.0.0 255.255.255.0 U 102 0 0 ens33 -
RS2
[root@Neo_Tang ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 101 0 0 ens33 192.168.1.0 0.0.0.0 255.255.255.0 U 101 0 0 ens33
2.3 VS 进行验证，看能否正常访问 RS1 和 RS2
[root@Neo_Tang ~]# curl http://192.168.1.12
<h1>RS1 web-server 192.168.1.12</h1>
[root@Neo_Tang ~]# curl http://192.168.1.13
<h1>RS2 web-server 192.168.1.13</h1>
2.4 VS 安装 ipvsadm
[root@Neo_Tang ~]# yum install ipvsadm -y
[root@Neo_Tang ~]# rpm -ql ipvsadm
/etc/sysconfig/ipvsadm-config
/usr/lib/systemd/system/ipvsadm.service
/usr/sbin/ipvsadm
/usr/sbin/ipvsadm-restore
/usr/sbin/ipvsadm-save
/usr/share/doc/ipvsadm-1.27
/usr/share/doc/ipvsadm-1.27/README
/usr/share/man/man8/ipvsadm-restore.8.gz
/usr/share/man/man8/ipvsadm-save.8.gz
/usr/share/man/man8/ipvsadm.8.gz
2.5 VS IP地址配置
[root@Neo_Tang ~]# ifconfig ens33:0 192.168.1.99/32 up
[root@Neo_Tang ~]# ifconfig | grep -B 2 "inet 192"
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.11 netmask 255.255.255.0 broadcast 192.168.1.255
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.99 netmask 0.0.0.0 broadcast 255.255.255.255
2.6 RS IP地址 和 ARP响应 、路由 配置
-
RS1
[root@neo-neo ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@neo-neo ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore [root@neo-neo ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce [root@neo-neo ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce [root@neo-neo ~]# ifconfig lo:0 192.168.1.99/32 broadcast 192.168.1.99 up [root@neo-neo ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.12 netmask 255.255.255.0 broadcast 192.168.1.255 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.1.99 netmask 0.0.0.0 loop txqueuelen 1000 (Local Loopback) [root@neo-neo ~]# route add -host 192.168.1.99 dev lo:0 [root@neo-neo ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 102 0 0 ens33 192.168.1.0 0.0.0.0 255.255.255.0 U 102 0 0 ens33 192.168.1.99 0.0.0.0 255.255.255.255 UH 0 0 0 lo -
RS2
[root@Neo_Tang ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore [root@Neo_Tang ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore [root@Neo_Tang ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce [root@Neo_Tang ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce [root@Neo_Tang ~]# [root@Neo_Tang ~]# ifconfig lo:0 192.168.1.99/32 broadcast 192.168.1.99 up [root@Neo_Tang ~]# ifconfig ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.13 netmask 255.255.255.0 broadcast 192.168.1.255 lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.1.99 netmask 0.0.0.0 loop txqueuelen 1000 (Local Loopback) [root@Neo_Tang ~]# route add -host 192.168.1.99 dev lo:0 [root@Neo_Tang ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 101 0 0 ens33 192.168.1.0 0.0.0.0 255.255.255.0 U 101 0 0 ens33 192.168.1.99 0.0.0.0 255.255.255.255 UH 0 0 0 lo
3、调度算法-RR
3.1 VS 进行集群设置和RS设置
[root@Neo_Tang ~]# ipvsadm -A -t 192.168.1.99:80 -s rr
[root@Neo_Tang ~]# ipvsadm -a -t 192.168.1.99:80 -r 192.168.1.12 -g
[root@Neo_Tang ~]# ipvsadm -a -t 192.168.1.99:80 -r 192.168.1.13 -g
[root@Neo_Tang ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.99:80 rr
-> 192.168.1.12:80 Route 1 0 0
-> 192.168.1.13:80 Route 1 0 0
3.2 用户A 进行web访问
[root@Tang-Neo ~]# for i in {1..10}; do curl http://192.168.1.99/index.html; done
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
3.3 用户B 进行web访问
[root@neo ~]# for i in {1..10}; do curl http://192.168.1.99/index.html; done
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
3.4 VS 的统计信息查看
[root@Neo_Tang ~]# ipvsadm -Ln -c
IPVS connection entries
pro expire state source virtual destination
TCP 00:56 FIN_WAIT 192.168.1.9:40314 192.168.1.99:80 192.168.1.12:80
TCP 00:56 FIN_WAIT 192.168.1.9:40322 192.168.1.99:80 192.168.1.12:80
TCP 01:09 FIN_WAIT 192.168.1.10:59242 192.168.1.99:80 192.168.1.12:80
TCP 01:09 FIN_WAIT 192.168.1.10:59240 192.168.1.99:80 192.168.1.13:80
TCP 00:56 FIN_WAIT 192.168.1.9:40318 192.168.1.99:80 192.168.1.12:80
TCP 00:56 FIN_WAIT 192.168.1.9:40312 192.168.1.99:80 192.168.1.13:80
TCP 00:56 FIN_WAIT 192.168.1.9:40316 192.168.1.99:80 192.168.1.13:80
TCP 00:56 FIN_WAIT 192.168.1.9:40310 192.168.1.99:80 192.168.1.12:80
TCP 00:56 FIN_WAIT 192.168.1.9:40326 192.168.1.99:80 192.168.1.12:80
TCP 00:56 FIN_WAIT 192.168.1.9:40308 192.168.1.99:80 192.168.1.13:80
TCP 01:08 FIN_WAIT 192.168.1.10:59232 192.168.1.99:80 192.168.1.13:80
TCP 01:09 FIN_WAIT 192.168.1.10:59236 192.168.1.99:80 192.168.1.13:80
TCP 00:56 FIN_WAIT 192.168.1.9:40320 192.168.1.99:80 192.168.1.13:80
TCP 00:56 FIN_WAIT 192.168.1.9:40324 192.168.1.99:80 192.168.1.13:80
TCP 01:08 FIN_WAIT 192.168.1.10:59230 192.168.1.99:80 192.168.1.12:80
TCP 01:08 FIN_WAIT 192.168.1.10:59228 192.168.1.99:80 192.168.1.13:80
TCP 01:09 FIN_WAIT 192.168.1.10:59244 192.168.1.99:80 192.168.1.13:80
TCP 01:09 FIN_WAIT 192.168.1.10:59238 192.168.1.99:80 192.168.1.12:80
TCP 01:09 FIN_WAIT 192.168.1.10:59246 192.168.1.99:80 192.168.1.12:80
TCP 01:08 FIN_WAIT 192.168.1.10:59234 192.168.1.99:80 192.168.1.12:80
[root@Neo_Tang ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.99:80 rr
-> 192.168.1.12:80 Route 1 0 10
-> 192.168.1.13:80 Route 1 0 10
[root@Neo_Tang ~]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 192.168.1.99:80 20 120 0 8120 0
-> 192.168.1.12:80 10 60 0 4060 0
-> 192.168.1.13:80 10 60 0 4060 0
4、调度算法-WRR
4.1 VS 进行集群设置和RS设置
[root@Neo_Tang ~]# ipvsadm -E -t 192.168.1.99:80 -s wrr
[root@Neo_Tang ~]# ipvsadm -e -t 192.168.1.99:80 -r 192.168.1.12 -g -w 3
[root@Neo_Tang ~]# ipvsadm -e -t 192.168.1.99:80 -r 192.168.1.13 -g -w 1
[root@Neo_Tang ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.99:80 wrr
-> 192.168.1.12:80 Route 3 0 0
-> 192.168.1.13:80 Route 1 0 0
[root@Neo_Tang ~]# ipvsadm -Z
[root@Neo_Tang ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.99:80 wrr
-> 192.168.1.12:80 Route 3 0 0
-> 192.168.1.13:80 Route 1 0 0
[root@Neo_Tang ~]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 192.168.1.99:80 0 0 0 0 0
-> 192.168.1.12:80 0 0 0 0 0
-> 192.168.1.13:80 0 0 0 0 0
4.2 用户A 进行web访问
[root@Tang-Neo ~]# for i in {1..10}; do curl http://192.168.1.99/index.html; done
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
4.3 用户B 进行web访问
[root@neo ~]# for i in {1..10}; do curl http://192.168.1.99/index.html; done
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS2 web-server 192.168.1.13</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
<h1>RS1 web-server 192.168.1.12</h1>
4.4 VS 的统计信息查看（符合权重比 3:1）
[root@Neo_Tang ~]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP 192.168.1.99:80 20 120 0 8120 0
-> 192.168.1.12:80 15 90 0 6090 0
-> 192.168.1.13:80 5 30 0 2030 0
[root@Neo_Tang ~]# ipvsadm -Ln -c
IPVS connection entries
pro expire state source virtual destination
TCP 00:47 FIN_WAIT 192.168.1.10:59266 192.168.1.99:80 192.168.1.12:80
TCP 00:47 FIN_WAIT 192.168.1.10:59256 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40334 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40346 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40340 192.168.1.99:80 192.168.1.13:80
TCP 00:41 FIN_WAIT 192.168.1.9:40338 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40350 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40342 192.168.1.99:80 192.168.1.12:80
TCP 00:47 FIN_WAIT 192.168.1.10:59258 192.168.1.99:80 192.168.1.12:80
TCP 00:47 FIN_WAIT 192.168.1.10:59250 192.168.1.99:80 192.168.1.12:80
TCP 00:47 FIN_WAIT 192.168.1.10:59260 192.168.1.99:80 192.168.1.13:80
TCP 00:47 FIN_WAIT 192.168.1.10:59252 192.168.1.99:80 192.168.1.13:80
TCP 00:47 FIN_WAIT 192.168.1.10:59264 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40332 192.168.1.99:80 192.168.1.13:80
TCP 00:47 FIN_WAIT 192.168.1.10:59254 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40348 192.168.1.99:80 192.168.1.13:80
TCP 00:47 FIN_WAIT 192.168.1.10:59262 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40336 192.168.1.99:80 192.168.1.12:80
TCP 00:47 FIN_WAIT 192.168.1.10:59248 192.168.1.99:80 192.168.1.12:80
TCP 00:41 FIN_WAIT 192.168.1.9:40344 192.168.1.99:80 192.168.1.12:80