LB群集--lvs-dr模型
lvs简介: LVS是一个开源的软件，由毕业于国防科技大学的章文嵩博士于1998年5月创立，可以实现LINUX平台下的简单负载均衡。LVS是Linux Virtual Server的缩写，意思是Linux虚拟服务器.
Lvs 的集群工作模式有3种：VS/NAT，vs/tun,vs/dr.
Lvs的调度算法：
LVS的算法分为两大类：
静态算法：只是根据算法进行调度并不考虑后端REALSERVER的实际连接情况
rr-论调算法，假如有两台服务器A，B，第一个请求给A，第二个给B，第三个给A依次往复
wrr-加权论调，假如有两台服务器A，B，A的性能是B的两倍，则在论调的同时给A上面分配的请求也大致会是B上面的两倍
dh-假如调度器的后面是两台缓存服务器A,B而不是真正的REALSERVER，则会尽可能的把相同请求或者把同一用户的请求转发到同一个缓存服务器上面以提高缓存命中率
sh-假如公司有两台防火墙让员工上网，则会把某个员工往外的访问及向内返回的请求结果定向到同一台防火墙上面，方便防火墙做established的状态检测
动态算法：前端的调度器会根据后端REALSERVER的实际连接情况来分配请求
活动链接：当前有数据包传输
非活动链接：当前连接出于建立状态但是没有数据传输
lc-同时检查后端REALSERVER上面活动状态和非活动状态的连接数使用(活动连接数*256+非活动连接数)数字小的将接收下次访问请求
wlc-加权的lc，使用(活动连接数*256+非活动连接数)/权重，数字小的将接收下次访问请求，是最常用的算法
sed-不考虑非活动状态，使用（活动状态+1）*256，数字小的将接收下次访问请求，+1主要是为了提高权重大的服务器的响应能力
nq-假设有两台服务器A,B，权重比为10：1，按照sed算法，只有当A服务器已经响应了10个请求之时两者的计算数值才相同，为了避免权重小的服务器过于空闲，nq沿用sed算法但是确保让每个服务器都不空闲，只有在不考虑非活动连接的情况下nq才能取代wlc算法
lblc-在dh的基础上面考虑后台服务器的负载能力
lblcr-在lblc的基础上，假设有A,B两台缓存服务器，某个用户第一次访问被重定向到A，第二次访问时A负载很大，B过于空闲这时也会打破原来的规则把客户的第二次访问重定向给B
Lvs的dr模型
一：Web1的搭建 192.168.0.101
[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255
配置一个vip地址
[root@zzu ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16
inet addr:192.168.0.101 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3707 errors:0 dropped:0 overruns:0 frame:0
TX packets:915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)
Interrupt:67 Base address:0x2000
lo:0 Link encap:Local Loopback
inet addr:192.168.0.100 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
设置real服务器的arp选项保证在进行arp解析的时候只有director的vip进行响应
[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf
设置一条特殊的路由，保证在回复客户端时使用的是vip的地址
[root@zzu ~]# route add -host 192.168.0.100 dev lo:0
[root@zzu ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm
[root@zzu Server]# cd /var/www/html/
[root@zzu html]# vim index.html
web1
[root@zzu html]# links http://192.168.0.101
二：web2的搭建 192.168.0.102
[root@zzu ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255
配置一个vip地址
[root@zzu ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:4B:61:16
inet addr:192.168.0.102 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe4b:6116/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3707 errors:0 dropped:0 overruns:0 frame:0
TX packets:915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:354625 (346.3 KiB) TX bytes:127781 (124.7 KiB)
Interrupt:67 Base address:0x2000
lo:0 Link encap:Local Loopback
inet addr:192.168.0.100 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
设置real服务器的arp选项保证在进行arp解析的时候只有director的vip进行响应
[root@zzu ~]# echo "net.ipv4.conf.all.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_announce = 2" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >>/etc/sysctl.conf
[root@zzu ~]# echo "net.ipv4.conf.lo.arp_ignore = 1" >>/etc/sysctl.conf
[root@zzu ~]# route add -host 192.168.0.100 dev lo:0
设置一条特殊的路由，保证在恢复客户端使使用的是vip的地址
[root@zzu ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.0.100 0.0.0.0 255.255.255.255 UH 0 0 0 lo
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
[root@zzu Server]# rpm -ivh httpd-2.2.3-31.el5.i386.rpm
[root@zzu Server]# cd /var/www/html/
[root@zzu html]# vim index.html
Web2
[root@zzu html]# links http://192.168.0.102
三：director服务器的搭建
[root@zzu ~]# yum install ipvsadm*
四：lvs-dr模型下rr的测试
[root@zzu ~]# ipvsadm -A 192.168.0.100:80 -s rr
unexpected argument 192.168.0.100:80
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g
[root@zzu ~]# ipvsdm –ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr
-> 192.168.0.102:80 Route 1 0 0
-> 192.168.0.101:80 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr
-> 192.168.0.102:80 Route 1 0 6
-> 192.168.0.101:80 Route 1 0 6
五：lvs-dr模型下rr的测试 （ppc）
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:80 -s rr -p 300
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:80 -r 192.168.0.101 –g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 0
-> 192.168.0.102:80 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 5
-> 192.168.0.102:80 Route 1 0 0
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:22 -s rr -p 300
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:22 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 5
-> 192.168.0.102:80 Route 1 0 0
TCP 192.168.0.100:22 rr persistent 300
-> 192.168.0.102:22 Route 1 0 0
-> 192.168.0.101:22 Route 1 0 0
[root@zzu ~]#
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:80 rr persistent 300
-> 192.168.0.101:80 Route 1 0 5
-> 192.168.0.102:80 Route 1 0 0
TCP 192.168.0.100:22 rr persistent 300
-> 192.168.0.102:22 Route 1 0 10
-> 192.168.0.101:22 Route 1 0 0
六：lvs-dr模型下rr的测试 （pcc）
[root@zzu ~]# ipvsadm -A -t 192.168.0.100:0 -s rr -p 300
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -a -t 192.168.0.100:0 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:0 rr persistent 300
-> 192.168.0.101:0 Route 1 0 0
-> 192.168.0.102:0 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.0.100:0 rr persistent 300
-> 192.168.0.101:0 Route 1 0 9
-> 192.168.0.102:0 Route 1 0 0
七：lvs-dr模型下带防火墙标记的持续连接（80和443）
将http和https打上标签
1：搭建https服务器web1
[root@zzu ~]# yum install openssl*
[root@zzu ~]# cd /etc/pki/
[root@zzu pki]# ll
drwx------ 3 root root 4096 2012-02-08 CA
drwxr-xr-x 2 root root 4096 2012-02-08 nssdb
drwxr-xr-x 2 root root 4096 2012-02-08 rpm-gpg
drwxr-xr-x 5 root root 4096 2012-02-08 tls
[root@zzu pki]# vim tls/openssl.cnf
45 dir = /etc/pki/CA
88 countryName = optional
89 stateOrProvinceName = optional
90 organizationName = optional
136 countryName_default = CN
141 stateOrProvinceName_default = beijing
144 localityName_default = Beijing
2：创建3个目录和两个文件
[root@zzu pki]# cd CA
[root@zzu CA]# mkdir certs newcerts crl
[root@zzu CA]# touch index.txt serial
[root@zzu CA]# echo "01" >>serial
[root@zzu CA]# openssl genrsa 1024 >private/cakey.pem
Generating RSA private key, 1024 bit long modulus
..............++++++
.....................................................................++++++
e is 65537 (0x10001)
[root@zzu CA]# openssl req -new -key private/cakey.pem -days 3650 -x509 -out cacert.pem
Country Name (2 letter code) [CN]:
State or Province Name (full name) [beijing]:
Locality Name (eg, city) [beijing]:
Organization Name (eg, company) [My Company Ltd]:qinghua
Organizational Unit Name (eg, section) []:qinghua
Common Name (eg, your name or your server's hostname) []:www.qinghua.com
2：为http办法证书
[root@zzu ~]# mkdir -pv /etc/httpd/certs
[root@zzu ~]# cd /etc/httpd/certs/
[root@zzu certs]# openssl genrsa 1024 > httpd.key
[root@zzu certs]# openssl req -new -key httpd.key -out httpd.csr
Country Name (2 letter code) [CN]:
State or Province Name (full name) [beijing]:
Locality Name (eg, city) [beijing]:
Organization Name (eg, company) [My Company Ltd]:bjdx
Organizational Unit Name (eg, section) []:sec
Common Name (eg, your name or your server's hostname) []:www.bj.com
[root@zzu certs]# openssl ca -in httpd.csr -out httpd.cert
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Feb 7 13:28:38 2012 GMT
Not After : Feb 6 13:28:38 2013 GMT
Subject:
countryName = CN
stateOrProvinceName = beijing
organizationName = bjdx
organizationalUnitName = sec
commonName = www.bj.com
3：绑紧证书文件
[root@zzu Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm
Preparing... ########################################### [100%]
1:distcache ########################################### [100%]
[root@zzu Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm
Preparing... ########################################### [100%]
1:mod_ssl ########################################### [100%]
[root@zzu ~]# cd /etc/httpd/certs/
[root@zzu certs]# cp /etc/pki/CA/cacert.pem ./
[root@zzu certs]# ll
-rw-r--r-- 1 root root 1168 02-07 21:34 cacert.pem
-rw-r--r-- 1 root root 0 02-07 21:28 httpd.cert
-rw-r--r-- 1 root root 643 02-07 21:27 httpd.csr
-rw-r--r-- 1 root root 887 02-07 21:26 httpd.key
[root@zzu ~]# vim /etc/httpd/conf.d/ssl.conf
112 SSLCertificateFile /etc/http/certs/httpd.cert
119 SSLCertificateKeyFile /etc/http/certs/httpd.key
128 SSLCertificateChainFile /etc/http/certs/cacert.pem
[root@zzu certs]# service httpd restart 重新启动www服务
Stopping httpd: [FAILED]
Starting httpd: [ OK ]
4.搭建https 服务器Web2
[root@server2 ~]# mkdir -pv /etc/httpd/certs
mkdir: created directory `/etc/httpd/certs'
[root@server2 ~]# cd /etc/httpd/certs
[root@server2 certs]# ll
total 0
[root@server2 certs]# scp 192.168.0.101:/etc/httpd/certs/* ./
The authenticity of host '192.168.0.101 (192.168.0.101)' can't be established.
RSA key fingerprint is 91:71:d8:d9:f2:63:a6:78:2f:0c:1e:e8:32:aa:55:3c.
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added '192.168.0.101' (RSA) to the list of known hosts.
[email protected]'s password:
cacert.pem 100% 1168 1.1KB/s 00:00
httpd.cert 100% 3082 3.0KB/s 00:00
httpd.csr 100% 643 0.6KB/s 00:00
httpd.key 100% 887 0.9KB/s 00:00
[root@server2 certs]# ll
-rw-r--r-- 1 root root 1168 Apr 30 17:33 cacert.pem
-rw-r--r-- 1 root root 3082 Apr 30 17:33 httpd.cert
-rw-r--r-- 1 root root 643 Apr 30 17:33 httpd.csr
-rw-r--r-- 1 root root 887 Apr 30 17:33 httpd.key
[root@server2 Server]# rpm -ivh distcache-1.4.5-14.1.i386.rpm
[root@server2 Server]# rpm -ivh mod_ssl-2.2.3-31.el5.i386.rpm
[root@server2~]#scp192.168.0.101:/etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf
[email protected]'s password:
ssl.conf 100% 9655 9.4KB/s 00:00
[root@server2 ~]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: httpd: apr_sockaddr_info_get() failed for server2
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName
[ OK ]
5：设置director服务器
[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 80 -j MARK --set-mark 1
[root@zzu ~]# iptables -t mangle -A PREROUTING -d 192.168.0.100 -p tcp --dport 443 -j MARK --set-mark 1
[root@zzu ~]# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere 192.168.0.100 tcp dpt:http MARK set 0x1
MARK tcp -- anywhere 192.168.0.100 tcp dpt:https MARK set 0x1
[root@zzu ~]# ipvsadm -A -f 1 -s rr -p 1800
[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -f 1 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 rr persistent 1800
-> 192.168.0.102:0 Route 1 0 0
-> 192.168.0.101:0 Route 1 0 0
访问测试：
八：lvs-dr模型下带防火墙标记的持续连接（ftp的被动方式）
ftp1 服务器
[root@zzu Server]# rpm -ivh vsftpd-2.0.5-16.el5.i386.rpm
Preparing... ########################################### [100%]
1:vsftpd ########################################### [100%]
[root@zzu ~]# cd /var/ftp/
[root@zzu ftp]# mkdir ftp1
[root@zzu ftp]# ll
total 8
drwxr-xr-x 2 root root 4096 Feb 7 22:27 ftp1
drwxr-xr-x 3 root root 4096 Feb 7 22:26 pub
[root@zzu ~]# vim /etc/vsftpd/vsftpd.conf
12 pasv_min_port=10000
13 pasv_max_port=20000
14 pasv_enable=YES
[root@zzu ftp]# service vsftpd restart
Shutting down vsftpd: [ OK ]
Starting vsftpd for vsftpd: [ OK ]
ftp2服务器同一
director服务器的设置
[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 10000:20000 -j MARK --set-mark 21
[root@zzu ~]# iptables -t mangle -A PREROUTING -p tcp -d 192.168.0.100/32 --dport 21 -j MARK --set-mark 21
[root@zzu ~]# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- anywhere 192.168.0.100 tcp dpts:ndmp:dnp MARK set 0x15
MARK tcp -- anywhere 192.168.0.100 tcp dpt:ftp MARK set 0x15
[root@zzu ~]# ipvsadm -A -f 21 -s rr -p 1800
[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.101 -g
[root@zzu ~]# ipvsadm -a -f 21 -r 192.168.0.102 -g
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 21 rr persistent 1800
-> 192.168.0.102:0 Route 1 0 0
-> 192.168.0.101:0 Route 1 0 0
[root@zzu ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 21 rr persistent 1800
-> 192.168.0.102:0 Route 1 0 8
-> 192.168.0.101:0 Route 1 0 0
[root@zzu ~]# ipvsadm –lcn 我们在director上查看链接的状态
IPVS connection entries
pro expire state source virtual destination
TCP 00:19 FIN_WAIT 192.168.0.5:1309 192.168.0.100:21 192.168.0.102:21
TCP 00:19 FIN_WAIT 192.168.0.5:1310 192.168.0.100:10499 192.168.0.102:10499
TCP 00:14 FIN_WAIT 192.168.0.5:1306 192.168.0.100:14859 192.168.0.102:14859
TCP 00:14 FIN_WAIT 192.168.0.5:1305 192.168.0.100:21 192.168.0.102:21
IP 28:19 ERR! 192.168.0.5:0 0.0.0.21:0 192.168.0.102:0
欢迎加入郑州阳仔的网络工程师自由交流群--132444800(请注明自己的身份，就说是51cto的博友）