linux集群服务[LVS负载均衡集群服务]——————安装LVS集群、配置LVS负载均衡服务、DR模式、NAT模式、TUN模式

1. 安装LVS集群

1）查看安装LVS管理工具：

• LVS-server主

[root@lvs-server ~]# rpm -qa ipvsadm
[root@lvs-server ~]# yum install ipvsadm -y

Installed:
  ipvsadm.x86_64 0:1.27-7.el7                                                   

Complete!
[root@lvs-server ~]# rpm -qa ipvsadm
ipvsadm-1.27-7.el7.x86_64
[root@lvs-server ~]#

• LVS-server备

[root@lvs2-server ~]# yum install ipvsadm -y
Installed:
  ipvsadm.x86_64 0:1.27-7.el7                                                                      

Complete!
[root@lvs2-server ~]# rpm -qa ipvsadm
ipvsadm-1.27-7.el7.x86_64
[root@lvs2-server ~]#

2）内核层面的，所以需要链接一下：

• LVS-server主

[root@lvs-server ~]# ln -s /usr/src/kernels/`uname -r` /usr/src/linux
[root@lvs-server ~]# ll /usr/src/
total 0
drwxr-xr-x. 2 root root  6 Mar 10  2016 debug
drwxr-xr-x. 2 root root  6 Mar 10  2016 kernels
lrwxrwxrwx  1 root root 38 Aug 31 15:09 linux -> /usr/src/kernels/3.10.0-514.el7.x86_64

• LVS-server备

[root@lvs2-server ~]# ln -s /usr/src/kernels/`uname -r` /usr/src/linux
[root@lvs2-server ~]# ll /usr/src/
total 0
drwxr-xr-x. 2 root root  6 Mar 10  2016 debug
drwxr-xr-x. 2 root root  6 Mar 10  2016 kernels
lrwxrwxrwx  1 root root 38 Aug 31 15:21 linux -> /usr/src/kernels/3.10.0-514.el7.x86_64

3）内核中查看ipvs：没有的话输入命令（ipvsadm、modprobe ip_vs）

• LVS-server主

[root@lvs-server ~]# ipvsadm # 
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@lvs-server ~]# lsmod | grep ip_vs
ip_vs                 141092  0 
nf_conntrack          111302  8 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c              12644  2 xfs,ip_vs

• LVS-server备

[root@lvs2-server ~]# modprobe ip_vs
[root@lvs2-server ~]# lsmod | grep ip_vs
ip_vs                 141092  0 
nf_conntrack          111302  6 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4
libcrc32c              12644  2 xfs,ip_vs

2.配置LVS负载均衡服务

2.1 DR模式

使用vip：10.0.0.10，域名www.example.com（对应服务）。

1）给LVS配置VIP地址：

• LVS-server主

[root@lvs-server ~]# ip addr show ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:5b:cc:9a brd ff:ff:ff:ff:ff:ff
    inet 172.25.5.10/24 brd 172.25.5.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::4608:756c:3af9:e967/64 scope link 
       valid_lft forever preferred_lft forever
[root@lvs-server ~]# ip addr add 10.0.0.10/24 dev ens3 label ens3:0
[root@lvs-server ~]# ip addr show ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:5b:cc:9a brd ff:ff:ff:ff:ff:ff
    inet 172.25.5.10/24 brd 172.25.5.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet 10.0.0.10/24 scope global ens3:0
       valid_lft forever preferred_lft forever
    inet6 fe80::4608:756c:3af9:e967/64 scope link 
       valid_lft forever preferred_lft forever
[root@lvs-server ~]# ifconfig 
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.25.5.10  netmask 255.255.255.0  broadcast 172.25.5.255
        inet6 fe80::4608:756c:3af9:e967  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:5b:cc:9a  txqueuelen 1000  (Ethernet)
        RX packets 1300  bytes 184543 (180.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 459  bytes 63407 (61.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens3:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.10  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 52:54:00:5b:cc:9a  txqueuelen 1000  (Ethernet)

2）清空所有ipvs规则：

[root@lvs-server ~]# ipvsadm -C

3）添加一个虚拟服务器：

[root@lvs-server ~]# ipvsadm -A -t 10.0.0.10:80 -s wrr

4）添加节点：(后端1tomcat8080端口的+后端2nginx80端口的)

[root@lvs-server ~]# ipvsadm -a -t 10.0.0.10:80 -r 172.25.5.15:8080 -g -w 1
[root@lvs-server ~]# ipvsadm -a -t 10.0.0.10:80 -r 172.25.5.11:80 -g -w 1

5）查看：

[root@lvs-server ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.10:80 wrr
  -> 172.25.5.11:80               Route   1      0          0         
  -> 172.25.5.15:80               Route   1      0          0

6）现在客户无法看到，因为服务器端没有VIP，发现这个不是自己的包。

7）两个客户端添加VIP：

• tomcat

[root@tomcat ~]# ip addr add 10.0.0.10/32 dev lo label lo:1
[root@tomcat ~]# ifconfig

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo:1: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 10.0.0.10  netmask 255.255.255.255
        loop  txqueuelen 1  (Local Loopback)

• nginx

[root@nginx2 sbin]# ip addr add 10.0.0.10/32 dev lo label lo:1
[root@nginx2 sbin]# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo:1: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 10.0.0.10  netmask 255.255.255.255
        loop  txqueuelen 1  (Local Loopback)

8）服务器添加网关为负载均衡服务器的IP：

9）做arp抑制使后端的真实服务器，无法接收到询问VIP的数据包：

• tomcat

[root@tomcat ~]# yum install arptables.x86_64 -y
Installed:
  arptables.x86_64 0:0.0.4-8.el7                                                                   

Complete!

[root@tomcat ~]# arptables -A OUTPUT -s 10.0.0.10 -j mangle --mangle-ip-s 172.25.5.15
[root@tomcat ~]# arptables -nL
Chain INPUT (policy ACCEPT)
-j DROP -d  10.0.0.10

Chain OUTPUT (policy ACCEPT)
-j mangle -s 10.0.0.10 --mangle-ip-s 172.25.5.15 

Chain FORWARD (policy ACCEPT)

[root@tomcat ~]# arptables-save > /etc/sysconfig/arptables
[root@tomcat ~]# cat /etc/sysconfig/arptables
*filter
:INPUT ACCEPT
:OUTPUT ACCEPT
:FORWARD ACCEPT
-A INPUT -j DROP -d 10.0.0.10 
-A OUTPUT -j mangle -s 10.0.0.10 --mangle-ip-s 172.25.5.15
[root@tomcat ~]# systemctl start arptables.service

• nginx

[root@nginx2 ~]# yum install -y arptables.x86_64

Installed:
  arptables.x86_64 0:0.0.4-8.el7                                                                   

Complete!

[root@nginx2 ~]# arptables -A INPUT -d 10.0.0.10  -j DROP
[root@nginx2 ~]# arptables -A OUTPUT -s 10.0.0.10 -j mangle --mangle-ip-s 172.25.5.11
[root@nginx2 ~]# arptables -nL
Chain INPUT (policy ACCEPT)
-j DROP -d 10.0.0.10 

Chain OUTPUT (policy ACCEPT)
-j mangle -s 10.0.0.10 --mangle-ip-s 172.25.5.11 

Chain FORWARD (policy ACCEPT)
[root@nginx2 ~]# arptables-save > /etc/sysconfig/arptables
[root@nginx2 ~]# cat /etc/sysconfig/arptables
*filter
:INPUT ACCEPT
:OUTPUT ACCEPT
:FORWARD ACCEPT
-A INPUT -j DROP -d 10.0.0.10 
-A OUTPUT -j mangle -s 10.0.0.10 --mangle-ip-s 172.25.5.11
[root@nginx2 ~]# systemctl start arptables.service

10）开启ipvs服务：

[root@lvs-server ~]# ipvsadm --save > /etc/sysconfig/ipvsadm
[root@lvs-server ~]# systemctl start ipvsadm.service

11）客户端开始访问：

[root@lvs-master ~]# curl 10.0.0.10
web2!!!web2!!!
[root@lvs-master ~]# curl 10.0.0.10
web1!!!
[root@lvs-master ~]# curl 10.0.0.10
web2!!!web2!!!
[root@lvs-master ~]# curl 10.0.0.10
web1!!!
[root@lvs-master ~]# curl 10.0.0.10
web2!!!web2!!!
[root@lvs-master ~]# curl 10.0.0.10
web1!!!

总结：

• 如果显示错误illegal virtual server address[:port] specified，说明你选取的ip段非法，选取私网。
• LVS的虚拟ip必须是24位，而后端web需要是32位。

2.2 NAT模式

NAT模式中的LVS就像一个路由一样，所以要打开那个路由功能。
cip：10.0.0.13
VIP：10.0.0.10
DIP：172.25.5.1
rip：172.25.5.15
rip2：172.25.5.16

1）给LVS添加24位VIP和24位DIP。

2）两个后端只需要有自己的rip就可以了。

3）LVS服务器添加NAT策略：

[root@lvs-master ~]# ipvsadm -A -t 10.0.0.10:80 -s rr
[root@lvs-master ~]# ipvsadm -a -t 10.0.0.10:80 -r 172.25.5.15:80 -m
[root@lvs-master ~]# ipvsadm -a -t 10.0.0.10:80 -r 172.25.5.16:80 -m
[root@lvs-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.10:80 rr
  -> 172.25.5.15:80               Masq    1      0          0         
  -> 172.25.5.16:80               Masq    1      0          0

4）重启ipvsadm服务：

[root@lvs-master ~]# systemctl restart ipvsadm.service 
[root@lvs-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.10:80 rr
  -> 172.25.5.15:80               Masq    1      0          0         
  -> 172.25.5.16:80               Masq    1      0          0

5）开启LVS服务器的内核路由功能：

[root@lvs-master ~]# sysctl -p | grep ip_forward
[root@lvs-master ~]# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_use_pmtu = 0
[root@lvs-master ~]# vim /etc/sysctl.conf 

net.ipv4.ip_forward = 1
:wq
[root@lvs-master ~]# sysctl -p 
net.ipv4.ip_forward = 1
[root@lvs-master ~]# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0

6）将后端两个web服务器的网关设置为LVS服务器的DIP。

7）客户端进行测试：

[root@client ~]# curl 10.0.0.10
web2!!!web2!!!
[root@client ~]# curl 10.0.0.10
web1!!!
[root@client ~]# curl 10.0.0.10
web2!!!web2!!!
[root@client ~]# curl 10.0.0.10
web1!!!

2.3 TUN模式