使用lvs实现负载均衡原理以及配置详解(fullNAT模式)
lvs中nat和fullnat的区别：
nat模式下报文变化
发送 接收
cip ---> vip
cip ---> rip ( DNAT )
rip ---> cip
vip ---> cip ( SNAT )
fullnat模式下报文变化
发送 接收
cip ---> vip
lip ---> rip ( SNAT + DNAT )
rip ---> lip
vip ---> cip ( SNAT + DNAT )
注释：
cip为客户端的地址
vip为虚拟地址
rip为真实的服务器
lip为本地地址
SNAT为来源地址转换
DNAT为目的地址转换粗略的说一下自己的理解：（以我自己的实验为例子）
首先保证server1调度器端和真机可以进行数据的传输，当真机数据包过来之后到达的是server1的目的地址也就是VIP，
即就是cip ---> vip。server1要到达后端server2和server3，网段也得一致，进行了lip ---> rip，由于fullNAT
模式进行了两次SNAT+DNAT转换，数据包回来也是如此
fullNAT模式的优势：
LVS 当前应用主要采用 DR 和 NAT 模式,但这 2 种模式要求 RealServer 和 LVS
在同一个 vlan 中,导致部署成本过高;TUNNEL 模式虽然可以跨 vlan,但 RealServer
上需要部署 ipip 模块等,网络拓扑上需要连通外网,较复杂,不易运维。
为了解决上述问题,我们在 LVS 上添加了一种新的转发模式:FULLNAT,该
模式和 NAT 模式的区别是:Packet IN 时,除了做 DNAT,还做 SNAT(用户 ip->内
网 ip),从而实现 LVS-RealServer 间可以跨 vlan 通讯,RealServer 只需要连接到内
网;首先使用ipvsadm –help查看并没有fullNAT模块：
[root@server1 ~]# ipvsadm --help
1.添加fullNAT模块：
一开始更改虚拟机的内存：
在虚拟机查看空闲内存至少为大于10G：
在真机操作打开阿帕奇确保server虚拟机镜像正常：
[kiosk@foundation38 Desktop]$ systemctl start httpd.service 开启阿帕其服务在虚拟机安装软件搭建服务：
[root@server1 ~]# ls
keepalived-2.0.6 ldirectord-3.9.5-3.1.x86_64.rpm
keepalived-2.0.6.tar.gz libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm
kernel-2.6.32-220.23.1.el6.src.rpm Lvs-fullnat-synproxy.tar.gz
[root@server1 ~]# yum install -y rpm-build 解压软件
[root@server1 ~]# ls
keepalived-2.0.6 ldirectord-3.9.5-3.1.x86_64.rpm
keepalived-2.0.6.tar.gz libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm
kernel-2.6.32-220.23.1.el6.src.rpm Lvs-fullnat-synproxy.tar.gz
[root@server1 ~]# rpm -ivh kernel-2.6.32-220.23.1.el6.src.rpm 安装源码包
[root@server1 ~]# ls
keepalived-2.0.6 libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm
keepalived-2.0.6.tar.gz Lvs-fullnat-synproxy.tar.gz
kernel-2.6.32-220.23.1.el6.src.rpm rpmbuild
ldirectord-3.9.5-3.1.x86_64.rpm
[root@server1 ~]# cd rpmbuild/
[root@server1 rpmbuild]# cd SPECS/
[root@server1 SPECS]# ls
Kernel.spec
[root@server1 SPECS]# rpmbuild -bp kernel.spec 安装有依赖性
error: Failed build dependencies:
redhat-rpm-config is needed by kernel-2.6.32-220.23.1.el6.x86_64
patchutils is needed by kernel-2.6.32-220.23.1.el6.x86_64
xmlto is needed by kernel-2.6.32-220.23.1.el6.x86_64
asciidoc is needed by kernel-2.6.32-220.23.1.el6.x86_64
elfutils-libelf-devel is needed by kernel-2.6.32-220.23.1.el6.x86_64
binutils-devel is needed by kernel-2.6.32-220.23.1.el6.x86_64
newt-devel is needed by kernel-2.6.32-220.23.1.el6.x86_64
python-devel is needed by kernel-2.6.32-220.23.1.el6.x86_64
perl(ExtUtils::Embed) is needed by kernel-2.6.32-220.23.1.el6.x86_64
hmaccalc is needed by kernel-2.6.32-220.23.1.el6.x86_64
[root@server1 SPECS]# yum install redhat-rpm-config patchutils xmlto asciidoc elfutils-libelf-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed -y 依次安装解决依赖性
[root@server1 ~]#yum install -y asciidoc-8.4.5-4.1.el6.noarch.rpm 安装自己下载的包为了解决依赖性
[root@server1 ~]# yum install -y slang-devel-2.2.1-1.el6.x86_64.rpm
[root@server1 ~]# yum install newt-devel-0.52.11-3.el6.x86_64.rpm
[root@server1 ~]# cd rpmbuild/SPECS/
[root@server1 SPECS]# ls
kernel.spec
[root@server1 SPECS]# rpmbuild -bp kernel.spec 会卡着重新打开一个终端连接server1
[root@server1 ~]# yum provides */rngd 寻找所需要的安装包
Loaded plugins: product-id, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
rng-tools-2-13.el6_2.x86_64 : Random number generator related utilities
Repo : rhel-source
Matched from:
Filename : /etc/sysconfig/rngd
Filename : /sbin/rngd
Filename : /etc/rc.d/init.d/rngd
[root@server1 ~]# yum install -y rng-tools 安装工具
[root@server1 ~]# rngd -r /dev/urandom 生成随机数工具会不卡
[root@server1 SPECS]# cd
在原来的终端操作:
[root@server1 ~]# ls
asciidoc-8.4.5-4.1.el6.noarch.rpm lvs-fullnat-synproxy
keepalived-2.0.6 Lvs-fullnat-synproxy.tar.gz
keepalived-2.0.6.tar.gz newt-devel-0.52.11-3.el6.x86_64.rpm
kernel-2.6.32-220.23.1.el6.src.rpm rpmbuild
ldirectord-3.9.5-3.1.x86_64.rpm slang-devel-2.2.1-1.el6.x86_64.rpm
libnfnetlink-devel-1.0.0-1.el6.x86_64.rpm
[root@server1 ~]# cd rpmbuild/BUILD
[root@server1 BUILD]# ls
kernel-2.6.32-220.23.1.el6
[root@server1 BUILD]# cd kernel-2.6.32-220.23.1.el6/
[root@server1 kernel-2.6.32-220.23.1.el6]# ls
linux-2.6.32-220.23.1.el6.x86_64 vanilla-2.6.32-220.23.1.el6
[root@server1 kernel-2.6.32-220.23.1.el6]# cd linux-2.6.32-220.23.1.el6.x86_64/
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# ls
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# pwd
/root/rpmbuild/BUILD/kernel-2.6.32-220.23.1.el6/linux-2.6.32-220.23.1.el6.x86_64
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# cp /root/lvs-fullnat-synproxy/lvs-2.6.32-220.23.1.el6.patch . 将补丁复制到当前
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# ls
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# ll lvs-2.6.32-220.23.1.el6.patch
-rw-r--r-- 1 root root 475082 Jul 31 10:44 lvs-2.6.32-220.23.1.el6.patch
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# patch -p1 < lvs-2.6.32-220.23.1.el6.patch 进行打补丁操作
[root@server1 linux-2.6.32-220.23.1.el6.x86_64]# make 进行源码编译
root@server1 linux-2.6.32-220.23.1.el6.x86_64]# cd /boot/
[root@server1 boot]# ls
[root@server1 boot]# cd grub/
[root@server1 grub]# vim grub.conf 更改default
## 更改defult=0
[root@server1 grub]# reboot ##重启
##再次连接
[root@server1 ~]# uname -r 查看更改的内核版本
2.6.32
[root@server1 ~]# yum remove ipvsadm 卸载ipvsadm
[root@server1 ~]# cd lvs-fullnat-synproxy/
[root@server1 lvs-fullnat-synproxy]# tar zxf lvs-tools.tar.gz
[root@server1 lvs-fullnat-synproxy]# cd tools/
[root@server1 tools]# ls
ipvsadm keepalived quagga rpm
[root@server1 tools]# cd keepalived/
[root@server1 keepalived]# ls
[root@server1 keepalived]# ./configure --with-kernel-dir="/lib/modules/`uname -r`/build"
[root@server1 keepalived]# yum install -y popt-devel 安装依赖性
[root@server1 keepalived]# ./configure --with-kernel-dir="/lib/modules/`uname -r`/build" 进行源码编译三步
[root@server1 keepalived]#make
[root@server1 keepalived]#make install
[root@server1 ~]# cd lvs-fullnat-synproxy/
[root@server1 lvs-fullnat-synproxy]# ls
[root@server1 lvs-fullnat-synproxy]# cd tools/
[root@server1 tools]# ls
ipvsadm keepalived quagga rpm
[root@server1 tools]# cd ipvsadm/
[root@server1 ipvsadm]# ls
[root@server1 ipvsadm]# make 进行编译
[root@server1 ipvsadm]# make install
[root@server1 ipvsadm]# cd 
[root@server1 ~]# ipvsadm -l 进入查看大小已经变成了2的22次方
IP Virtual Server version 1.2.1 (size=4194304)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:http rr
2.fullNAT实验测试：
server1依旧作为调度器：
[root@server1 ~]# ip addr 添加一个虚拟IP
1: lo: mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:8a:d4:d6 brd ff:ff:ff:ff:ff:ff
inet 172.25.84.4/24 brd 172.25.84.255 scope global eth0
inet6 fe80::5054:ff:fe8a:d4d6/64 scope link
valid_lft forever preferred_lft forever
3: eth1: mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:1b:c4:71 brd ff:ff:ff:ff:ff:ff
inet 172.25.254.100/24 scope global eth1 
server2和server3作为后端服务器，打开阿帕其，网关指向server1的虚拟IP即可：
在server1写入策略：(fullNAT模式无法在虚拟机测试)
[root@server1 ~]# ipvsadm -C
[root@server1 ~]# ipvsadm -A -t 172.25.254.100:80 -s wrr
[root@server1 ~]# ipvsadm -a -t 172.25.254.100:80 -r 172.25.84.2:80 -b -b表示fullNAT模式
[root@server1 ~]# ipvsadm -a -t 172.25.254.100:80 -r 172.25.84.3:80 -b
[root@server1 ~]# ipvsadm -P -t 172.25.254.100:80 -z 127.0.0.1:80
[root@server1 ~]# ipvsadm -G -t 172.25.254.100:80
VIP:VPORT TOTAL SNAT_IP CONFLICTS CONNS
172.25.254.100:80 1
127.0.0.1 0 0
[root@server1 ~]# ipvsadm -ln 查看策略
IP Virtual Server version 1.2.1 (size=4194304)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.25.254.100:80 wrr
-> 172.25.84.2:80 FullNat 1 0 0
-> 172.25.84.3:80 FullNat 1 0 0
[root@server1 ~]# ipvsadm -lnc 查看策略信息
IPVS connection entries
pro expire state source virtual destination
[root@server1 ~]# 