LVS负载均衡集群

 

【centos6.5】

一、LVS-NAT案例

二、LVS-DR案例

一、LVS-NAT案例

实验环境：

每台主机上都以安装好apache（现实中用nginx），实验建议用apache，实验反应速度块

LVS调度器（BL）作为web、web2俩台web服务器池的网关

BL俩块网卡，分别连接内外网

BL主机（负载调度器）

外网地址：172.16.23.10，同时也作为整个群集的VIP

内网地址：192.168.23.10

Web 主机：192.168.23.131

Web2主机：192.168.23.129

NFS主机：192.168.23.130

【所有的主机都需要手动配置ip，并添加网关】

[root@BL ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

HWADDR=00:0c:29:ac:ff:92  【复制本地Mac地址】

TYPE=Ethernet

#UUID=b7ea0783-e15f-4f78-84ff-017f1c97781d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=192.168.23.10

NETMASK=255.255.255.0

GATEWAY=192.168.23.1

[root@BL ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1

HWADDR=00:0c:29:ac:ff:9c 【复制本地mac地址】

TYPE=Ethernet

#UUID=b7ea0783-e15f-4f78-84ff-017f1c97781d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=172.16.23.10

NETMASK=255.255.0.0

GATEWAY=172.16.0.1

【web2的网关必须是BL的ip192.168.23.10】

[root@web ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2

HWADDR=00:0c:29:b1:a6:35

TYPE=Ethernet

#UUID=b7ea0783-e15f-4f78-84ff-017f1c97781d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=192.168.23.131

GATEWAY=192.168.23.10

NETMASK=255.255.255.0

【web2的网关必须是BL的ip192.168.23.10】

[root@web2 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2

HWADDR=00:0c:29:b2:da:2b

TYPE=Ethernet

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=192.168.23.129

GATEWAY=192.168.23.10

NETMASK=255.255.255.0

1.1、NFS共享存储部署

1.1.1、安装nfs-utils rpcbind软件包

[root@NFS ~]# rpm -q nfs-utils rpcbind

nfs-utils-1.2.3-39.el6.x86_64

rpcbind-0.2.0-11.el6.x86_64

1.1.2 设置共享目录，将/opt/www作为共享目录，要求192.168.23.0/24网段只读取权限访问网页数据（若增加读写速度和安全性，可结合raid5+LVM）

[root@NFS ~]# mkdir /opt/www

[root@NFS ~]# vim /etc/exports

/opt/www        192.168.23.0/24(ro,sync,no_root_squash)

保存退出

1.1.3 启动服务并查看共享目录

[root@NFS ~]# /etc/init.d/rpcbind start

[root@NFS ~]# /etc/init.d/nfs start

启动 NFS 服务：                                            [确定]

关掉 NFS 配额：                                            [确定]

启动 NFS mountd：                                          [确定]

启动 NFS 守护进程：                                        [确定]

正在启动 RPC idmapd：                                      [确定]

[root@NFS ~]# showmount -e 192.168.23.130

Export list for 192.168.23.130:

/opt/www 192.168.23.0/24

[root@NFS ~]# chkconfig rpcbind on

[root@NFS ~]# chkconfig nfs on

[root@NFS ~]# echo "

it works

" >/opt/www/index.html

1.2  web 、web2服务器配置

1.2.1 web 配置

[root@web ~]# rpm -q httpd

[root@web ~]#ls

[root@web ~]#tar xf httpd-2.2.17.tar.gz -C /usr/src/

[root@web ~]#cd /usr/src/httpd-2.2.17/

[root@web ~]#./configure --prefix=/usr/local/apache && make && make install

[root@web ~]#sed -i '/#S/ s/#//' /usr/local/apache/conf/httpd.conf

[root@web ~]# sed -n '/com:80/p' /usr/local/apache/conf/httpd.conf

ServerName www.example.com:80

[root@web ~]#/usr/local/apache/bin/apachectl start

[root@web ~]# netstat -anpt |grep httpd

tcp        0      0 :::80                       :::*                        LISTEN      24700/httpd

[root@web ~]# yum -y install nfs-utils

[root@web ~]# rpm -q nfs-utils

nfs-utils-1.2.3-39.el6.x86_64

[root@BL ~]# sed -n '7p'  /etc/sysctl.conf

net.ipv4.ip_forward = 1

[root@BL ~]# sysctl -p  【使内核参数立即生效】

net.ipv4.ip_forward = 1

net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.accept_source_route = 0

kernel.sysrq = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key

error: "net.bridge.bridge-nf-call-iptables" is an unknown key

error: "net.bridge.bridge-nf-call-arptables" is an unknown key

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.shmmax = 68719476736

kernel.shmall = 4294967296

1.3  LVS-NAT 部署

1.3.1 加载ip_vs模块，安装ipcsadm工具

[root@BL ~]# modprobe ip_vs

[root@BL ~]# cat /proc/net/ip_vs

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port Forward Weight ActiveConn InActConn

[root@BL ~]# ls

anaconda-ks.cfg      install.log.syslog             公共的  图片  音乐

httpd-2.2.17.tar.gz  ipvsadm-1.27-7.el7.x86_64.rpm  模板    文档  桌面

install.log          nginx-1.6.0.tar.gz             视频    下载

[root@BL ~]# yum -y install ipvsadm

[root@BL ~]# rpm -q ipvsadm

ipvsadm-1.26-2.el6.x86_64

1.3.2配置负载调度器SNAT转发规则

【ipvsadm是LVS的管理工具，LVS的能力由IVPS模块实现，ipvsadm向用户提供一个命令接口，用于将配置的虚拟服务、真是服务等传给IPVS模块】

【-r 指定真实服务器】【-m 指定LVS的工作模式为NAT模式】【-a 添加真实服务器】

【-t 指定VIP地址及TCP端口】【-n 数字格式显示主机地址和端口】 【-L 列表查看】

【-s  指定使用的调用算法】【rr表示轮询】

[root@BL ~]# /etc/init.d/ipvsadm stop

ipvsadm: Clearing the current IPVS table:                  [确定]

ipvsadm: Unloading modules:                                [确定]

[root@BL ~]# ipvsadm -A -t 172.168.23.10:80 -s rr 【添加172.168.23.10为虚拟服务器，并指定VIP地址及端口号，指定调用算法为轮询】

[root@BL ~]# ipvsadm -a -t 172.168.23.10:80 -r 192.168.23.131 -m -w 1 【添加192.168.23.131为真实服务器，指定VIP地址为172.168.23.10 端口为80 ，工作模式为NAT模式】

[root@BL ~]# ipvsadm -a -t 172.168.23.10:80 -r 192.168.23.129 -m -w 1

[root@BL ~]# ipvsadm -L -n  【列表查看以数字形式显示主机地址和端口】

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  172.168.23.10:80 rr

  -> 192.168.23.129:80            Masq    1      0          0        

  -> 192.168.23.131:80            Masq    1      0          0 

[root@BL ~]# /etc/init.d/ipvsadm save  【保存配置单，】

ipvsadm: Saving IPVS table to /etc/sysconfig/ipvsadm:      [确定]

[root@BL ~]# chkconfig ipvsadm on  【开机自己启动】

1.3.3开启路由转发功能

[root@BL ~]# cp -p /etc/sysctl.conf  /etc/sysctl.conf.origin

[root@BL ~]# sed -i '7 s/0/1/' /etc/sysctl.conf

[root@BL ~]# sed -n '7p'  /etc/sysctl.conf

net.ipv4.ip_forward = 1

[root@BL ~]#  service iptables stop

iptables：将链设置为政策 ACCEPT：filter                    [确定]

iptables：清除防火墙规则：                                 [确定]

iptables：正在卸载模块：                                   [确定]

[root@BL ~]# setenforce 0

[root@web ~]# service iptables stop

iptables：将链设置为政策 ACCEPT：filter                    [确定]

iptables：清除防火墙规则：                                 [确定]

iptables：正在卸载模块：                                   [确定]

[root@web ~]# setenforce 0

[root@web2 ~]# service iptables stop

iptables：将链设置为政策 ACCEPT：filter                    [确定]

iptables：清除防火墙规则：                                 [确定]

iptables：正在卸载模块：                                   [确定]

[root@web2 ~]# setenforce 0

【web和web2测试页的路径，用源代码安装的和yum安装的测试页的路径不一样】

[root@web2 ~]# cat  /usr/local/apache/htdocs/index.html 【测试页的文件路径】

[root@BL ~]# curl 192.168.23.131

It works!

[root@BL ~]#

[root@BL ~]# curl 192.168.23.129

It works!

[root@BL ~]#

[root@BL ~]# curl 172.16.23.10

It works!

[root@BL ~]#

测试

 

 

[root@BL ~]# curl 172.16.23.10

web1

[root@BL ~]# curl 172.16.23.10

web2

[root@BL ~]# curl 172.16.23.10

web1

[root@BL ~]# curl 172.16.23.10

web2

 

 

二、LVS-DR案例

实验环境【BL只有一个ip】：

BL主机（负载调度器）

VIP：192.168.23.66/32 【虚拟的ip】

BL（负载调度器）：192.168.23.10/24

Web 主机：192.168.23.131/24

Web2主机：192.168.23.129/24

【此处192.168.23.0/24网段模拟公网ip】

修改vip子网掩码，网关

[root@BL ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

HWADDR=00:0c:29:ac:ff:92

TYPE=Ethernet

UUID=b7ea0783-e15f-4f78-84ff-017f1c97781d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=192.168.23.10

NETMASK=255.255.255.0

GATEWAY=192.168.23.1

 

[root@web ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

HWADDR=00:0C:29:99:0E:7A

TYPE=Ethernet

UUID=b7ea0783-e15f-4f78-84ff-017f1c97781d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=192.168.23.131

GAETWAY=192.168.23.10

NETMASK=255.255.255.0

 

[root@web2 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2

HWADDR=00:0c:29:b2:da:2b

TYPE=Ethernet

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=none

IPADDR=192.168.23.129

GATEWAY=192.168.23.10

NETMASK=255.255.255.0

2.1BL设置

[root@BL ~]# rpm -q ipvsadm

ipvsadm-1.26-2.el6.x86_64

[root@BL ~]# ipvsadm -C  【删除所有配置条目】

[root@BL ~]# ipvsadm -ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

[root@BL ~]# vim /opt/vip.sh

#!/bin/bash

#VIP

VIP="192.168.23.66"

/sbin/ifconfig eth0:vip $VIP broadcast $VIP netmask 255.255.255.255

/sbin/route add -host $VIP dev eth0:vip

[root@BL ~]# /opt/vip.sh

[root@BL ~]# ip a show eth0

2: eth0: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:ac:ff:92 brd ff:ff:ff:ff:ff:ff

    inet 192.168.23.10/24 brd 192.168.23.255 scope global eth0

    inet 192.168.23.66/32 brd 192.168.23.66 scope global eth0:vip

    inet6 fe80::20c:29ff:feac:ff92/64 scope link

       valid_lft forever preferred_lft forever

[root@BL ~]# route -n  【查看路由表】

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

192.168.23.66   0.0.0.0         255.255.255.255 UH    0      0        0 eth0

192.168.23.0    0.0.0.0         255.255.255.0   U     1      0        0 eth0

0.0.0.0         192.168.23.1    0.0.0.0         UG    0      0        0 eth0

 [root@BL ~]# echo "/opt/vip.sh" >>/etc/rc.local  【保存开机自启】

[root@BL etc]# ipvsadm -A -t 192.168.23.66:80 -s rr

[root@BL etc]# ipvsadm -a -t 192.168.23.66:80 -r 192.168.23.129:80 -g

[root@BL etc]# ipvsadm -a -t 192.168.23.66:80 -r 192.168.23.131:80 -g

[root@BL etc]# ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  192.168.23.66:80 rr

  -> 192.168.23.129:80            Route   1      0          0        

  -> 192.168.23.131:80            Route   1      0          0    

  -> 192.168.23.131:80            Route   1      0          0

[root@BL ~]# ipvsadm-save >/etc/sysconfig/ipvsadm  【保存配置】

 

2.2web服务器配置

[root@web ~]# vim /opt/lvs-dr

#!/bin/bash

#lvs-dr

VIP="192.168.23.66"

/sbin/ifconfig lo:vip $VIP broadcast $VIP netmask 255.255.255.255

/sbin/route add -host $VIP dev lo:vip

echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore

echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce

echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore

echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

[root@web ~]# chmod +x /opt/lvs-dr

[root@web ~]# /opt/lvs-dr

[root@web ~]# echo "/opt/lvs-dr" >>/etc/rc.local 【保存路径到配置单中】

[root@web ~]# ip a

1: lo: mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet 192.168.23.66/32 brd 192.168.23.66 scope global lo:vip

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth2: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:b1:a6:35 brd ff:ff:ff:ff:ff:ff

    inet 192.168.23.131/24 brd 192.168.23.255 scope global eth2

    inet6 fe80::20c:29ff:feb1:a635/64 scope link

       valid_lft forever preferred_lft forever

[root@web ~]# netstat -rn 【查看路由表】

Kernel IP routing table

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface

192.168.23.66   0.0.0.0         255.255.255.255 UH        0 0          0 lo

192.168.23.0    0.0.0.0         255.255.255.0   U         0 0          0 eth2

[root@web ~]# scp /opt/lvs-dr 192.168.23.129:/opt

The authenticity of host '192.168.23.129 (192.168.23.129)' can't be established.

RSA key fingerprint is 0a:27:71:e8:77:61:ca:38:a7:06:49:34:5b:89:4f:22.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added '192.168.23.129' (RSA) to the list of known hosts.

[email protected]'s password: 123123

lvs-dr                                      100%  338     0.3KB/s   00:00 

2.3 web2服务配置

[root@web2 ~]# ls /opt/

lvs-dr  rh

[root@web2 ~]# /opt/lvs-dr

[root@web2 ~]# ip a

1: lo: mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet 192.168.23.66/32 brd 192.168.23.66 scope global lo:vip

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth2: mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:b2:da:2b brd ff:ff:ff:ff:ff:ff

    inet 192.168.23.129/24 brd 192.168.23.255 scope global eth2

    inet6 fe80::20c:29ff:feb2:da2b/64 scope link

       valid_lft forever preferred_lft forever

[root@web2 ~]# route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

192.168.23.66   0.0.0.0         255.255.255.255 UH    0      0        0 lo

192.168.23.0    0.0.0.0         255.255.255.0   U     1      0        0 eth2

0.0.0.0         192.168.23.10   0.0.0.0         UG    0      0        0 eth2

[root@web2 ~]# echo "/opt/lvs-dr" >>/etc/rc.local

 

【iface该设定使用的网络接口】【destination目的地】

实验补充：

1、因复制后的虚拟机设备名不是eth0，实验中需注意

2.一般出错是ip配置文件

3、最后测试时在不同网页，每个主机试试，可能网段冲突，多试试