Nginx总结(三)--- Nginx+LVS+Keepalived 实践 (附:LVS没有轮询等问题)

一、应用场景
KEEPALIVED:检测节点是否健康及故障自动切换
LVS:四层软负载(越底层、效率越高)有三种IP负载均衡技术(VS/NAT、VS/TUN和VS/DR),十种调度算法(rrr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq)
NGINX:七层HTTP负载
本次实践中的负载架构图:
Nginx总结(三)--- Nginx+LVS+Keepalived 实践 (附:LVS没有轮询等问题)_第1张图片
实践中的坑与问题:见三、实践注意点与思考

二、实践过程
  0.环境

所用系统:CentOS 7.5
   NG01服务器:192.168.31.121
   NG02服务器:192.168.31.131
   LVS-Master负载均衡服务器:192.168.31.141
   LVS-Backup负载均衡服务器:172.16.122.132
   VIP 虚拟IP 192.168.31.100

  2.LVS-Master安装LVS的管理工具ipvsadm

yum -y install ipvsadm
[ipvsadm -C 为清除LVS设置命令]

  3.LVS-Master安装keepalived

yum -y install keepalived

  4.配置LVS-Master的keepalived -> /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
  router_id LVS_01
}

vrrp_instance VI_1 { # vrrp 虚拟路由冗余协议
  state MASTER
  interface ens33
  virtual_router_id 51
  priority 100
  advert_int 1
  authentication {
	auth_type PASS
	auth_pass 1111
  }
  virtual_ipaddress {
	192.168.31.100
  }
}

virtual_server 192.168.31.100 80 {
  delay_loop 6
  lb_algo rr
  lb_kind DR
  persistence_timeout 0
  protocol TCP

  real_server 192.168.31.121 80 {
	weight 1
	TCP_CHECK {
	connect_port 80    
  delay_before_retry 3
  nb_get_retry 3
  connect_timeout 3
	}
  }

  real_server 192.168.31.131 80 {
	weight 1
	TCP_CHECK {
	connect_port 80
  delay_before_retry 3
  nb_get_retry 3
  connect_timeout 3
	}
  }
}

  LVS-Backup负载均衡服务器配置同上1~4步,需要改变的是:router_id LVS_02、state BACKUP、priority 50
  5.LVS的DR模式下,两台NG服务器上给lo:0绑定VIP地址、ARP广播,脚本内容:

#!/bin/bash
#description: Config realserver
VIP=192.168.31.100

/etc/rc.d/init.d/functions

case "$1" in
start)
	 /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP
	 /sbin/route add -host $VIP dev lo:0
	 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
	 echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
	 echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
	 echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
	 sysctl -p >/dev/null 2>&1
	 echo "RealServer Start OK"
	 ;;
stop)
	 /sbin/ifconfig lo:0 down
	 /sbin/route del $VIP >/dev/null 2>&1
	 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
	 echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
	 echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
	 echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
	 echo "RealServer Stoped"
	 ;;
*)
	 echo "Usage: $0 {start|stop}"
	 exit 1
esac

exit 0

  保存运行脚本./rs.sh start(可能遇到权限问题,见三)
  6.在LVS服务器上Keepalived 相关操作命令

#启动Keepalived
systemctl start keepalived
#关闭Keepalived
systemctl start keepalived
#重启Keepalived
systemctl restart keepalived
#查看状态Keepalived
systemctl status keepalived

  7.测试:
    ① 通过 ipvsadm -L 命令可以查看VIP是否已经成功映射到两台real服务器,如果发现有问题,可以通过tail -f /var/log/message查看错误原因

[root@localhost keepalived]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.31.100:http rr
-> 192.168.31.121:http          Route   1      0          0         
-> 192.168.31.131:http          Route   1      0          0         

    ② 测试负载均衡(可能遇到没有轮询负载的问题,见三)
Nginx总结(三)--- Nginx+LVS+Keepalived 实践 (附:LVS没有轮询等问题)_第2张图片
Nginx总结(三)--- Nginx+LVS+Keepalived 实践 (附:LVS没有轮询等问题)_第3张图片 
三、实践注意点与思考
  0.先说大坑:为什么LVS设置了轮询,浏览器测试还是不能轮询?这关系到两个地方的配置:
    01./etc/keepalived/keepalived.conf的persistence_timeout会话保持时间配置,测试轮询时设置为0;
    02.查看ipvsadm默认超时时间(巨坑,导致我一直在浏览器刷不出LVS轮询,也是看到了参考文献3才知道的)
      [root@DR1 keepalived]# ipvsadm -L --timeout
      Timeout (tcp tcpfin udp): 900 120 300
      900 120 300这三个数值分别是TCP TCPFINUDP的时间.也就是说一条tcp的连接经过lvs后,lvs会把这台记录保存15分钟,就是因为这个时间过长,所以很多人都会发现做好LVS DR之后轮询现象并没有发生,实践中将此数值调整很小小,使用以下命令调整:
      [root@DR1 ~]# ipvsadm --set 1 2 1
    再次测试轮询效果,就可以了!而实际配置中还是按照默认配置,那么在大量IP访问VIP时,就有轮询效果?有待验证

  1.NG节点配置nginx.conf error-log日志级别为info,再reload一下,可以发现两台LVS的keepalived都在定是检测HG健康(注:记得将error-log级别改回来,不然日志很多,遗留问题:21:19:17 [info] 109687#0: *15087 recv() failed (104: Connection reset by peer) while waiting for request, client: 192.168.31.141, server: 0.0.0.0:18081 原因:这是正常的请求INFO日志,keepalive的tcp_cheack心跳检测,在nginx的配置中error.log输出级别为被设置为info,keepalived与MG只做了两次握手,所以NG一直包这个'错误')
  2.遇到的问题Received advert with lower priority 80, ours 100, forcing new election 原因:备机上防火墙开着(要关掉)
  3.还有一些思考在 四、参考文献的2中,推荐了解一下原理,讲的很好!
四、参考文献
  1.LVS+Keepalived+Nginx实现HA https://www.jianshu.com/p/88589646aae8 (keepalive的配置较详细,其中的 三、LVS负载均衡服务器配置 为LVS-NAT模式,理论上是正确的,但实践未果;六、DR模式不设置两个arp相关的参数貌似也可以)
  2.LVS | LVS 的三种工作方式(DR原理)(二)(推荐)https://blog.csdn.net/liupeifeng3514/article/details/79038451
  3.LVS-DR RR(轮询模式)搭建(ipvsadm默认超时时间帮助了我解决LVS不能轮询的问题,感谢)https://blog.csdn.net/Traumerei/article/details/63686693
  4.LVS+Keepalived+httpd安装及配置(比较全面细致的实践文档)https://blog.csdn.net/sz_bdqn/article/details/46705251

你可能感兴趣的:(Nginx)