keepalived + nginx 搭建负载均衡集群

第一章 keepalived

1.1 keepalived 服务说明

Keepalived软件起初是专为LVS负载均衡软件设计的,用来管理并监控LVS集群系统中各个服务节点的状态,后来又加入了可以实现高可用的VRRP功能。因此,Keepalived除了能够管理LVS软件外,还可以作为其他服务(例如:Nginx、Haproxy、MySQL等)的高可用解决方案软件。

  Keepalived软件主要是通过VRRP协议实现高可用功能的。VRRP是Virtual Router RedundancyProtocol(虚拟路由器冗余协议)的缩写,VRRP出现的目的就是为了解决静态路由单点故障问题的,它能够保证当个别节点宕机时,整个网络可以不间断地运行。

1.2 keepalived 服务三个重要功能

  管理LVS负载均衡软件

  实现LVS集群节点的健康检查中

作为系统网络服务的高可用性(failover)

* 在本次测试环境中通过keepalived 的网络高可用功能实现负载均衡集群

 

1.3 Keepalived  的工作原理

Keepalived的工作原理:

  Keepalived高可用对之间是通过VRRP通信的,因此,我们从 VRRP开始了解起:

    1) VRRP,全称 Virtual Router Redundancy Protocol,中文名为虚拟路由冗余协议,VRRP的出现是为了解决静态路由的单点故障。

    2) VRRP是通过一种竟选协议机制来将路由任务交给某台 VRRP路由器的。

    3) VRRP用 IP多播的方式(默认多播地址(224.0_0.18))实现高可用对之间通信。

    4) 工作时主节点发包,备节点接包,当备节点接收不到主节点发的数据包的时候,就启动接管程序接管主节点的开源。备节点可以有多个,通过优先级竞选,但一般 Keepalived系统运维工作中都是一对。

    5) VRRP使用了加密协议加密数据,但Keepalived官方目前还是推荐用明文的方式配置认证类型和密码。

  介绍完 VRRP,接下来我再介绍一下 Keepalived服务的工作原理:

  Keepalived高可用对之间是通过 VRRP进行通信的, VRRP是遑过竞选机制来确定主备的,主的优先级高于备,因此,工作时主会优先获得所有的资源,备节点处于等待状态,当主挂了的时候,备节点就会接管主节点的资源,然后顶替主节点对外提供服务。

  在 Keepalived服务对之间,只有作为主的服务器会一直发送 VRRP广播包,告诉备它还活着,此时备不会枪占主,当主不可用时,即备监听不到主发送的广播包时,就会启动相关服务接管资源,保证业务的连续性.接管速度最快可以小于1秒。

 

第二章 Nginx

2.1  nginx  服务说明

  Nginx是一款自由的、开源的、高性能的HTTP服务器和反向代理服务器;同时也是一个IMAP、POP3、SMTP代理服务器;Nginx可以作为一个HTTP服务器进行网站的发布处理,另外nginx可以作为反向代理进行负载均衡的实现。 在本次测试中使用upstream 和proxy_pass 实现反向代理和负载均衡。

第三章 功能测试

 

3.1 测试环境

安装编译工具及库文件:

# yum -y install make zlib zlib-devel gcc-c++ libtool  openssl openssl-devel

gcc:安装 nginx 需要先将官网下载的源码进行编译,编译依赖 gcc 环境。

zlib:zlib 库提供了很多种压缩和解压缩的方式,nginx 使用 zlib 对 http 包的内容进行 gzip。

openssl:OpenSSL 是一个强大的安全套接字层密码库,囊括主要的密码算法、常用的密钥和证书封装管理功能及SSL协议,并提供丰富的应用程序供测试或其它目的使用。nginx 不仅支持http 协议,还支持 https(即在 ssl 协议上传输 http),所以需要在 linux 安装 openssl 库。

系统版本

# cat /etc/centos-release

CentOS Linux release 7.5.1804 (Core)

内核版本

# uname -a

Linux localhost.localdomain 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Keepalived 版本

# keepalived -v

Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Nginx 版本

#./nginx -v

nginx version: nginx/1.14.1

部署环境

ESXI  5.5

3.2 部署架构

keepalived + nginx 搭建负载均衡集群_第1张图片

3.3  配置说明

三台设备接口地址和虚拟服务对应地址如下:

server_1 :

接口地址: 10.2.61.21

虚拟服务地址: 10.2.61.24

 

server_2:

接口地址:10.2.61.22

虚拟服务地址:10.2.61.25

 

server_3:

接口地址:10.2.61.23

虚拟服务地址: 10.2.61.26

 

3.3.1 server_1 配置

Keepalived 配置文件:

 

! Configuration File for keepalived

 

global_defs {

   notification_email {

     [email protected]

     [email protected]

     [email protected]

   }

   notification_email_from [email protected]

   smtp_server 192.168.200.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

   vrrp_skip_check_adv_addr

   #vrrp_strict

   vrrp_garp_interval 0

   vrrp_gna_interval 0

   router_id lb01

 

}

 

vrrp_script nginx_check {   # nginx 服务监测脚本

   script "/usr/local/nginx/conf/nginx_check.sh"

   interval 4 #脚本执行间隔时间

   weight   -25 #脚本触发消减的权重值

#   timeout  3

 

}

 

vrrp_instance VI_1 { #集群1

    state MASTER #当前设备为主机

    interface ens192

    virtual_router_id 21 #路由标识符

    priority 100 #优先级100

    preempt_delay 300   #抢占式模式下,节点上线后触发新选举操作的延迟时长;

  advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322 #明文认证密码

    }

    virtual_ipaddress { #集群绑定的虚拟服务地址

       10.2.61.24

    }

    track_script { #绑定的执行脚本

    nginx_check

    

    }

}

 

vrrp_instance VI_2 { #集群2

    state BACKUP #当前主机为备机

    interface ens192

    virtual_router_id 22 #同步组2 标识符

    priority 80 #当前主机在集群中的优先级

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322 #明文认证密码

    }

    virtual_ipaddress { #集群绑定虚拟服务地址

       10.2.61.25

    }

}

 

vrrp_instance VI_3 { #集群3

    state BACKUP #当前主机在集群中为备机

    interface ens192

    virtual_router_id 23 #集群标识符

    priority 60 #当前主机在集群中优先级

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322 #明文认证密码

    }

    virtual_ipaddress { #集群绑定虚拟服务地址

       10.2.61.26

    }

}

 

Nginx 服务配置文件:

 

#user  nobody;

worker_processes  1;

 

#error_log  logs/error.log;

#error_log  logs/error.log  notice;

#error_log  logs/error.log  info;

 

#pid        logs/nginx.pid;

 

 

events {

    worker_connections  1024;

}

 

 

http {

    include       mime.types;

    default_type  application/octet-stream;

 

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

    #                  '$status $body_bytes_sent "$http_referer" '

    #                  '"$http_user_agent" "$http_x_forwarded_for"';

 

    #access_log  logs/access.log  main;

 

    sendfile        on;

    #tcp_nopush     on;

 

    #keepalive_timeout  0;

    keepalive_timeout  65;

 

    #gzip  on;

    upstream lb_hb { #定义负载均衡池

 

server 202.110.184.34:80  ;

server 111.161.3.158:80   backup;

server 202.120.127.248:80 backup;

 

    }  

    upstream lb_nx { #定义负载均衡池

server 10.2.61.28:80;

 

    }

    upstream lb_jx { #定义负载均衡池

server 10.2.61.28:8088;

 

    }

    server { #定义虚拟服务,通过proxy_pass 模块绑定负载均衡池

        listen      8088;

        server_name www.hb.sgcc.com.cn;

        location / {

proxy_pass http://lb_hb;

proxy_set_header Host       $proxy_host;

}

    }

 server { #定义虚拟服务,通过proxy_pass 模块绑定负载均衡池

listen     8088;

server_name   www.nx.sgcc.com.cn;

location / {

proxy_pass http://lb_nx;

proxy_set_header Host $proxy_host;

}

   }

    

    server { #定义虚拟服务,通过proxy_pass 模块绑定负载均衡池

listen     8088;

server_name   www.jx.sgcc.com.cn;

location / {

proxy_pass http://lb_jx;

proxy_set_header Host $proxy_host;

}

   }

}

 

3.3.2 server_2 配置

Keepalived 服务配置文件:

 

! Configuration File for keepalived

 

global_defs {

   notification_email {

     [email protected]

     [email protected]

     [email protected]

   }

   notification_email_from [email protected]

   smtp_server 192.168.200.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

   vrrp_skip_check_adv_addr

#   vrrp_strict

   vrrp_garp_interval 0

   vrrp_gna_interval 0

   router_id lb02

}

vrrp_script nginx_check {

   script "/usr/local/nginx/conf/nginx_check.sh"

   interval 4

   weight   -25

   #timeout  3

 

        }

 

vrrp_instance VI_1 {

    state BACKUP

    interface ens192

    virtual_router_id 21

    priority 60

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322

    }

    virtual_ipaddress {

       10.2.61.24

    }

}

 

vrrp_instance VI_2 {

    state MASTER

    interface ens192

    virtual_router_id 22

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322

    }

    virtual_ipaddress {

       10.2.61.25

    }

    track_script {

 

    nginx_check

   }

}

 

vrrp_instance VI_3 {

    state BACKUP

    interface ens192

    virtual_router_id 23

    priority 80

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322

    }

    virtual_ipaddress {

       10.2.61.26

    }

}

 

Nginx 服务配置文件:

 

#user  nobody;

worker_processes  1;

 

#error_log  logs/error.log;

#error_log  logs/error.log  notice;

#error_log  logs/error.log  info;

 

#pid        logs/nginx.pid;

 

 

events {

    worker_connections  1024;

}

 

 

http {

    include       mime.types;

    default_type  application/octet-stream;

 

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

    #                  '$status $body_bytes_sent "$http_referer" '

    #                  '"$http_user_agent" "$http_x_forwarded_for"';

 

    #access_log  logs/access.log  main;

 

    sendfile        on;

    #tcp_nopush     on;

 

    #keepalive_timeout  0;

    keepalive_timeout  65;

 

    #gzip  on;

    upstream lb_hb {

 

server 202.110.184.34:80  ;

server 111.161.3.158:80   backup;

server 202.120.127.248:80 backup;

 

    }  

    upstream lb_nx {

server 10.2.61.28:80;

 

    }

    upstream lb_jx {

server 10.2.61.28:8088;

 

    }

    server {

        listen      8088;

        server_name www.hb.sgcc.com.cn;

        location / {

proxy_pass http://lb_hb;

proxy_set_header Host       $proxy_host;

}

    }

    server {

listen     8088;

server_name   www.nx.sgcc.com.cn;

location / {

proxy_pass http://lb_nx;

proxy_set_header Host $proxy_host;

}

   }

    

    server {

listen    8088;

server_name   www.jx12.sgcc.com.cn;

location / {

proxy_pass http://lb_jx;

proxy_set_header Host $proxy_host;

}

   }

}

 

 

 

3.3.3 server_3 配置

Keepalived 配置文件:

! Configuration File for keepalived

 

global_defs {

   notification_email {

     [email protected]

     [email protected]

     [email protected]

   }

   notification_email_from [email protected]

   smtp_server 192.168.200.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

   vrrp_skip_check_adv_addr

#   vrrp_strict

   vrrp_garp_interval 0

   vrrp_gna_interval 0

   router_id lb02

}

vrrp_script nginx_check {

   script "/usr/local/nginx/conf/nginx_check.sh"

   interval 4

   weight   -25

   #timeout  3

 

        }

 

vrrp_instance VI_1 {

    state BACKUP

    interface ens192

    virtual_router_id 21

    priority 60

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322

    }

    virtual_ipaddress {

       10.2.61.24

    }

}

 

vrrp_instance VI_2 {

    state MASTER

    interface ens192

    virtual_router_id 22

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322

    }

    virtual_ipaddress {

       10.2.61.25

    }

    track_script {

 

    nginx_check

   }

}

 

vrrp_instance VI_3 {

    state BACKUP

    interface ens192

    virtual_router_id 23

    priority 80

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1322

    }

    virtual_ipaddress {

       10.2.61.26

    }

}

Nginx 配置文件:

 

 

#user  nobody;

worker_processes  1;

 

#error_log  logs/error.log;

#error_log  logs/error.log  notice;

#error_log  logs/error.log  info;

 

#pid        logs/nginx.pid;

 

 

events {

    worker_connections  1024;

}

 

 

http {

    include       mime.types;

    default_type  application/octet-stream;

 

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

    #                  '$status $body_bytes_sent "$http_referer" '

    #                  '"$http_user_agent" "$http_x_forwarded_for"';

 

    #access_log  logs/access.log  main;

 

    sendfile        on;

    #tcp_nopush     on;

 

    #keepalive_timeout  0;

    keepalive_timeout  65;

 

    #gzip  on;

    upstream lb_hb {

 

server 202.110.184.34:80  ;

server 111.161.3.158:80   backup;

server 202.120.127.248:80 backup;

 

    }  

    upstream lb_nx {

server 10.2.61.28:80;

 

    }

    upstream lb_jx {

server 10.2.61.28:8088;

 

    }

    server {

        listen      8088;

        server_name www.hb.sgcc.com.cn;

        location / {

proxy_pass http://lb_hb;

proxy_set_header Host       $proxy_host;

}

    }

    server {

listen     8088;

server_name   www.nx.sgcc.com.cn;

location / {

proxy_pass http://lb_nx;

proxy_set_header Host $proxy_host;

}

   }

    

    server {

listen    8088;

server_name   www.jx12.sgcc.com.cn;

location / {

proxy_pass http://lb_jx;

proxy_set_header Host $proxy_host;

}

   }

}

nginx 服务监测脚本:


 

#!/bin/bash

# keepalived script check nginx services status


#while true

#do

nginx_session=$(netstat -ntlp|grep nginx |wc -l)
echo "$nginx_session"
if [ $nginx_session -lt 1 ]
then

/usr/bin/systemctl stop keepalived

# exit
fi

#sleep 3

#done


 

3.4 模拟Server_1 故障设备故障测试:

Server_1 log :

Dec 11 10:45:15 localhost systemd-logind: Removed session 4.

Dec 11 10:45:16 localhost Keepalived[19875]: Stopping

Dec 11 10:45:16 localhost systemd: Stopping LVS and VRRP High Availability Monitor...

Dec 11 10:45:16 localhost Keepalived_vrrp[19877]: VRRP_Instance(VI_1) sent 0 priority

Dec 11 10:45:16 localhost Keepalived_vrrp[19877]: VRRP_Instance(VI_1) removing protocol VIPs.

Dec 11 10:45:16 localhost Keepalived_healthcheckers[19876]: Stopped

Dec 11 10:45:17 localhost Keepalived_vrrp[19877]: Stopped

Dec 11 10:45:17 localhost Keepalived[19875]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Dec 11 10:45:17 localhost systemd: Stopped LVS and VRRP High Availability Monitor.

 

Server_3 log : server_3 集群1 优先级 80 server_1 故障时承载业务

Dec 11 10:45:17 localhost Keepalived_vrrp[11307]: VRRP_Instance(VI_1) Transition to MASTER STATE

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: VRRP_Instance(VI_1) Entering MASTER STATE

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: VRRP_Instance(VI_1) setting protocol VIPs.

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens192 for 10.2.61.24

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:18 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:23 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:23 localhost Keepalived_vrrp[11307]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens192 for 10.2.61.24

Dec 11 10:45:23 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:23 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:23 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

Dec 11 10:45:23 localhost Keepalived_vrrp[11307]: Sending gratuitous ARP on ens192 for 10.2.61.24

 

 

 

数据包分析:

 

Server_3作为集群3 主负载节点发送组播数据包

 keepalived + nginx 搭建负载均衡集群_第2张图片

 

Server_3作为集群1 备份负载节点发送组播数据包

 keepalived + nginx 搭建负载均衡集群_第3张图片

 

 第四章 总结:

  本次测试主要为了验证keepalived+ nginx 模式下构建 主 + 主 + 主 业务集群。三个集群按照优先级顺序分别为指定的三个业务的master 设备和backup 设备。在测试中有几个点。

  1.keepalived 配置分别在三台设备事先规划的master 设备上配置nginx_check.sh 脚本 # 注意不要在一个里面都进行配置,keepalived 会加载所有配置导致优先级混乱。

  2.在本次配置中没有配置   nopreempt ,在主备模式下配置抢占 。本次测试为延时抢占

  3.track_script { #绑定的执行脚     nginx_check }   必须给定执行脚本,否则   vrrp_script nginx_check  给定脚本不执行。

  4.nginx 配置配置中 定义 server 时只定义 listen 端口与 server_name   不指定监听地址,因为nginx 加载配置文件时会监测当前网卡是否监测此地址

  

  本次测试主要验证 keepalived + nginx 的主 + 主 + 主模式,后续会分别对模块进行梳理。

 

转载于:https://www.cnblogs.com/zy09/p/10103186.html

你可能感兴趣的:(keepalived + nginx 搭建负载均衡集群)