nginx+keepalived实现emq集群负载均衡高可用（七）

前言：压测emq集群之前需要有负载均衡服务，做负载均衡的一般常用nginx，也可以用haproxy，我这里用nginx。

---

虽然一个nginx就可以实现负载均衡的功能，但nginx作为一个入口，如果挂了后续服务都将受到影响，nginx+keeplived能实现nginx的高可用。目前nginx+keeplived有两种搭建模式：1，主从热备 2，双机热备。我用第二种，更多利用服务器资源。

部署机器 128,135.

先安装nginx：

#准备依赖包：
cd /opt

yum -y install gcc pcre-devel zlib-devel openssl-devel openssl

wget http://nginx.org/download/nginx-1.15.12.tar.gz

安装Nginx，编译过程中开启 --with-stream,tcp转发必须模块，因为emqx转发的是tcp连接

tar -zxvf nginx-1.15.12.tar.gz

cd nginx-1.15.12

./configure --with-http_stub_status_module --with-http_ssl_module --with-stream --prefix=/opt/nginx

make && make install

编辑配置文件

vim /opt/nginx/conf/nginx.conf

#在文件最后加上自己定义tcp转发配置文件
include emqx.conf;

新建tcp转发配置文件，用18084代理1883端口

vim /opt/nginx/conf/emqx.conf

stream
{
upstream emqx_cluster {
        server 192.168.1.128:1883;
        server 192.168.1.135:1883;
		server 192.168.1.136:1883;
   }

 server{
		listen  18084 so_keepalive=on;
       proxy_connect_timeout 10s;
       proxy_timeout 20s;
       proxy_pass emqx_cluster;
  }
}

检查配置文件是否ok

./sbin/nginx -t

nginx: the configuration file /opt/software/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /opt/software/nginx/conf/nginx.conf test is successful

#出现上面两句说明配置ok，可以正常启动

启动nginx

./sbin/nginx

查看是否启动成功

ps -ef | grep nginx


[root@iZwz97pma26sz5rguhiof1Z nginx]# ps -ef | grep nginx
root     19440     1  0 10:42 ?        00:00:00 nginx: master process ./sbin/nginx
nobody   19441 19440  0 10:42 ?        00:00:00 nginx: worker process
nobody   19442 19440  0 10:42 ?        00:00:00 nginx: worker process
nobody   19443 19440  0 10:42 ?        00:00:00 nginx: worker process
root     19452 10808  0 10:42 pts/1    00:00:00 grep --color=auto nginx

#启动成功

 

---

再安装keepalived：

方法一：yum安装（一般版本较旧）

#安装依赖
yum install -y curl gcc openssl-devel libnl3-devel net-snmp-devel

#安装
yum install keepalived -y

#查看版本
keepalived -v

#初始化及启动

#启动keepalived
systemctl start keepalived 

#加入开机启动keepalived  
systemctl enable keepalived

#重新启动keepalived  
systemctl restart keepalived

#查看keepalived状态  
systemctl status keepalived  

方法二：源码包安装，在keepalived官网下载最新包

[root@master src]# pwd
/usr/local/src
[root@master src]# wget http://www.keepalived.org/software/keepalived-2.0.7.tar.gz
[root@master src]# tar xvf keepalived-2.0.7.tar.gz
[root@master src]# cd keepalived-2.0.7
[root@master keepalived-2.0.7]# ./configure --prefix=/usr/local/keepalived
[root@master keepalived-2.0.7]# make && make install


# keepalived启动脚本变量引用文件，默认文件路径是/etc/sysconfig/，也可以不做软链接，直接修改启动脚本中文件路径即可（安装目录下）
[root@localhost /]# cp /usr/local/keepalived/etc/sysconfig/keepalived  /etc/sysconfig/keepalived 
 
# 将keepalived主程序加入到环境变量（安装目录下）
[root@localhost /]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
 
# keepalived启动脚本（源码目录下），放到/etc/init.d/目录下就可以使用service命令便捷调用
[root@localhost /]# cp /usr/local/src/keepalived-2.0.7/keepalived/etc/init.d/keepalived  /etc/init.d/keepalived
 
# 将配置文件放到默认路径下
[root@localhost /]# mkdir /etc/keepalived
[root@localhost /]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf

加为系统服务：chkconfig –add keepalived
开机启动：chkconfig keepalived on
查看开机启动的服务：chkconfig –list
启动、关闭、重启service keepalived start|stop|restart

在128和135的/etc/keepalived下新建nginx_check.sh脚本（该脚本是检查nginx服务时候挂了，挂了keepalived会重启nginx）

#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
echo "$counter"
if [ "${counter}" = "0" ]; then
    /opt/nginx/sbin/nginx
    sleep 2
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0" ]; then
        service keepalived stop
    fi
fi

#该脚本的的意思是
查看nginx进程数，若为0，重启nginx，在此查看nginx进程数，若还为0（启动失败），则杀死keepalived服务

给nginx_check.sh脚本可执行权限

chmod 755 nginx_check.sh

128的keepalived.conf：

! Configuration File for keepalived

global_defs {
    
}

vrrp_script check_nginx {
    script "/etc/keepalived/nginx_check.sh" 
    interval 2 
    weight -20 
}

vrrp_instance VI_1 {
    state MASTER 
    interface ens33 
    virtual_router_id 100  
    priority 100 
    advert_int 1 
    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {
        192.168.1.178 
    }


    track_script {
        check_nginx 
    }
}

vrrp_instance VI_2 {
    state BACKUP 
    interface ens33 
    virtual_router_id 101  
    priority 90  
    advert_int 1 
    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {
        192.168.1.249 
    }
track_script {
        check_nginx 
    }
}

135的keepalived.conf：

! Configuration File for keepalived

global_defs {
    
}

vrrp_script check_nginx {
    script "/etc/keepalived/nginx_check.sh" 
    interval 2 
    weight -20 
}

vrrp_instance VI_1 {
    state BACKUP 
    interface ens33 
    virtual_router_id 100  
    priority 90 
    advert_int 1 
    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {
        192.168.1.178 
    }
track_script {
        check_nginx 
    }
}

vrrp_instance VI_2 {
    state MASTER
    interface ens33 
    virtual_router_id 101  
    priority 100  
    advert_int 1 
    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {
        192.168.1.249 
    }
track_script {
        check_nginx 
    }
}