Nginx+Keepalived+rsync部署手册

1、Nginx编译部署

下载解压安装包
wget http://nginx.org/download/nginx-1.16.1.tar.gz
tar zxf /opt/nginx-1.16.1.tar.gz -C /opt/
cd nginx-1.16.1

添加用户
useradd -M -s /sbin/nologin nginx

编译安装
./configure --user=nginx --group=nginx --prefix=/data/nginx --with-http_stub_status_module --with-http_ssl_module --with-stream --with-pcre
make && make install

解决依赖关系
错误提示：./configure: error: C compiler cc is not found
#yum -y install gcc gcc-c++ autoconf automake make
错误提示：./configure: error: the HTTP cache module requires md5 functions from OpenSSL library.
#yum -y install openssl openssl-devel
错误提示：./configure: error: the HTTP rewrite module requires the PCRE library.
#yum -y install pcre-devel

初始化配置
ln -s /data/nginx/sbin/* /usr/local/sbin/
chown -R nginx:nginx /data/nginx/
chmod +x /etc/rc.d/rc.local
vim /etc/rc.d/rc.local
添加 /usr/local/sbin/nginx -c /data/nginx/conf/nginx.conf

2、Nginx主备配置
注：使用keepalived抢占模式，Nginx-1节点为更改nginx配置节点。
yum install -y keepalived
cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.init
vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   router_id lbh_nginx
   script_user root
   enable_script_security
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_nginx.sh"
    interval 2
}
vrrp_instance VI_1 {
    state MASTER			##两个节点都必须是BACKUP状态
    #nopreempt				##非抢占模式
    interface  eth0
    virtual_router_id 99
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.130.1.244
    }
    track_script  {
    chk_nginx
    }
}

router_id nginx_master
script_user root ##解决WARNING告警

nginx-2 把priority 150 替换成 priority 100
把state MASTER 改成BACKUP

vim /etc/keepalived/chk_nginx.sh

#!/bin/bash
counter=`ps -C nginx --no-header |wc -l`
if [ $counter = "0" ];then
        echo "Nginx erver is died."
        echo "Starting..."
	/usr/local/sbin/nginx -c /data/nginx/conf/nginx.conf
        sleep 2
        counter=`ps -C nginx --no-header |wc -l`
        if [ $counter = "0" ];then
                systemctl stop keepalived
        fi
fi

chmod +x /etc/keepalived/chk_nginx.sh

若Nginx宿主机为Openstack云主机：
添加安全组策略
其他协议 --> 112 --> CIDR 0.0.0.0/0
（否则BACKUP获取不到来自224.0.0.18组播通告地址的MASTER信息，会导致脑裂，BACKUP总是会争抢MASTER角色。）

3、Rsync+Inotify同步信息
注：不是主主同步，更改nginx-2配置不会自动同步至nginx-1。

cd /opt
wget https://www.samba.org/ftp/rsync/src/rsync-3.1.3.tar.gz
wget https://github.s3.amazonaws.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

Nginx-1节点

安装rsync
tar axf /opt/rsync-3.1.3.tar.gz -C /opt/
cd /opt/rsync-3.1.3
./configure --prefix=/data/rsync
make && make install
ln -sf /data/rsync/bin/rsync /usr/bin/rsync

建立密码认证文件
echo “Lbh#nginx123” > /data/rsync/rsync.passwd
chmod 600 /data/rsync/rsync.passwd

安装inotify
tar axf /opt/inotify-tools-3.14.tar.gz -C /opt/
cd /opt/inotify-tools-3.14
./configure --prefix=/data/inotify
make && make install
ln -sf /data/inotify/bin/inotifywait /usr/bin/inotifywait

两节点间免密登录

配置同步脚本
vim /data/rsync/rsync.sh

#!/bin/bash
src=/data/nginx/conf
des=data
rsync_passwd_file=/data/rsync/rsync.passwd
ip1=10.130.1.61
user=root
cd ${src}                            
/usr/bin/inotifywait -mrq --format  '%Xe %w%f' -e modify,create,delete,attrib,close_write,move ./ | while read file
do
        INO_EVENT=$(echo $file | awk '{print $1}')      
        INO_FILE=$(echo $file | awk '{print $2}')       
        echo "-------------------------------$(date)------------------------------------"
        echo $file
        if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]]         
        then
                echo 'CREATE or MODIFY or CLOSE_WRITE or MOVED_TO'
                rsync -avzcR --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
        fi
       
        if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]]
        then
                echo 'DELETE or MOVED_FROM'
                rsync -avzR --delete --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
        fi
        
        if [[ $INO_EVENT =~ 'ATTRIB' ]]
        then
                echo 'ATTRIB'
                if [ ! -d "$INO_FILE" ]
                then
                        rsync -avzcR --password-file=${rsync_passwd_file} $(dirname ${INO_FILE}) ${user}@${ip1}::${des}
                      
                fi
        fi
done

后台执行同步脚本
nohup sh /data/rsync/rsync.sh &
echo “nohup sh /data/rsync/rsync.sh &” >> /etc/rc.local
chmod +x /etc/rc.d/rc.local
查看后台日志
tail /data/rsync/nohup.out

Nginx-2节点

安装rsync（同nginx-1节点）

建立密码认证文件
echo “root:nginx!@#123” > /data/rsync/rsync.passwd
在nginx-1端建立的密码文件，只有密码，没有用户名；而在同步服务端里建立的密码文件，用户名与密码都有。
chmod 600 /data/rsync/rsync.passwd

建立rsync配置文件

uid = root
gid = root
use chroot = no
max connections = 10
strict modes = yes
pid file = /data/rsync/run/rsyncd.pid
lock file = /data/rsync/run/rsync.lock
log file = /data/rsync/logs/rsyncd.log
[data]									###与server端脚本上的des对应
path = /data/nginx/conf
comment = web file
read only = no
write only = no
hosts allow = 10.130.1.56				####server端ip 主nginx
hosts deny = *
list = false
uid = root
gid = root
auth users = root
secrets file = /data/rsync/rsync.passwd

启动rsync
/data/rsync/bin/rsync --daemon --config=/data/rsync/rsync.conf
echo “/data/rsync/bin/rsync --daemon --config=/data/rsync/rsync.conf” >> /etc/rc.local

测试配置同步功能
Nginx-1节点
mkdir /data/nginx/conf/conf.d

Nginx-2节点
ll /data/nginx/conf/
看到/data/nginx/conf/conf.d 目录已创建
tailf /data/rsync/logs/rsyncd.log