实验背景

A文件服务器的/opt/dataA 和 B文件服务器的/opt/dataB目录实时同步

A正常时，A为主，B为备，B去实时同步A

A挂了时，B为主，待A起来后，A为备，去实时同步B，并且能补上挂掉期间B增加的数据，反之亦然

keepalived + rsync +inotify 实现真正的高效数据实时同步

https://fandenggui.com/post/keepalived-rsync-inotify.html

真正的inotify+rsync实时同步彻底告别同步慢

http://www.ttlsa.com/web/let-infotify-rsync-fast

一、实验环境

操作系统：CentOS7.2

serverA： 192.168.1.102

serverB： 192.168.1.103

vip： 192.168.1.120

二、软件安装

在 serverA和serverB上

# yum -y install epel-release

# yum -y install keepalived inotity-tools rsync httpd

# systemctl enable httpd keepalived

rsync-inotify.service 会由后面的Keepalived程序根据主备角色变化情况，自动的运行和关闭

设置selinux为宽松模式

# setenforce 0

# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config

关闭防火墙

# systemctl stop firewalld

# systemctl disable firewalld

主机互信

我们同步采用rsync的ssh模式，需要让serverA和 serverB 相互互信

三、主机配置

serverA

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第1张图片

# vim /opt/data_realtime_rsync.sh

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第2张图片

#################################################

#!/bin/bash

# File name:data_realtime_sync.sh

# File path: /opt/data_realtime_sync.sh

src_dir="/opt/dataA"

dest_dir="/opt/dataB"

src_ip="192.168.1.102"

dest_ip="192.168.1.103"

. /etc/init.d/functions

cd ${src_dir}

inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |

while read file;do

INO_EVENT=$(echo $file | awk '{print $1}')

INO_FILE=$(echo $file | awk '{print $2}')

if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then

for ip in ${dest_ip};do

rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}

done

if [[ $INO_EVENT =~ 'ATTRIB' ]];then

if [ ! -d "$INO_FILE" ];then

for ip in ${dest_ip};do

rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}

done

if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then

for ip in ${dest_ip};do

rsync -avzcR --delete -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}

done

##############################################

# vim /etc/keepalived/keepalived.conf

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第3张图片

###########################################

! Configuration File for keepalived

global_defs {

router_id LVS_DEVEL

}

vrrp_script check_httpd {

script "/etc/keepalived/check_httpd.sh"

interval 5

}

vrrp_instance VI_1 {

state BACKUP

interface eno16777736

virtual_router_id 100

priority 100

advert_int 1

nopreempt

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

check_httpd

}

virtual_ipaddress {

192.168.1.120

}

notify_master "/etc/keepalived/rsync-inotify.sh start"

notify_backup "/etc/keepalived/rsync-inotify.sh stop"

notify_stop "/etc/keepalived/rsync-inotify.sh send"

}

##########################################

# vim /etc/keepalived/check_httpd.sh

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第4张图片

##############################

#!/bin/sh

httpd_status1=$(ps -C httpd --no-heading|wc -l)

if [ "${httpd_status1}" = "0" ]; then

systemctl start httpd.service

sleep 5

httpd_status2=$(ps -C httpd --no-heading|wc -l)

if [ "${httpd_status2}" = "0" ]; then

systemctl stop keepalived.service

#############################

# vim /etc/keepalived/rsync-inotify.sh

########################

#!/bin/bash

case "$1" in

start )

systemctl start rsync-inotity.service

;;

stop )

systemctl stop rsync-inotity.service

;;

restart )

systemctl restart rsync-inotity.service

;;

send )

echo "Keepalived stoppd in 192.168.1.102" | mail -s "Alarm Mail" [email protected]

;;

* )

echo "Usage:$0 start|stop|restart|send"

;;

esac

########################

# vim /etc/systemd/system/rsync-inotity.service

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第5张图片

################################

[Unit]

Description=File instant rsync script

[Service]

ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh &

[Install]

WantedBy=multi-user.target

################################

# vim /etc/full-rsync.conf

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第6张图片

#################################

src_dir="/opt/dataA"

dest_dir="/opt/dataB"

src_ip="192.168.1.102"

dest_ip="192.168.1.103"

##################################

# vim /etc/systemd/system/full-rsync.service

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第7张图片

####################################

[Unit]

Description=Full rsync script

After=network.target sshd.service

Wants=sshd.service

[Service]

EnvironmentFile=/etc/full-rsync.conf

ExecStart=/usr/bin/rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}

Type=oneshot

[Install]

WantedBy=multi-user.target

#####################################

serverB

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第8张图片

# vim /opt/data_realtime_rsync.sh

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第9张图片

#############################

#!/bin/bash

# File name:data_realtime_sync.sh

# File path: /opt/data_realtime_sync.sh

src_dir="/opt/dataB"

dest_dir="/opt/dataA"

src_ip="192.168.1.103"

dest_ip="192.168.1.102"

. /etc/init.d/functions

cd ${src_dir}

inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |

while read file;do

INO_EVENT=$(echo $file | awk '{print $1}')

INO_FILE=$(echo $file | awk '{print $2}')

if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then

for ip in ${dest_ip};do

rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}

done

if [[ $INO_EVENT =~ 'ATTRIB' ]];then

if [ ! -d "$INO_FILE" ];then

for ip in ${dest_ip};do

rsync -avzcR -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}

done

if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then

for ip in ${dest_ip};do

rsync -avzcR --delete -e ssh $(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}

done

###################################

# vim /etc/keepalived/keepalived.conf

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第10张图片

###########################################

! Configuration File for keepalived

global_defs {

router_id LVS_DEVEL

}

vrrp_script check_httpd {

script "/etc/keepalived/check_httpd.sh"

interval 5

}

vrrp_instance VI_1 {

state BACKUP

interface eno16777736

virtual_router_id 90

priority 100

advert_int 1

nopreempt

authentication {

auth_type PASS

auth_pass 1111

}

track_script {

check_httpd

}

virtual_ipaddress {

192.168.1.120

}

notify_master "/etc/keepalived/rsync-inotify.sh start"

notify_backup "/etc/keepalived/rsync-inotify.sh stop"

notify_stop "/etc/keepalived/rsync-inotify.sh send"

}

##########################################

# vim /etc/keepalived/check_httpd.sh

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第11张图片

##############################

#!/bin/sh

httpd_status1=$(ps -C httpd --no-heading|wc -l)

if [ "${httpd_status1}" = "0" ]; then

systemctl start httpd.service

sleep 5

httpd_status2=$(ps -C httpd --no-heading|wc -l)

if [ "${httpd_status2}" = "0" ]; then

systemctl stop keepalived.service

#############################

# vim /etc/keepalived/rsync-inotify.sh

########################

#!/bin/bash

case "$1" in

start )

systemctl start rsync-inotity.service

;;

stop )

systemctl stop rsync-inotity.service

;;

restart )

systemctl restart rsync-inotity.service

;;

send )

echo "Keepalived stoppd in 192.168.1.102" | mail -s "Alarm Mail" [email protected]

;;

* )

echo "Usage:$0 start|stop|restart|send"

;;

esac

########################

# vim /etc/systemd/system/rsync-inotity.service

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第12张图片

################################

[Unit]

Description=File instant rsync script

[Service]

ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh &

[Install]

WantedBy=multi-user.target

################################

# vim /etc/full-rsync.conf

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第13张图片

#################################

src_dir="/opt/dataB"

dest_dir="/opt/dataA"

src_ip="192.168.1.103"

dest_ip="192.168.1.102"

##################################

# vim /etc/systemd/system/full-rsync.service

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第14张图片

####################################

[Unit]

Description=Full rsync script

After=network.target sshd.service

Wants=sshd.service

[Service]

EnvironmentFile=/etc/full-rsync.conf

ExecStart=/usr/bin/rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}

Type=oneshot

[Install]

WantedBy=multi-user.target

#####################################

四、测试

# systemctl status httpd

# systemctl status keepalived

# systemctl status rsync-inotity

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第15张图片

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第16张图片

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第17张图片

五、全量同步

手动方式

A 全量同步B

# vim full_rsync_from_B_to_A.sh

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第18张图片

######################################################

#!/bin/bash

src_dir="/opt/dataA"

dest_dir="/opt/dataB"

src_ip="192.168.1.102"

dest_ip="192.168.1.103"

rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}/

#######################################################

B 全量同步A

# vim full_rsync_from_A_to_B.sh

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第19张图片

#######################################################

#!/bin/bash

src_dir="/opt/dataB"

dest_dir="/opt/dataA"

src_ip="192.168.1.103"

dest_ip="192.168.1.102"

rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}/

#######################################################

系统服务

# systemctl daemon-reload

# systemctl start full-rsync

# systemctl status full-rsync

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第20张图片

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第21张图片

通过/etc/rc.d/rc.local 开机执行

# vim /etc/rc.d/rc.local

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第22张图片

##################

# Full rsync

src_dir="/xxx"

dest_dir="/xxx"

src_ip="xx.xx.xx.xx"

dest_ip="xx.xx.xx.xx"

rsync -avz --delete -e ssh root@${dest_ip}:${dest_dir}/ ${src_dir}

##################

# chmod +x /etc/rc.d/rc.local

六、优化inotity

如果不进行优化，遇到瓶颈会有如下错误提示：

Failed to watch /your/path; upper limit on inotify watches reached!

Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.

【keepalived + rsync +inotify 】实现真正的高效数据实时同步_第23张图片

主要优化 max_user_watches 及 max_queued_events

具体大小可以等于当前数据盘的 inode最大值

这里以1T数据盘为例：

# echo "67108864" > /proc/sys/fs/inotify/max_user_watches

# echo "67108864" > /proc/sys/fs/inotify/max_queued_events

将下面参数写入 /etc/sysctl.conf，永久生效

# vim

/etc/sysctl.conf

fs.inotify.max_user_watches = 67108864

fs.inotify.max_queued_events = 67108864

七、参考

keepalived + rsync +inotify 实现真正的高效数据实时同步

https://fandenggui.com/post/keepalived-rsync-inotify.html

真正的inotify+rsync实时同步彻底告别同步慢

http://www.ttlsa.com/web/let-infotify-rsync-fast

【Keepalived 】keepalived check and notify scripts

大家知道keepalived会有四种状态的变化，每种状态变化时，都可以调用一个脚本

当进入Master状态时会呼叫notify_master

当进入Backup状态时会呼叫notify_backup

当发现异常情况时进入Fault状态呼叫notify_fault

当Keepalived程序终止时则呼叫notify_stop

进入Master和Backup这两种状态很容易理解了，就是分别变为主和从

进入Fault，简单的说就是keepalived发现自己有问题了，不能能再去参与Master竞选了，你得修好他才行。

进入这种状态一般是keepalived自身出问题了，或者keepalived检测的网卡出问题不通了，再或者就是我们自己写的检测业务的脚本返回错误

进入Stop 这个就容易理解了，执行service keepalived stop 或者 systemctl stop keepalived 就会进入这个状态。

设置selinux为宽松模式

# setenforce 0

# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config

CentOS防火墙默认是不允许keepalived使用 vrrp的组播。

如果不开启组播ip,keepalived双机不能实现热备的效果,只能实现负载的效果,即虚拟ip不能实现漂移。

Check that the multicast IP and protocol for VRRP are allowed in the firewall on both servers.

For firewalld:

添加规则

# firewall-cmd --direct --permanent--add-rule ipv4 filterINPUT 0 --in-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

# firewall-cmd --direct --permanent--add-ruleipv4 filterOUTPUT 0 --out-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

# firewall-cmd --reload

删除规则

# firewall-cmd --direct --permanent--remove-ruleipv4 filter INPUT 0 --in-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

# firewall-cmd --direct --permanent--remove-rule ipv4 filter OUTPUT 0 --out-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

# firewall-cmd --reload

For iptables:

添加规则

# iptables -A INPUT -p vrrp -j ACCEPT

# iptables -A OUTPUT -p vrrp -j ACCEPT

# service iptables save

删除规则

# iptables -D INPUT -p vrrp -j ACCEPT

# iptables -D OUTPUT -p vrrp -j ACCEPT

# service iptables save

keepalived基本应用解析

http://blog.51cto.com/pangge/1301878

keepalived官方文档

http://www.keepalived.org/doc/introduction.html

使用keepalived实现redis的故障切换

http://peiqiang.net/2014/11/21/keepalived-and-redis.html

Kamailio High Availability Done Right with Keepalived

http://blog.unicsolution.com/2015/01/kamailio-high-availability-with.html

Keepalived Check and Notify Scripts

https://tobrunet.ch/keepalived-check-and-notify-scripts

【keepalived + rsync +inotify 】实现真正的高效数据实时同步

实验背景

一、实验环境

二、软件安装

三、主机配置

四、测试

五、全量同步

六、优化inotity

七、参考

【Keepalived 】keepalived check and notify scripts

你可能感兴趣的:(【keepalived + rsync +inotify 】实现真正的高效数据实时同步)