【MySQL】rsync+sersync文件实时同步
一：什么是Rsync？
Rsync（Remote Synchronize）是一款开源的、快速的、多功能的、可以实现全量及增量的本地或远程数据同步备份的优秀工具，并且支持多种操作系统平台运行。
二：什么Sersync？
1、sersync是基于inotify开发的，类似于inotify-tools的工具，Sersync可以记录下被监听目录中发生变化的（包括增加、删除、修改）具体某一个文件或者某一个目录的名字，然后使用rsync同步的时候，只同步发生变化的文件或者目录，因此效率更高。
2、主要应用场景为数据体积大，并且文件很多。
小结：Rsync+sersync
1、sersync可以记录下被监听目录中发生变化的（包括增加、删除、修改）具体某一个文件或某一个目录的名字；
2、rsync在同步的时候，只同步发生变化的这个文件或者这个目录（每次发生变化的数据相对整个同步目录数据来说是很小的，rsync在遍历查找比对文件时，速度很快），因此，效率很高。
三：环境
备份服务器：192.168.1.151 操作系统：Centos7.9
数据源服务器：192.168.1.150 操作系统：Centos7.9
四：备份服务器操作
1、关闭 selinux
永久/临时关闭：
vi /etc/selinux/config
SELINUX=disabled
setenforce 0
2、关闭防火墙
systemctl stop firewalld.service
3、安装rsync服务端软件
yum install rsync xinetd -y
vim /etc/rc.d/rc.local
/usr/bin/rsync --daemon --config=/etc/rsyncd.conf
systemctl start xinetd
4、创建rsyncd.conf配置文件
vim /etc/rsyncd.conf
uid = root
gid = root
use chroot = yes
max connections = 0
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
secrets file = /etc/rsync.pass
motd file = /etc/rsyncd.Motd
[back_data]
path=/backup
comment = A directory in which data is stored
ignore errors = yes
read only = no
hosts allow = 192.168.1.150
5、创建用户认证文件
vim /etc/rsync.pass
jd:jd123456
6、设置文件权限
chmod 600 /etc/rsyncd.conf
chmod 600 /etc/rsync.pass
7、启动rsync和xinetd
/usr/bin/rsync --daemon --config=/etc/rsyncd.conf
systemctl start rexinetd
五：数据源服务器操作
(1)安装rsync客户端软件
1、关闭 selinux
永久/临时关闭：
vim /etc/selinux/config
SELINUX=disabled
setenforce 0
2、关闭防火墙
systemctl stop firewalld.service
3、安装rsync客户端软件
yum install rsync xinetd -y
vim /etc/rc.local
/usr/bin/rsync --daemon
vim /etc/reyncd.conf
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
motd file = /etc/rsyncd.Motd
[Sync]
comment = Sync
uid = root
gid = root
port= 873
授权可执行权限并启动
chmod +x /etc/rc.d/rc.local
systemctl start xinetd
4、创建认证密码文件
vim /etc/passwd.txt
jd123456
chmod 600 /etc/passwd.txt
5、测试数据同步
数据源服务器192.168.1.150到备份服务器~.151之间的数据同步
rsync --help可以了解rsync怎么用的
rsync -avH --port=873 --progress --delete /backup [email protected]::back_data --password-file=/etc/passwd.txt
[root@jd-mysql backup]# rsync -avH --port=873 --progress --delete /backup [email protected]::back_data --password-file=/etc/passwd.txt
sending incremental file list
backup/
backup/all_db.sql
918,458 100% 120.67MB/s 0:00:00 (xfr#1, to-chk=4/6)
backup/backup_db.sh
169 100% 20.63kB/s 0:00:00 (xfr#2, to-chk=3/6)
backup/hunan_jiaojiao.sql
13,631 100% 1.62MB/s 0:00:00 (xfr#3, to-chk=2/6)
backup/tennis.sql
8,043 100% 981.81kB/s 0:00:00 (xfr#4, to-chk=1/6)
backup/ws.sql
2,072 100% 252.93kB/s 0:00:00 (xfr#5, to-chk=0/6)
sent 943,009 bytes received 115 bytes 1,886,248.00 bytes/sec
total size is 942,373 speedup is 1.00
可以测试，成功了，不过得每次执行这句命令才可以同步数据，所以引出sersync
(2)安装sersync工具，实时触发rsync进行同步
备注：Linux下支持inotify的内核最小为2.6.13，可以输入命令：#uname -a查看内核
CentOS 7.0内核为3.10.0，默认已经支持inotify
1、修改inotify默认参数（inotify默认内核参数值太小） 修改参数：
临时修改：
[root@jd-mysql backup]# sysctl -w fs.inotify.max_queued_events="99999999"
fs.inotify.max_queued_events = 99999999
[root@jd-mysql backup]# sysctl -w fs.inotify.max_user_watches="99999999"
fs.inotify.max_user_watches = 99999999
[root@jd-mysql backup]# sysctl -w fs.inotify.max_user_instances="65535"
fs.inotify.max_user_instances = 65535
永久修改：
vim /etc/sysctl.conf
fs.inotify.max_queued_events=99999999
fs.inotify.max_user_watches=99999999
fs.inotify.max_user_instances=65535
2、安装sersync
64位下载：curl -O https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/sersync/sersync2.5.4_64bit_binary_stable_final.tar.gz
[root@jd-mysql backup]# curl -O https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/sersync/sersync2.5.4_64bit_binary_stable_final.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 710k 100 710k 0 0 576k 0 0:00:01 0:00:01 --:--:-- 576k
[root@jd-mysql backup]# ls
sersync2.5.4_64bit_binary_stable_final.tar.gz
[root@jd-mysql backup]# tar xf sersync2.5.4_64bit_binary_stable_final.tar.gz
[root@jd-mysql backup]# ls
GNU-Linux-x86 sersync2.5.4_64bit_binary_stable_final.tar.gz
[root@jd-mysql backup]# mv GNU-Linux-x86 /usr/local/sersync
3、创建rsync
[root@jd-mysql backup]# cd /usr/local/sersync
[root@jd-mysql sersync]# cp confxml.xml confxml.xml-bak
[root@jd-mysql sersync]# cp confxml.xml data_configxml.xml
[root@jd-mysql sersync]# ls
confxml.xml confxml.xml-bak data_configxml.xml sersync2
4、修改配置 data_configxml.xml 文件
[root@jd-mysql sersync]# vim data_configxml.xml
----- 24行 -----
<localpath watch="/backup">
<remote ip="192.168.1.151" name="back_data"/>
<!--<remote ip="192.168.8.39" name="tongbu"/>-->
<!--<remote ip="192.168.8.40" name="tongbu"/>-->
</localpath>
<rsync>
<commonParams params="-artuz"/>
<auth start="true" users="root" passwordfile="/etc/passwd.txt"/>
<userDefinedPort start="false" port="874"/><!-- port=874 -->
<timeout start="false" time="100"/><!-- timeout=100 -->
<ssh start="false"/>
</rsync>
5、启动服务
/usr/local/sersync/sersync2 -d -r -o /usr/local/sersync/data_configxml.xml
启动成功：
[root@jd-mysql sersync]# /usr/local/sersync/sersync2 -d -r -o /usr/local/sersync/data_configxml.xml
set the system param
execute：echo 50000000 > /proc/sys/fs/inotify/max_user_watches
execute：echo 327679 > /proc/sys/fs/inotify/max_queued_events
parse the command param
option: -d run as a daemon
option: -r rsync all the local files to the remote servers before the sersync work
option: -o config xml name： /usr/local/sersync/data_configxml.xml
daemon thread num: 10
parse xml config file
host ip : localhost host port: 8008
daemon start，sersync run behind the console
use rsync password-file :
user is root
passwordfile is /etc/passwd.txt
config xml parse success
please set /etc/rsyncd.conf max connections=0 Manually
sersync working thread 12 = 1(primary thread) + 1(fail retry thread) + 10(daemon sub threads)
Max threads numbers is: 22 = 12(Thread pool nums) + 10(Sub threads)
please according your cpu ，use -n param to adjust the cpu rate
------------------------------------------
rsync the directory recursivly to the remote servers once
working please wait...
execute command: cd /backup && rsync -artuz -R --delete ./ [email protected]::back_data --password-file=/etc/passwd.txt >/dev/null 2>&1
run the sersync:
watch path is: /backup
测试：
数据源端：
[root@jd-mysql backup]# mkdir 41-1951
[root@jd-mysql backup]# ls
41-1951 hosts sersync2.5.4_64bit_binary_stable_final.tar.gz
all_db.sql hunan_jiaojiao.sql tennis.sql
backup_db.sh passwd ws.sql
备份端：
[root@jd-mysql-2 backup]# ls
41-1951 hosts sersync2.5.4_64bit_binary_stable_final.tar.gz
all_db.sql hunan_jiaojiao.sql tennis.sql
backup_db.sh passwd ws.sql
[root@jd-mysql-2 backup]# pwd
/backup
6、设置sersync监控开机自动执行
[root@jd-mysql sersync]# vim /etc/rc.d/rc.local
/usr/local/sersync/sersync2 -d -r -o /usr/local/sersync/data_configxml.xml
max_binlog_size最大1G
root@(none) 11:30 mysql>show variables like "%max_binlog%";
+----------------------------+----------------------+
| Variable_name | Value |
+----------------------------+----------------------+
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
+----------------------------+----------------------+
3 rows in set (0.00 sec)
root@(none) 11:33 mysql>select 1073741824/1024/1024/1024
-> ;
+---------------------------+
| 1073741824/1024/1024/1024 |
+---------------------------+
| 1.000000000000 |
+---------------------------+
1 row in set (0.00 sec)