理论+实操：rsync远程同步

前言：

rsync同步简介

配置rsync备份源

rsync命令基本用法

rsync备份操作示例

rsync+inotify结合使用

一：关于rsync——一款快速增量备份工具

• remote sync ，远程同步，它是一个开源的快速备份工具，可以在不同主机之间镜像同步整个目录树
• 支持本地复制，或者与其他ssh、rsync主机同步
• 官方网站：http://rsync.samba.org
• 支持增量备份、保持连接和权限，且采用优化的同步算法，传输前执行压缩，因此非常适用于异地备份、镜像服务器等应用

二：配置rsync 同步源服务器

2.1 rsync同步源

• 指备份操作的远程服务器，也被称为备份源

2.2 配置rsync源基本思路

备注：在提供服务的同步源服务器上操作

• 建立rsyncd.conf配置文件
• 建立独立的账号文件
• 启用rsync——daemon模式

2.3 应用示例

• 用户gsybk，允许下行同步
• 操作的目录为/var/www/html

2.3 配置文件rsyncd.conf

• 需要手动建立，语法类似samba配置
• 认证配置auth users、secrets file ，不加则默认为匿名

2.4 rsync账号文件

• 采用“用户名：密码”的格式记录，每行一个用户记录
• 独立的账号数据，不依赖系统账号

2.5 启动rsync服务

• 通过–daemon独自提供服务
• 执行 kill $(cat /var/run/rsyncd.pid)关闭rsync服务

三： rsync 常用选项

-a：归档模式，递归并保留对象属性，等同于 -rlptgoD
-v：显示同步过程的详细（verbose）信息
-z：在传输文件时进行压缩（compress）
-H：保留硬连接文件
-A：保留ACL属性信息
–delete：删除目标位置有而原始位置没有的文件
–checksum：根据对象的校验和来决定是否跳过文件

四：实操演示

• 用户gsybk，允许下行同步
• 操作的目录为/var/www/html

基本思路：

• 建立rsyncd.conf配置文件
• 建立独立的账号文件
• 启用rsync——daemon模式

服务端IP： 192.168.247.160

客户端IP： 192.168.247.134

4.1 安装httpd作为数据环境

[root@a1 ~]# hostnamectl set-hostname rsyncd
[root@a1 ~]# su
su
[root@rsyncd ~]# yum install -y httpd

4.2 检查是否安装rsync服务

[root@rsyncd ~]# rpm -q rsync
rsync-3.1.2-6.el7_6.1.x86_64

若是没有安装，执行yum -y install rsync

4.3 检查防火墙和核心防护

[root@rsyncd ~]# systemctl stop firewalld
[root@rsyncd ~]# systemctl disable firewalld
[root@rsyncd ~]# setenforce ?
setenforce: SELinux is disabled
[root@rsyncd ~]# sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

4.4 修改/etc/rsyncd.conf配置文件

[root@rsyncd ~]# vim /etc/rsyncd.conf 
//下面是原有文件
# /etc/rsyncd: configuration file for rsync daemon mode

# See rsyncd.conf man page for more options.

# configuration example:

# uid = nobody
# gid = nobody
# use chroot = yes
# max connections = 4
# pid file = /var/run/rsyncd.pid
# exclude = lost+found/
# transfer logging = yes
# timeout = 900
# ignore nonreadable = yes
# dont compress   = *.gz *.tgz *.zip *.z *.Z *.rpm *.deb *.bz2

# [ftp]
#        path = /home/ftp
#        comment = ftp export area

直接删掉重新输入

 uid = nobody
 gid = nobody
 use chroot = yes
 address = 192.168.247.160
 port 873
 log file = /var/log/rsyncd.log
 pid file = /var/run/rsyncd.pid
 hosts allow = 192.168.247.160/24
[wwwroot]
 path = /var/www/html
 comment = Document Root of www.gsy.com
 read only = yes
 dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z
 auth users = gsybk
 secrets file = /etc/rsyncd_users.db

脚本解释

uid = nobody
gid = nobody
use chroot = yes ####禁锢在源目录
address = 192.168.247.160 ####监听地址
port 873 ####监听端口号
log file = /var/log/rsyncd.log ####日志文件位置
pid file = /var/run/rsyncd.pid ####存放进程ID的文件位置
hosts allow = 192.168.247.0/24 ####允许访问的客户机地址
[wwwroot] ####共享模块名称
path = /var/www/html ####源目录的实际路径
comment = Document Root of www.gsy.com
read only =yes #####是否只读
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z ####同步时不再压缩的文件类型
auth users =gsybk #####授权账户
secrets file = /etc/rsyncd_users.db ####存放账户信息的数据文件

4.5 修改账户文件/etc/rsyncd_users.db

采用“用户名:密码”的记录格式，每行一个用户记录
独立的账号数据，不依赖于系统账号

[root@rsyncd ~]# vim /etc/rsyncd_users.db
gsybk:123123
[root@rsyncd ~]# chmod 600 /etc/rsyncd_users.db
[root@rsyncd ~]# ls -ld /var/www/html  
drwxr-xr-x. 4 root root 29 Aug  8  2019 /var/www/html

4.6 启动rsync服务

通过–daemon独自提供服务

如果要停止这个服务 kill $(cat /var/run/rsyncd.pid)

[root@rsyncd ~]# rsync --daemon       
[root@rsyncd ~]# netstat -anpt |grep rsync
tcp        0      0 192.168.247.160:873     0.0.0.0:*               LISTEN      7531/rsync       

4.7 进行测试，在/var/www.httml/下创建文件

[root@rsyncd ~]# cd /var/www/html/
[root@rsyncd html]# ls
[root@rsyncd html]# echo "this is gsydsg" > gsy.html
[root@rsyncd html]# echo "this time is 20200223" > gsy1.html
[root@rsyncd html]# ls
gsy1.html  gsy.html

4.8 新起一台客户端，使用rsync进行远程操作

[root@b ~]# hostnamectl set-hostname kehuduan
[root@b ~]# su
[root@kehuduan ~]# rpm -q rsync
package rsync is not installed
[root@kehuduan ~]# yum install -y rsync

4.9 rsync用法

可以作为本地复制命令使用

rsync [选项] 原始位置 目标位置

###常用选项###
-a：归档模式，递归并保留对象属性，等同于 -rlptgoD
-v：显示同步过程的详细（verbose）信息
-z：在传输文件时进行压缩（compress）
-H：保留硬连接文件
-A：保留ACL属性信息
–delete：删除目标位置有而原始位置没有的文件
–checksum：根据对象的校验和来决定是否跳过文件

[root@kehuduan ~]# cd /opt
[root@kehuduan opt]# ls
[root@kehuduan opt]# rsync /etc/fstab /opt		
[root@kehuduan opt]# ls
fstab

4.10 配置源的两种表示方法

格式1：rsync [选项] 用户名@主机地址::共享模块名 目标路径

[root@kehuduan opt]# rsync -avz [email protected]::wwwroot /opt
Password: 123123
receiving incremental file list
./
gsy.html
gsy1.html

sent 65 bytes  received 209 bytes  60.89 bytes/sec
total size is 37  speedup is 0.14
[root@kehuduan opt]# ll
total 16
-rw-r--r--  1 root root  501 Mar 23 21:58 fstab
-rw-r--r--  1 root root   22 Mar 23 21:55 gsy1.html
-rw-r--r--  1 root root   15 Mar 23 21:54 gsy.html
[root@kehuduan opt]# cat gsy1.html 
this time is 20200223
[root@kehuduan opt]# cat gsy.html 
this is gsydsg

格式2： rsync [选项] rsync://用户名@主机地址/共享模块名 目标路径

[root@kehuduan opt]# ll
total 8
-rw-r--r--  1 root root  501 Mar 23 21:58 fstab
[root@kehuduan opt]# rsync -avz rsync://[email protected]/wwwroot /opt
Password: 
receiving incremental file list
./
gsy.html
gsy1.html
sent 65 bytes  received 209 bytes  109.60 bytes/sec
total size is 37  speedup is 0.14
[root@kehuduan opt]# ll
total 16
-rw-r--r--  1 root root  501 Mar 23 21:58 fstab
-rw-r--r--  1 root root   22 Mar 23 21:55 gsy1.html
-rw-r--r--  1 root root   15 Mar 23 21:54 gsy.html
[root@kehuduan opt]# cat gsy1.html 
this time is 20200223
[root@kehuduan opt]# cat gsy.html 
this is gsydsg

4.11 测试–delete：删除目标位置有而原始位置没有的文件

可以发现，原先存在的fstab文件被删除掉了

[root@kehuduan opt]# rm -rf * 
[root@kehuduan opt]# ll
[root@kehuduan opt]# rsync /etc/fstab /opt
[root@kehuduan opt]# ls
fstab
[root@kehuduan opt]# rsync -avzH --delete [email protected]::wwwroot /opt
Password: 
receiving incremental file list
deleting fstab
./
gsy.html
gsy1.html
sent 69 bytes  received 213 bytes  112.80 bytes/sec
total size is 37  speedup is 0.13
[root@kehuduan opt]# ll
-rw-r--r-- 1 root root 22 Mar 23 21:55 gsy1.html
-rw-r--r-- 1 root root 15 Mar 23 21:54 gsy.html

五：rsync的免交互处理

• 设置好一个文件，里面输入对应的密码
• 在输入命令时添加–password-file=密码文件路径

[root@kehuduan opt]# vi /mima
123123
[root@kehuduan opt]# chmod 600 /mima
[root@kehuduan opt]# rsync -avzH --password-file=/mima [email protected]::wwwroot /opt
receiving incremental file list
./
gsy.html
gsy1.html
sent 69 bytes  received 213 bytes  564.00 bytes/sec
total size is 37  speedup is 0.13
[root@kehuduan opt]# ll
total 8
-rw-r--r-- 1 root root 22 Mar 23 21:55 gsy1.html
-rw-r--r-- 1 root root 15 Mar 23 21:54 gsy.html

六： 设置周期性计划任务

每天晚上10点半对服务器网站目录更新一次

[root@kehuduan opt]# mkdir /webbak
[root@kehuduan opt]#  crontab -e
30 22 * * * /usr/bin/rsync -avz --delete --password-file=/mima [email protected]::wwwroot /webbak
[root@kehuduan opt]#systemctl restart crond
[root@kehuduan opt]#systemctl enable crond

七 ：rsync实时同步

7.1 周期性计划任务的不足之处：

• 执行备份的时间固定，延迟明显、实时性差
• 当同步源长期不变化时，密集的定期任务是不必要的

7.2 实时同步的优点：

• 一旦同步源出现变化，立即启动备份
• 只要同步源无变化，则不执行备份

7.3 实时同步需要在配置文件中，只读设为no

[root@rsyncd ~]# vi /etc/rsyncd.conf
read only = no 

7.4 inotify 介绍

从版本2.6.13开始提供
可以监控文件系统的变动情况，并作出通知响应
辅助软件：inotify-tools

[root@kehuduan opt]#  cat /proc/sys/fs/inotify/max_queued_events 
//监控队列大小
16384
[root@kehuduan opt]# cat /proc/sys/fs/inotify/max_user_instances 
//最多监控实例数
128
[root@kehuduan opt]# cat /proc/sys/fs/inotify/max_user_watches 
//每个实例最多监控文件数
8192

fs.inotify.max_queued_events：表示调用inotify_init时分配给inotify instance中可排队的event的数目的最大值，超出这个值的事件被丢弃，但会触发IN_Q_OVERFLOW事件。

fs.inotify.max_user_instances：表示每一个real user ID可创建的inotify instatnces的数量上限，默认128.

fs.inotify.max_user_watches：表示同一用户同时可以添加的watch数目（watch一般是针对目录，决定了同时同一用户可以监控的目录数量）

7.5 修改inotify内核参数

[root@kehuduan opt]# vi /etc/sysctl.conf
//追加
fs.inotify.max_queued_events = 32768
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
[root@kehuduan opt]# sysctl -p
fs.inotify.max_queued_events = 32768
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

7.6 安装inotify-tools辅助工具

inotifywait：用于持续监控，实时输出结果

inotifywatch：用于短期监控，任务完成后再出结果

[root@kehuduan opt]# mount.cifs //192.168.0.88/linuxs /abc -o vers=2.0
Password for root@//192.168.0.88/linuxs:  
[root@kehuduan opt]# tar xvzf /abc/inotify-tools-3.14.tar.gz -C /opt
[root@kehuduan opt]# cd /opt/inotify-tools-3.14/
[root@kehuduan inotify-tools-3.14]# ls
aclocal.m4  ChangeLog     config.h.in  configure     COPYING  INSTALL     libinotifytools  Makefile.am  man      NEWS    src
AUTHORS     config.guess  config.sub   configure.ac  depcomp  install-sh  ltmain.sh        Makefile.in  missing  README
[root@kehuduan inotify-tools-3.14]# ./configure 
[root@kehuduan inotify-tools-3.14]# make && make install
[root@kehuduan inotify-tools-3.14]# inotifywait -mrq -e modify,create,move,delete /webbak/
//这边要另外开个终端在/opt/myweb 写一个文件进文件监控

新开一个终端

连接成功
Last login: Mon Mar 23 21:57:01 2020
[root@kehuduan ~]# cd /webbak
[root@kehuduan webbak]# touch 123
[root@kehuduan webbak]# ls
123

反馈写入信息
inotifywait -mrq -e modify,create,move,delete /webbak

到服务端查看

[root@rsyncd ~]# cd /var/www/html
[root@rsyncd html]# ls
gsy1.html  gsy.html

123没有写入，是因为没有修改配置文件的只读

[root@rsyncd html]# vi /etc/rsyncd.conf 
 read only = no
 [root@rsyncd html]# kill $(cat /var/run/rsyncd.pid)
[root@rsyncd html]# rsync --daemon  

7.7 通过inotifywait触发rsync同步操作

通过inotifywait触发rsync同步操作
使用while、read持续获取监控结果
根据结果可以作进一步判断，决定执行何种操作

[root@kehuduan webbak]# vim /opt/inotify.sh
#!/bin/bash
dd=/webbak		
# 注意，写/webbak会将这个目录也同步上去，/webbak/则只会同步目录内的文件
user=gsybk
passfile=/mima
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib $dd"
RSYNC_CMD="rsync -azH --delete --password-file=$passfile $dd [email protected]::wwwroot"
 
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
        $RSYNC_CMD
#       echo "${FILE} was rsynced" >>/opt/inotify_rsync.log

    fi
done
[root@kehuduan webbak]# chmod 755 /opt/inotify.sh 
[root@kehuduan ~]# /opt/inotify.sh //新开一个客户端

7.8 添加权限

[root@rsyncd html]# chmod 777 /var/www/html/

[root@kehuduan webbak]# chmod 777 /webbak

7.9 创建测试文件

[root@kehuduan webbak]# ls
[root@kehuduan webbak]# echo "123" > 123

[root@rsyncd html]# tree .
.
└── webbak
    └── 123

1 directory, 1 file
[root@rsyncd html]# cat webbak/123 
123
[root@rsyncd html]# ll
total 0
drwx------ 2 nobody nobody 17 Mar 24 09:19 webbak

重启服务后，再次测试，发现可以将客端创建的数据文件同步到服务端，但是客户端会有报错

[root@kehuduan ~]# /opt/inotify.sh 
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.111.EOGk5s" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.123.yGrV12" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.22.uwbxYC" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.23.XrxbVc" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.951.CNMQRM" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.965483.5XBwOm" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.111.8j5rzM" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.123.Tohc9m" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.22.eFgYIX" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.23.VWpLiy" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.951.BolBS8" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.965483.HFBssJ" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.123.hboXtd" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
rsync: chgrp "/webbak" (in wwwroot) failed: Operation not permitted (1)
rsync: chgrp "/webbak/.123.323Xnt" (in wwwroot) failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

检查系统版本

[root@rsyncd html]# cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)
[root@kehuduan webbak]# cat /proc/version 
Linux version 3.10.0-693.el7.x86_64 ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017
[root@kehuduan webbak]# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 

递归设置权限，只能解决手动同步问题，不可以解决脚本同步

[root@rsyncd html]# chmod -R 777 /var/www/html

[root@kehuduan webbak]# rsync -avz --delete --password-file=/mima [email protected]::wwwroot /webbak/
receiving incremental file list
./
2146

sent 46 bytes  received 182 bytes  456.00 bytes/sec
total size is 4  speedup is 0.02

将服务端的/etc/rsyncd.conf配置文件中的uid和gid改为root

可以解决

 uid = root
 gid = root

cc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Tue Aug 22 21:09:27 UTC 2017
[root@kehuduan webbak]# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

递归设置权限，只能解决手动同步问题，不可以解决脚本同步

[root@rsyncd html]# chmod -R 777 /var/www/html

[root@kehuduan webbak]# rsync -avz --delete --password-file=/mima [email protected]::wwwroot /webbak/
receiving incremental file list
./
2146

sent 46 bytes received 182 bytes 456.00 bytes/sec
total size is 4 speedup is 0.02

将服务端的/etc/rsyncd.conf配置文件中的uid和gid改为root

可以解决

uid = root
gid = root