Day32 课堂作业
1. rsync介绍与实践
1、什么是rsync?
rsync
,Rsync
英文全称为Remote synchronization
缩写rsync
是开源、高速的、可实现本地以及远程,全量以及增量的数据复制(拷贝)工具。
官方链接资料:
http://www.samba.org/ftp/rsync/rsync.html
2、全量和增量区别
全量复制:
[root@nfs01 ~]# cp -a /etc/ /opt/
[root@nfs01 ~]# \cp -a /etc/ /opt
[root@nfs01 ~]# touch /etc/oldboy.txt
[root@nfs01 ~]# \cp -a /etc/ /opt/
增量复制:只复制
oldboy.txt
3、rsync的作用
工作中需要定时/实时数据备份。本地服务器目录、不同机器、
不同机房之间的数据备份。都可以用rsync
完成。
4、rsync功能特性
- 支持拷贝普通文件与特殊文件,如链接文件,设备文件等。
- 支持排除指定文件或目录同步的功能,相当于打包命令
tar
的排除功能。- 可以做到保持原文件或目录的权限、时间、软硬链接、属主、组等所有属性均不改变。
- 可实现增量复制,既只复制发生变化的数据,因此数据传输效率很高。
- 可以使用
rcp
,rsh
,ssh
等方式来配合进行隧道加密传输文件(rsync
本身不对数据加密)。- 可以通过
socket
(进程方式)传输文件和数据(服务端和客户端)。- 支持匿名或认证(无需系统用户)进程模式传输,安全的进行数据备份及镜像
5、增量复制原理
使用
quick check
算法,只对增量的部分复制,根据大小属性的变化进行复制。
版本2.x
比对差异后复制,版本3.x
一边比对一边复制。
参考地址:https://coolshell.cn/articles/7425.html
6、rsync三种工作模式介绍
1)本地(
local
)
rsync
就是一个命令语法:
rsync
[OPTION...] SRC... [DEST]
rsync
参数选项 [源目录或文件] [目的目录或文件]命令操作:
rsync
命令
a. 把数据从一个地方复制到另一个地方(仅在一台机器增量),相当于cp
。
b. 通过加参数实现删除文件和清空文件内容的功能,相当于rm
命令。
c. 查看属性信息功能,相当于ls
。a. 拷贝实践
不保持属性:
[root@backup~]# ls /opt
[root@backup ~]# rsync /etc/hosts /opt
[root@backup ~]# ls /opt
hosts
[root@backup ~]# \cp /etc/hosts /opt
保持属性:
[root@backup ~]# rsync -zrtopg /etc/hosts /opt/
[root@backup ~]# ls -lhi /etc/hosts /opt/hosts
16829878 -rw-r--r--. 1 root root 332 4月 12 11:24 /etc/hosts
71373 -rw-r--r-- 1 root root 332 4月 12 11:24 /opt/hosts
b. 删除实践
删除文件内容:
--------------------------------------------------------------
==>创建一个空文件/null.txt<==
--------------------------------------------------------------
[root@backup ~]# touch /null.txt
[root@backup ~]# cat /opt/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.1.5 lb01
172.16.1.6 lb02
172.16.1.7 web01
172.16.1.8 web02
172.16.1.9 web03
172.16.1.31 nfs01
172.16.1.41 backup
172.16.1.51 db01 db01.etiantian.org
172.16.1.61 m01
--------------------------------------------------------------
==>让后面hosts和一样前面null.txt<==
--------------------------------------------------------------
[root@backup ~]# rsync -r --delete /null.txt /opt/hosts
[root@backup ~]# cat /opt/hosts
删除目录下所有文件:
--------------------------------------------------------------
==>创建一个目录/null<==
--------------------------------------------------------------
[root@backup ~]# mkdir /null
--------------------------------------------------------------
==>让后面/opt和前面/null目录内容保持一致<==
--------------------------------------------------------------
[root@backup ~]# rsync -r --delete /null/ /opt/
[root@backup ~]# ls /opt/
c. 查看属性
[root@backup ~]# rsync /etc/hosts
-rw-r--r-- 332 2019/04/12 11:24:41 hosts
2)远程Shell模式
借助类似
ssh
隧道传输数据,适合不同的机器之间复制。
异地拷贝,相当于scp
,区别:scp
是远程全量拷贝
有两种情况
第一种:pull
:(拉) 从远端拉取到本地。语法:
rsync
[OPTION...] [USER@]HOST:SRC... [DEST]
rsync
[参数选项] [认证用户]@[主机地址]:[源路径] [本地路径]第二种:
push
:(推)从本地推到远端。语法:
rsync
[OPTION...] SRC... [USER@]HOST:[DEST]
rsync
[参数选项] [本地路径] [认证用户]@[主机地址]:[目标路径]
pull
的实践:
--------------------------------------------------------------
==>两条命令在目前是等价的<==
--------------------------------------------------------------
rsync -avz [email protected]:/opt/hosts /opt
rsync -avz -e "ssh -p 22" [email protected]:/opt/hosts /opt
--------------------------------------------------------------
==>输入1.41的root密码即可<==
--------------------------------------------------------------
[root@nfs01 /opt]# rsync -avz -e "ssh -p 22" [email protected]:/opt/hosts /opt
[email protected]'s password:
receiving incremental file list
sent 20 bytes received 44 bytes 11.64 bytes/sec
total size is 332 speedup is 5.19
--------------------------------------------------------------
==>在本地检查结果<==
--------------------------------------------------------------
[root@nfs01 ~]# cd /opt/
[root@nfs01 /opt]# ls
hosts
[root@nfs01 /opt]#
rsync -avz /etc/hosts [email protected]:/opt/
rsync -avz /etc/hosts -e "ssh -p 22" [email protected]:/opt/
上述命令是等价的。
-e
指定通道ssh
ssh
服务连接客户端-p 22
指定22
端口。
push
实践:
--------------------------------------------------------------
==>加密传输。<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc/hosts [email protected]:/opt/
The authenticity of host '172.16.1.41 (172.16.1.41)' can't be established.
ECDSA key fingerprint is SHA256:qZSBkrmOv7xO/63qOU1uLXkPyNVHdkqvrNAcAmXqNEk.
ECDSA key fingerprint is MD5:23:d0:cb:a9:f4:7c:0b:eb:2d:07:00:e1:a3:12:d8:33.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.1.41' (ECDSA) to the list of known hosts.
[email protected]'s password:
sending incremental file list
hosts
sent 219 bytes received 35 bytes 14.51 bytes/sec
total size is 332 speedup is 1.31
--------------------------------------------------------------
==>在backup里检查<==
--------------------------------------------------------------
[root@backup ~]# cd /opt/
[root@backup /opt]# ls
hosts
--------------------------------------------------------------
==>命令也可以这样写<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc/hosts -e "ssh -p 22" [email protected]:/opt/
[email protected]'s password:
sending incremental file list
sent 44 bytes received 12 bytes 22.40 bytes/sec
total size is 332 speedup is 5.93
小结
- a. 参照物,执行命令的机器
- b.
[email protected]
使用的用户和主机,就用主机和用户的密码。
强调一个重点:适合rsync
null
和null/
区别,
null
是目录和目录下的内容,null/
只是目录下的内容,不含本身。
3)rsync守护进程模式
首先要搭建rsync服务端(要有守护进程),然后才能在客户端实现推拉数据。
该模式是企业运维的重要模式(见后续)
rsync
服务模式:开启后台进程,接受别人访问。
7、rsync命令参数
表1:
rsync
命令参数:
参数 | 说明 |
---|---|
-v, --verbose | 显示输出过程 |
-z, --compress | 压缩 |
-a, --archive | 多参数集合(-rtopgDl) |
-r, --recursive | 递归 |
-t, --times | 保持修改时间属性 |
-o, --owne | r 保持属主不变 |
-p, --perms | 保持权限不变 |
-g, --group | 保持用户组不变 |
-l, --links | 保持拷贝软连接 |
-q, --quie | t 安静的拷贝 |
--bwlimit=KBPS | 限制I/O带宽,KBytes per second。 |
--delete | 删除 |
--exclude | 排除 |
--exclude-from | 从文件中排除 |
提示:企业常用参数组合:
-avz
或者-vzrtopg
2. 部署rsync实战
架构前的准备已在前面介绍了,在此,不在重复了
rsync 守护进程模式应用实践
(1)以下操作都是在backup服务器
1)安装(前期已经安装过了)
--------------------------------------------------------------
==>查看版本<==
--------------------------------------------------------------
[root@backup ~]# rsync --version
rsync version 3.1.2 protocol version 31
--------------------------------------------------------------
==>若没有安装,下面是安装的命令<==
--------------------------------------------------------------
yum install rsync
2)配置配置文件/etc/rsyncd.conf
--------------------------------------------------------------
==>操作前先备份是个好习惯<==
--------------------------------------------------------------
cp /etc/rsyncd.conf{,.ori}
--------------------------------------------------------------
==>执行以下命令<==
--------------------------------------------------------------
cat>/etc/rsyncd.conf<配置文件说明<==
--------------------------------------------------------------
cat>/etc/rsyncd.conf<注释
#created by oldboy ----------------------->注释
#site: http://www.oldboyedu.com ------->注释
uid = rsync ------------------------------->管理备份目录的用户
gid = rsync ------------------------------>管理备份目录的用户组
use chroot = no ----------------------->安全功能,数据是否锁定到备份目录
fake super = yes ---------------------->使用时,是否不需要切换root用户
max connections = 200 -------------->并发连接,同时多少客户端访问
timeout = 600 ----------------------->超时时间
pid file = /var/run/rsyncd.pid ------->进程号所在文件
lock file = /var/run/rsync.lock ------->锁文件
log file = /var/log/rsyncd.log --------->日志文件,查看报错等
ignore errors ---------------------------->忽略错误
read only = false ------------------------>可写
list = false -------------------------------->不允许列表
hosts allow = 172.16.1.0/24 -------------->哪些主机可以访问(和下面二选一)
hosts deny = 0.0.0.0/32 ------------------->哪些主机不可以访问(和上面二选一)
auth users = rsync_backup ---------------->远程虚拟连接用户
secrets file = /etc/rsync.password --------->存放密码的文件,格式:用户名:密码 ;权限:必须600
[backup] --------------------------------------->【模块名】远程访问使用模块名访问
comment = welcome to oldboyedu backup!-->注释
path = /backup/ --------------------------------->服务端用户备份的目录,用户和组为:rsync rsync
EOF
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# cat>/etc/rsyncd.conf< #rsync_config_______________start
> #created by oldboy
> #site: http://www.oldboyedu.com
> uid = rsync
> gid = rsync
> use chroot = no
> fake super = yes
> max connections = 200
> timeout = 600
> pid file = /var/run/rsyncd.pid
> lock file = /var/run/rsync.lock
> log file = /var/log/rsyncd.log
> ignore errors
> read only = false
> list = false
> hosts allow = 172.16.1.0/24
> hosts deny = 0.0.0.0/32
> auth users = rsync_backup
> secrets file = /etc/rsync.password
> [backup]
> comment = welcome to oldboyedu backup!
> path = /backup/
> EOF
--------------------------------------------------------------
==>查看内容如下<==
--------------------------------------------------------------
[root@backup ~]# cat /etc/rsyncd.conf
#rsync_config_______________start
#created by oldboy
#site: http://www.oldboyedu.com
uid = rsync
gid = rsync
use chroot = no
fake super = yes
max connections = 200
timeout = 600
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup]
comment = welcome to oldboyedu backup!
path = /backup/
参考资料
man rsync
查命令的参数
man rsyncd.conf
查配置参数
https://www.samba.org/ftp/rsync/rsync.html
3)创建用户和备份目录
--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
useradd rsync
id rsync
mkdir -p /backup
chown -R rsync.rsync /backup/
ls -ld /backup/
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# useradd rsync
[root@backup ~]# id rsync
uid=1001(rsync) gid=1001(rsync) 组=1001(rsync)
[root@backup ~]# mkdir -p /backup
[root@backup ~]# chown -R rsync.rsync /backup/
[root@backup ~]# ls -ld /backup/
drwxr-xr-x 2 rsync rsync 6 4月 15 12:12 /backup/
4)启动和检查
--------------------------------------------------------------
==>CentOS 6及以前启动方法<==
--------------------------------------------------------------
rsync --daemon
--------------------------------------------------------------
==>CentOS7 启动方法<==
--------------------------------------------------------------
systemctl start rsyncd
--------------------------------------------------------------
==>设置开机自启动<==
--------------------------------------------------------------
systemctl enable rsyncd
--------------------------------------------------------------
==>查看状态<==
--------------------------------------------------------------
systemctl status rsyncd
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# systemctl start rsyncd
[root@backup ~]# systemctl enable rsyncd
[root@backup ~]# systemctl status rsyncd
● rsyncd.service - fast remote file copy program daemon
Loaded: loaded (/usr/lib/systemd/system/rsyncd.service; enabled; vendor preset: disabled)
Active: active (running) since 一 2019-04-15 12:21:29 CST; 3h 15min ago
Main PID: 22814 (rsync)
CGroup: /system.slice/rsyncd.service
└─22814 /usr/bin/rsync --daemon --no-detach
4月 15 12:21:29 backup systemd[1]: Started fast remote file copy program daemon.
--------------------------------------------------------------
==>检查进程<==
--------------------------------------------------------------
[root@backup ~]# ps -ef|grep sync|grep -v grep
root 7521 1 0 11:39 ? 00:00:00 /usr/bin/rsync --daemon --no-detach
--------------------------------------------------------------
==>检查端口<==
--------------------------------------------------------------
[root@backup ~]# netstat -lntup|grep 873
tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 7521/rsync
tcp6 0 0 :::873 :::* LISTEN 7521/rsync
--------------------------------------------------------------
==>检查端口<==
--------------------------------------------------------------
[root@backup ~]# lsof -i :873
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsync 7521 root 3u IPv4 41439 0t0 TCP *:rsync (LISTEN)
rsync 7521 root 5u IPv6 41440 0t0 TCP *:rsync (LISTEN)
面试题:如何查看某端口对应什么服务?
答:netstat -lntup|grep 端口号
或者lsof -i :端口号
5)配置密码文件
--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
echo "rsync_backup:oldboy" > /etc/rsync.password
chmod 600 /etc/rsync.password
cat /etc/rsync.password
ls -l /etc/rsync.password
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# echo "rsync_backup:oldboy" > /etc/rsync.password
[root@backup ~]# chmod 600 /etc/rsync.password
[root@backup ~]# cat /etc/rsync.password
rsync_backup:oldboy
[root@backup ~]# ls -l /etc/rsync.password
-rw------- 1 root root 20 4月 15 11:51 /etc/rsync.password
至此,
rsync
服务端配置完成。
(2)以下操作都是在客户端服务器
以下方法2选1(推荐第二种)
方法1:认证密码文件(后面还需要输入参数,比较长)
--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
echo "oldboy" > /etc/rsync.password
chmod 600 /etc/rsync.password
cat /etc/rsync.password
ls -l /etc/rsync.password
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@nfs01 ~]# echo "oldboy" > /etc/rsync.password
[root@nfs01 ~]# chmod 600 /etc/rsync.password
[root@nfs01 ~]# cat /etc/rsync.password
oldboy
[root@nfs01 ~]# ls -l /etc/rsync.password
-rw------- 1 root root 7 4月 15 11:55 /etc/rsync.password
方法2:(后面不需要输入密码)
--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
echo ' export RSYNC_PASSWORD=oldboy' >>/etc/bashrc
tail -1 /etc/bashrc
. /etc/bashrc
echo $RSYNC_PASSWORD
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@nfs01 ~]# echo ' export RSYNC_PASSWORD=oldboy' >>/etc/bashrc
[root@nfs01 ~]# tail -1 /etc/bashrc
export RSYNC_PASSWORD=oldboy
[root@nfs01 ~]# . /etc/bashrc
[root@nfs01 ~]# echo $RSYNC_PASSWORD
oldboy
至此,
nfs01
的rsync
客户端配置完成
(3)守护进程模式,客户端rsync的命令语法:
配置服务器端守护进程,实现数据传输
1、服务器端守护进程。
2、客户端执行命令。
1)、
pull
:(拉)从远端拉取到本地。语法1(常用):(注意:与之前的语法有所变化)
rsync
[OPTION...] [USER@]HOST::SRC... [DEST]
rsync
[参数选项] [虚拟用户]@[主机地址]::[模块名] [本地路径]语法2:
rsync
[OPTION...] rsync://[USER@]HOST::SRC... [DEST]
rsync
[参数选项] rsync://[虚拟用户]@[主机地址]/[模块名] [本地路径]2)、
push
:(推)从本地推到远端。语法1(常用):
rsync
[OPTION...] [DEST] [USER@]HOST::SRC...
rsync
[参数选项] [本地路径] [虚拟用户]@[主机地址]::[模块名]
语法2:
rsync [OPTION...] [DEST] rsync://[USER@]HOST::SRC...
rsync [参数选项] [本地路径] rsync://[虚拟用户]@[主机地址]/[模块名]
(4)测试成果
以下在实际操作过程报错
错误1:
[root@nfs01 ~]# rsync -avz /etc/hosts [email protected]::backup --password-file=/etc/rsync.password
@ERROR: invalid uid rsync
rsync error: error starting client-server protocol (code 5) at main.c(1648) [sender=3.1.2]
解答:原因是在配置服务端的时候未创造
rsync
用户导致的
[root@backup ~]# useradd rsync
[root@backup ~]# id rsync
uid=1001(rsync) gid=1001(rsync) 组=1001(rsync)
[root@backup ~]# mkdir -p /backup
[root@backup ~]# chown -R rsync.rsync /backup/
[root@backup ~]# ls -ld /backup/
drwxr-xr-x 2 rsync rsync 6 4月 15 12:12 /backup/
错误2:
[root@nfs01 ~]# rsync -avz /etc/hosts [email protected]::backup --password-file=/etc/rsync.password
sending incremental file list
hosts
rsync: chgrp ".hosts.YDuTjO" (in backup) failed: Operation not permitted (1)
sent 223 bytes received 124 bytes 694.00 bytes/sec
total size is 332 speedup is 0.96
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
解答:增加如下参数到
/etc/rsyncd.conf
fake super = yes #不用root用户也可以存储文件的完整属性。
This allows the full attributes of a file to be stored without having to have the daemon actually running as root.
[root@backup ~]# grep fake /etc/rsyncd.conf
fake super = yes
改完配置,要重启服务:
[root@backup ~]# systemctl restart rsyncd
然后再测试:成功
--------------------------------------------------------------
==>客户端操作以下命令<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc/hosts [email protected]::backup --password-file=/etc/rsync.password
sending incremental file list
hosts
sent 89 bytes received 49 bytes 276.00 bytes/sec
total size is 332 speedup is 2.41
[root@nfs01 ~]# rsync -avz /etc [email protected]::backup --password-file=/etc/rsync.password
--------------------------------------------------------------
==>服务端检查<==
--------------------------------------------------------------
[root@backup ~]# ls /backup/
etc hosts
注意: 如果客户端按照环境变量的方式配置,则可以忽略
--password-file=/etc/rsync.password
参数。(前面也提到了,就是这种情况)
--------------------------------------------------------------
==>不需要参数和密码也可以<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc [email protected]::backup
sending incremental file list
sent 52,071 bytes received 644 bytes 105,430.00 bytes/sec
total size is 31,244,350 speedup is 592.70
测试增量
[root@nfs01 ~]# touch /etc/oldboy.txt
[root@nfs01 ~]# rsync -avz /etc [email protected]::backup
sending incremental file list
etc/oldboy.txt
sent 52,098 bytes received 655 bytes 105,506.00 bytes/sec
total size is 31,244,350 speedup is 592.28
第二种语法测试
[root@nfs01 ~]# rsync -avz /etc rsync://[email protected]/backup
sending incremental file list
sent 52,070 bytes received 644 bytes 105,428.00 bytes/sec
total size is 31,244,350 speedup is 592.71