pg备份还原工具--barman
一、系统
[root@barman ~]# cat /etc/issue CentOS release 6.5 (Final) Kernel \r on an \m [root@barman ~]# uname -a Linux barman 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/hosts 192.168.100.146 barman 192.168.100.147 pg
二、安装
2.1 简介
Barman (backup and recovery manager) is an administration tool for disaster recovery of PostgreSQL servers written in Python. Barman can perform remote backups of multiple servers in business critical environments, and helps DBAs during the recovery phase.
Barman’s most wanted features include: backup catalogues, retention policies, remote recovery, archiving and compression of WAL files and of backups. Barman is written and maintained by PostgreSQL professionals 2ndQuadrant.
Barman的版本更新情况：
2.2 安装依赖包
已安装软件版本检查：
[root@barman ~]# python --version Python 2.6.6 [root@barman ~]# rpm -qa | grep rsync rsync-3.0.6-9.el6_4.1.x86_64
下载地址：http://dl.fedoraproject.org/pub/epel/6/x86_64/
[root@barman ~]# rpm -ivh python-argparse-1.2.1-2.el6.noarch.rpm warning: python-argparse-1.2.1-2.el6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:python-argparse ########################################### [100%] [root@barman ~]# rpm -ivh python-argcomplete-0.6.7-2.el6.noarch.rpm warning: python-argcomplete-0.6.7-2.el6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:python-argcomplete ########################################### [100%] [root@barman ~]# rpm -ivh python-argh-0.23.2-1.el6.noarch.rpm warning: python-argh-0.23.2-1.el6.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:python-argh ########################################### [100%]
[root@barman ~]# yum install python-dateutil python-psycopg2
2.3 安装barman
下载地址：http://sourceforge.net/projects/pgbarman/files/1.3.0/
[root@barman ~]# rpm -ivh barman-1.3.0-1.rhel6.noarch.rpm Preparing... ########################################### [100%] 1:barman ########################################### [100%]
三、安装配置数据库
3.1 安装
在barman、pg上分别安装pg数据库，安装过程略
在barman上安装PostgreSQL仅为用于客户端，确保barman能够连上需要备份的pg
3.2 创建用户
在barman上创建barman用户并设置密码：
[root@barman ~]# useradd barman [root@barman ~]# passwd barman
在pg上创建postgres用户并设置密码：
[root@pg ~]# useradd postgres [root@pg ~]# passwd postgres
配置barman/postgres用户环境变量：
export PATH=/opt/pgsql/bin:$PATH:$HOME/bin export PGDATA=/opt/pgsql/data export PGUSER=postgres export PGPORT=5432 export LD_LIBRARY_PATH=/opt/pgsql/lib:$LD_LIBRARY_PATH
3.3 配置
[postgres@pg data]$ vim postgresql.conf listen_addresses = '*' port = 5432 wal_level = archive 若为HOT STANDBY环境则此处设置为hot_standby archive_mode = on archive_command = 'rsync -a %p barman@barman:/backup/pg1/incoming/%f' [postgres@pg data]$ vim pg_hba.conf 添加如下一条 host all all 192.168.100.146/32 trust
四、配置barman
4.1 创建归档目录
[root@barman ~]# mkdir -p /backup/pg1/incoming [root@barman ~]# chown barman:barman /backup -R
4.2 配置ssh互信
在barman上设置：
[barman@barman ~]$ ssh-keygen -t rsa [barman@barman ~]$ ssh-copy-id -i .ssh/id_rsa.pub postgres@pg [barman@barman ~]$ ssh postgres@pg date Thu Apr 10 02:11:43 CST 2014
在pg上设置：
[postgres@pg ~]$ ssh-keygen -t rsa [postgres@pg ~]$ ssh-copy-id -i .ssh/id_rsa.pub barman@barman [postgres@pg ~]$ ssh barman@barman date Thu Apr 10 02:14:01 CST 2014
配置完成后启动数据库：
[postgres@pg ~]$ pg_ctl start server starting
测试：
[barman@barman ~]$ psql -c 'SELECT version()' -h pg -U postgres version -------------------------------------------------------------------------------------------------------------- PostgreSQL 9.3.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4), 64-bit (1 row)
4.3 barman.conf设置
（设置/etc/barman.conf或～/.barman.conf）
[root@barman ~]# vi /etc/barman.conf [barman] barman_home = /var/lib/barman 备份文件路径 barman_user = barman log_file = /var/log/barman/barman.log compression = gzip [main] description = "Main PostgreSQL Database" incoming_wals_directory = /backup/pg1/incoming ssh_command = ssh postgres@pg conninfo = host=pg user=postgres
4.4 主要参数说明
immediate_checkpoint
1.3.0加入了该参数，当设置为true时，barman在执行备份时不需要再像之前一样等待PostgreSQL服务器发生一个checkpoint，而是主动触发一次checkpoint。默认设置为false。 该参数也可即设即用，可在执行barman backup时加上--immediate-checkpoint / --no-immediate-checkpoint。
bandwidth_limit / tablespace_bandwidth_limit
用于限制带宽使用，避免备份过多占用带宽资源。默认设置为0,即不限制。bandwidth_limit用作全局带宽限制，也可通过tablespace_bandwidth_limit对每个表空间所用带宽做特定限制。
network_compression = true|false
设定是否启用网络数据压缩。默认为false。
minimum_redundancy
最小冗余设置。当备份集个数减少到该值时，剩下的备份集将禁止删除，避免误删了所有的备份。
例如：当设置为1时，
[barman@barman ~]$ barman list-backup main main 20140411T015540 - Fri Apr 11 01:55:45 2014 - Size: 19.5 MiB - WAL Size: 82.8 KiB [barman@barman ~]$ barman delete main 20140411T015540 Skipping delete of backup 20140411T015540 for server main due to minimum redundancy requirements (1) [barman@barman ~]$ barman list-backup main main 20140411T015540 - Fri Apr 11 01:55:45 2014 - Size: 19.5 MiB - WAL Size: 82.8 KiB
保留策略
retention_policy：指定备份集的保留策略
wal_retention_policy：指定归档的保留策略
retention_policy_mode：策略的执行模式，目前只有auto
保留策略的执行为自动或手动，目前只实现了自动，手动将在之后版本中提供。自动根据保留策略对无用的备份或归档进行清理是通过barman cron。
五、barman管理
[barman@barman ~]$ barman --help
usage: barman [-h] [-v] [-c CONFIG] [-q] [-f {console}]
{status,list-server,list-files,list-backup,show-backup,cron,rebuild-xlogdb,recover,backup,check,show-server,delete}
...
positional arguments:
{status,list-server,list-files,list-backup,show-backup,cron,rebuild-xlogdb,recover,backup,check,show-server,delete}
cron Run maintenance tasks
list-server List available servers, with useful information
show-server Show all configuration parameters for the specified
servers
status Shows live information and status of the PostgreSQL
server
check Check if the server configuration is working. This
command returns success if every checks pass, or
failure if any of these fails
backup Perform a full backup for the given server
list-backup List available backups for the given server (supports
'all')
show-backup This method shows a single backup information
list-files List all the files for a single backup
recover Recover a server at a given time or xid
delete Delete a backup
rebuild-xlogdb Rebuild the WAL file database guessing it from the
disk content.
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-c CONFIG, --config CONFIG
uses a configuration file (defaults: ~/.barman.conf,
/etc/barman.conf, /etc/barman/barman.conf)
-q, --quiet be quiet (default: False)
-f {console}, --format {console}
output format (default: console)
5.1 检查备份信息
5.1.1 配置检查
[barman@barman ~]$ barman show-server main Server main: conninfo: host=pg user=postgres ssh_command: ssh postgres@pg retention_policy: None archive_mode: None minimum_redundancy: 0 custom_decompression_filter: None compression: gzip retention_policy_mode: auto archive_command: None basebackups_directory: /var/lib/barman/main/base data_directory: None tablespace_bandwidth_limit: None lock_file: /var/lib/barman/main/main.lock current_xlog: None description: Main PostgreSQL Database bandwidth_limit: None custom_compression_filter: None incoming_wals_directory: /backup/pg1/incoming immediate_checkpoint: False backup_directory: /var/lib/barman/main active: True network_compression: False post_backup_script: None wals_directory: /var/lib/barman/main/wals last_shipped_wal: None server_txt_version: None wal_retention_policy: main pre_backup_script: None
5.1.2 状态检查
[barman@barman ~]$ barman check main Server main: ssh: OK PostgreSQL: OK archive_mode: OK archive_command: OK directories: OK retention policy settings: OK compression settings: OK minimum redundancy requirements: OK (have 0 backups, expected at least 0)
[barman@barman ~]$ barman status main Server main: Description: Main PostgreSQL Database PostgreSQL version: 9.3.4 PostgreSQL Data directory: /opt/pgsql/data PostgreSQL 'archive_command' setting: rsync -a %p barman@barman:/backup/pg1/incoming/%f Archive status: last shipped WAL segment 000000010000000000000008 Current WAL segment: 000000010000000000000009 Retention policies: not enforced No. of available backups: 3 First available backup: 20140410T031606 Last available backup: 20140410T032409 Minimum redundancy requirements: satisfied (3/0)
5.2 查看可监听的数据库服务器
[barman@barman ~]$ barman list-server main - Main PostgreSQL Database
5.3 查看指定备份集的详细信息
[barman@barman ~]$ barman show-backup main 20140410T031606 Backup 20140410T031606: Server Name : main Status : DONE PostgreSQL Version: 90304 PGDATA directory : /opt/pgsql/data Base backup information: Disk usage : 19.5 MiB Timeline : 1 Begin WAL : 000000010000000000000005 End WAL : 000000010000000000000005 WAL number : 1 Begin time : 2014-04-10 03:16:06.343719 End time : 2014-04-10 03:16:11.085018 Begin Offset : 40 End Offset : 184 Begin XLOG : 0/5000028 End XLOG : 0/50000B8 WAL information: No of files : 1 Disk usage : 16.0 KiB Last available : 000000010000000000000006 Catalog information: Retention Policy: not enforced Previous Backup : - (this is the oldest base backup) Next Backup : 20140410T031738
5.4 执行全备
[barman@barman ~]$ barman backup main --若想对多个库同时进行备份，可执行barman backup all Starting backup for server main in /var/lib/barman/main/base/20140410T031606 Backup start at xlog location: 0/5000028 (000000010000000000000005, 00000028) Copying files. Copy done. Asking PostgreSQL server to finalize the backup. Backup end at xlog location: 0/50000B8 (000000010000000000000005, 000000B8) Backup completed
基础备份：
[barman@barman ~]$ ls /var/lib/barman/main/base/ 20140410T031606
wal备份：
[barman@barman ~]$ ls /var/lib/barman/main/wals/ 0000000100000000 xlog.db xlog.db.lock
5.5 查看备份集
[barman@barman ~]$ barman list-backup main main 20140410T032409 - Thu Apr 10 03:24:13 2014 - Size: 19.5 MiB - WAL Size: 0 B main 20140410T031738 - Thu Apr 10 03:17:41 2014 - Size: 19.5 MiB - WAL Size: 0 B main 20140410T031606 - Thu Apr 10 03:16:11 2014 - Size: 19.5 MiB - WAL Size: 16.0 KiB
5.6 恢复
确保用户在指定的恢复路径中有权限创建PGDATA目录
barman recover --help usage: barman recover [-h] [--target-tli TARGET_TLI] [--target-time TARGET_TIME] [--target-xid TARGET_XID] [--target-name TARGET_NAME] [--exclusive] [--tablespace NAME:LOCATION] [--remote-ssh-command SSH_COMMAND] server_name backup_id destination_directory Recover a server at a given time or xid positional arguments: server_name specifies the server name for the command backup_id specifies the backup ID to recover destination_directory the directory where the new server is created optional arguments: -h, --help show this help message and exit --target-tli TARGET_TLI target timeline (default: None) --target-time TARGET_TIME target time. You can use any valid unambiguous representation. e.g: "YYYY-MM-DD HH:MM:SS.mmm" (default: None) --target-xid TARGET_XID target transaction ID (default: None) --target-name TARGET_NAME target name created previously with pg_create_restore_point() function call (default: None) --exclusive set target xid to be non inclusive (default: False) --tablespace NAME:LOCATION tablespace relocation rule (default: None) --remote-ssh-command SSH_COMMAND This options activates remote recovery, by specifying the secure shell command to be launched on a remote host. It is the equivalent of the "ssh_command" serveroption in the configuration file for remote recovery. Example: "ssh postgres@db2" (default: None)
5.6.1 本地恢复
[barman@barman ~]$ barman recover main 20140410T032409 /tmp/data Starting local restore for server main using backup 20140410T032409 Destination directory: /tmp/data Copying the base backup. Copying required wal segments. The archive_command was set to 'false' to prevent data losses. Your PostgreSQL server has been successfully prepared for recovery! Please review network and archive related settings in the PostgreSQL configuration file before starting the just recovered instance.
5.6.2 远程恢复
[barman@barman ~]$ barman recover --remote-ssh-command "ssh postgres@pg" main 20140410T032409 /tmp/data Starting remote restore for server main using backup 20140410T032409 Destination directory: /tmp/data Copying the base backup. Copying required wal segments. The archive_command was set to 'false' to prevent data losses. Your PostgreSQL server has been successfully prepared for recovery! Please review network and archive related settings in the PostgreSQL configuration file before starting the just recovered instance.
5.6.3 迁移表空间
测试未成功
5.6.4 PITR
[barman@barman ~]$ barman backup main [barman@barman ~]$ barman list-backup main main 20140410T071548 - Thu Apr 10 07:15:53 2014 - Size: 19.5 MiB - WAL Size: 0 B postgres=# create table t1(id int); CREATE TABLE [barman@barman ~]$ barman backup main [barman@barman ~]$ barman list-backup main main 20140410T072226 - Thu Apr 10 07:22:32 2014 - Size: 19.5 MiB - WAL Size: 0 B main 20140410T071548 - Thu Apr 10 07:15:53 2014 - Size: 19.5 MiB - WAL Size: 0 B
5.6.4.1 target-time
[barman@barman ~]$ barman recover --target-time "2014-04-10 07:16:00" main 20140410T071548 /opt/pgsql/data Starting local restore for server main using backup 20140410T071548 Destination directory: /opt/pgsql/data Doing PITR. Recovery target time: '2014-04-10 07:16:00' Copying the base backup. Copying required wal segments. Generating recovery.conf The archive_command was set to 'false' to prevent data losses. Your PostgreSQL server has been successfully prepared for recovery! Please review network and archive related settings in the PostgreSQL configuration file before starting the just recovered instance. [barman@barman ~]$ pg_ctl start server starting postgres=# \d No relations found.
5.6.4.2 target-xid
[barman@barman ~]$ barman backup main [barman@barman ~]$ barman list-backup main main 20140411T015540 - Fri Apr 11 01:55:45 2014 - Size: 19.5 MiB - WAL Size: 0 B postgres=# create table t1(id int); CREATE TABLE postgres=# select txid_current(); txid_current -------------- 1824 (1 row) postgres=# create table t2(id int); CREATE TABLE postgres=# select txid_current(); txid_current -------------- 1826 (1 row) postgres=# create table t3(id int); CREATE TABLE postgres=# select txid_current(); txid_current -------------- 1828 (1 row) postgres=# checkpoint ; CHECKPOINT postgres=# select pg_switch_xlog(); pg_switch_xlog ---------------- 0/18016390 (1 row)
包含所指事务
恢复：
[barman@barman ~]$ barman recover --target-xid "1825" main 20140411T015540 /opt/pgsql/data
验证：
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+------+-------+----------
public | t1 | table | postgres
public | t2 | table | postgres
(2 rows)
{说明包括了1825事务}
不包含所指事务
恢复：
[barman@barman ~]$ barman recover --target-xid "1825" --exclusive main 20140411T015540 /opt/pgsql/data
验证：
postgres=# \d List of relations Schema | Name | Type | Owner --------+------+-------+---------- public | t1 | table | postgres (1 row)
5.6.4.3 target-name
接着上一部分执行：
postgres=# \d
List of relations
Schema | Name | Type | Owner
--------+------+-------+----------
public | t1 | table | postgres
public | t2 | table | postgres
public | t3 | table | postgres
(3 rows)
postgres=# create table t4(id int);
CREATE TABLE
postgres=# select pg_create_restore_point('target1');
pg_create_restore_point
-------------------------
0/19015098
(1 row)
postgres=# create table t5(id int);
CREATE TABLE
postgres=# select pg_create_restore_point('target2');
pg_create_restore_point
-------------------------
0/19016170
(1 row)
postgres=# create table t6(id int);
CREATE TABLE
postgres=# select pg_create_restore_point('target3');
pg_create_restore_point
-------------------------
0/19019160
(1 row)
postgres=# select pg_switch_xlog();
pg_switch_xlog
----------------
0/19019180
(1 row)
恢复：
[barman@barman ~]$ barman recover --target-name "target1" main 20140411T015540 /opt/pgsql/data [barman@barman pgsql]$ cat data/recovery.conf restore_command = 'cp barman_xlog/%f %p' recovery_end_command = 'rm -fr barman_xlog' recovery_target_name = 'target1' [barman@barman pgsql]$ pg_ctl start server starting
验证：
postgres=# \d List of relations Schema | Name | Type | Owner --------+------+-------+---------- public | t1 | table | postgres public | t2 | table | postgres public | t3 | table | postgres public | t4 | table | postgres (4 rows)
5.6.4.4 target-tli
用于恢复时间线的设置
5.7 删除备份集
[barman@barman ~]$ barman delete main 20140410T032409 Deleting backup 20140410T032409 for server main Done
5.8 快捷的备份集ID代替（latest/oldest）
适用：
show-backup | delete | list-files | recover
例如：
barman show-backup main latest
5.9 重新生成归档
[barman@barman ~]$ barman rebuild-xlogdb main Rebuilding xlogdb for server main Done rebuilding xlogdb for server main (history: 0, backup_labels: 1, wal_file: 4)
5.10 列举备份文件
[barman@barman ~]$ barman list-files main 20140411T015540 /var/lib/barman/main/base/20140411T015540/backup.info /var/lib/barman/main/base/20140411T015540/pgdata/pg_hba.conf /var/lib/barman/main/base/20140411T015540/pgdata/postmaster.opts /var/lib/barman/main/base/20140411T015540/pgdata/pg_ident.conf /var/lib/barman/main/base/20140411T015540/pgdata/postgresql.conf /var/lib/barman/main/base/20140411T015540/pgdata/postgresql.conf.origin /var/lib/barman/main/base/20140411T015540/pgdata/backup_label.old /var/lib/barman/main/base/20140411T015540/pgdata/backup_label /var/lib/barman/main/base/20140411T015540/pgdata/PG_VERSION /var/lib/barman/main/base/20140411T015540/pgdata/pg_clog/0000 /var/lib/barman/main/base/20140411T015540/pgdata/pg_notify/0000 /var/lib/barman/main/base/20140411T015540/pgdata/base/12891/12733 /var/lib/barman/main/base/20140411T015540/pgdata/base/12891/12707_fsm /var/lib/barman/main/base/20140411T015540/pgdata/base/12891/12718_fsm /var/lib/barman/main/base/20140411T015540/pgdata/base/12891/12702 /var/lib/barman/main/base/20140411T015540/pgdata/base/12891/12848 /var/lib/barman/main/base/20140411T015540/pgdata/base/12891/12707 ……
同时可指定需要列出的文件类型，如：
[barman@barman ~]$ barman list-files --target data main 20140411T015540
--target 值为data/standalone/wal/full，含义如下：
data: lists just the data files;
standalone: lists the base backup files, including required WAL files;
wal: lists all WAL files from the beginning of the base backup to the start of the following one (or until the end of the log);
full: same as data + wal.
5.11 附加执行脚本
barman支持在备份之前或之后/归档之前或之后附带执行定制脚本，关于该脚本在barman.conf有相关设置：
; Pre/post backup hook scripts
;pre_backup_script = env | grep ^BARMAN
;post_backup_script = env | grep ^BARMAN
; Pre/post archive hook scripts
;pre_archive_script = env | grep ^BARMAN
;post_archive_script = env | grep ^BARMAN
六、总结
barman功能特点总结如下：
使用简单，通过简单的命令就可实现所有功能;
支持本地/远程备份恢复;
能够同时管理多个备份端，并能同时对多个备份端进行操控;
支持在线全备、wal增量备份;
支持多种压缩功能，如：gzip、bzip2及用户自定义压缩工具;
提供灵活多变的备份、归档保留策略;
支持流量压缩，可定义到表空间;
支持迁移表空间（未测）;
支持自动维护;
支持PITR。
七、参考文献
官方网站：
http://www.pgbarman.org/
http://2ndquadrant.com/
八、license
Barman is the exclusive property of 2ndQuadrant Italia and its code is distributed under GNU General Public License 3.
Contributions to Barman are welcome, and will be listed in the file AUTHORS.
2ndQuadrant Italia requires that any contributions provide a copyright assignment and a disclaimer of any work-for-hire ownership claims from the employer of the developer.
This lets us make sure that all of the Barman distribution remains free code.