Pg异步复制系统Bucardo
一、系统
# cat /etc/issue CentOS release 6.5 (Final) Kernel \r on an \m # uname -a Linux barman 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
二、安装
2.1 简介
Bucardo is an asynchronous PostgreSQL replication system, allowing for both multi-master and multi-slave operations. It was developed at Backcountry.com by Jon Jensen and Greg Sabino Mullane of End Point Corporation, and is now in use at many other organizations. Bucardo is free and open source software released under the BSD license. Bucardo is a replication system for Postgres that provides both master-master and master-slave capabilities. It is asynchronous and trigger based. Its primary goals are to provide master-master replication for load balancing and failover, and to provide load balancing and data warehousing via master-slave replication.
2.2 安装依赖包
版本要求： Perl 5 (at least 5.8.3) Postgres (at least 8.2) DBI module (at least 1.51) DBD::Pg module (at least 2.0.0) DBIx::Safe module (at least 1.2.4)
2.2.1 DBIx::Safe
需要首先安装DBIx::Safe
（1）通过yum在线安装 # wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm # rpm -ivh epel-release-6-8.noarch.rpm warning: epel-release-6-8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY Preparing... ########################################### [100%] 1:epel-release ########################################### [100%] # yum install perl-DBIx-Safe # rpm -qa | grep perl-DBIx-Safe perl-DBIx-Safe-1.2.5-6.el6.noarch （2）源码编译安装 下载地址：http://bucardo.org/wiki/Bucardo tar xvfz dbix_safe.tar.gz cd DBIx-Safe-1.2.5 perl Makefile.PL make make test make install
2.2.2 DBD::Pg
# yum install perl-DBD-Pg # rpm -qa | grep perl-DBD-Pg perl-DBD-Pg-2.15.1-4.el6_3.x86_64
2.2.3 Test::Simple
# yum install perl-Test-Simple # rpm -qa | grep perl-Test-Simple perl-Test-Simple-0.92-136.el6.x86_64
2.3 安装bucardo
下载最新版本，地址：https://github.com/bucardo/bucardo
# unzip bucardo-master.zip # cd bucardo-master # INSTALL_BUCARDODIR=/opt/bucardo perl Makefile.PL # make # make install
2.4 安装Pg
版本要求：Version >=8.2
# ./configure --prefix=/opt/pg93 --with-perl # make # make install $ initdb $ pg_ctl start
三、配置
3.1 安装Pl/PerlU
(之前安装Pg时已经加入了--with-perl则不需要再在此安装)
# yum install postgresql-plperl
3.2 初始化bucardo管理库
# mkdir -p /var/run/bucardo # chmod 777 /var/run/bucardo # su - postgres
切换到bucardo安装目录下，不然会报找不到bucardo.shema
$ cd /opt/bucardo/ $ ./bucardo install -U postgres -d postgres This will install the bucardo database into an existing Postgres cluster. Postgres must have been compiled with Perl support, and you must connect as a superuser Current connection settings: 1. Host: <none> 2. Port: 5432 3. User: postgres 4. Database: postgres 5. PID directory: /var/run/bucardo Enter a number to change it, P to proceed, or Q to quit: P Postgres version is: 9.3 Creating superuser 'bucardo' Attempting to create and populate the bucardo database and schema ERROR: role "bucardo" already exists STATEMENT: CREATE USER bucardo SUPERUSER; ERROR: language "plpgsql" already exists STATEMENT: CREATE LANGUAGE plpgsql; Database creation is complete Updated configuration setting "piddir" Installation is now complete. If you see errors or need help, please email [email protected] You may want to check over the configuration variables next, by running: ./bucardo show all Change any setting by using: ./bucardo set foo=bar
查看引入的对象：
$ psql bucardo bucardo psql (9.3.4) Type "help" for help. bucardo=# \d List of relations Schema | Name | Type | Owner ---------+-------------------------------+----------+--------- bucardo | bucardo_config | table | bucardo bucardo | bucardo_custom_trigger | table | bucardo bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo bucardo | bucardo_log_message | table | bucardo bucardo | bucardo_rate | table | bucardo bucardo | customcode | table | bucardo bucardo | customcode_id_seq | sequence | bucardo bucardo | customcode_map | table | bucardo bucardo | customcols | table | bucardo bucardo | customcols_id_seq | sequence | bucardo bucardo | customname | table | bucardo bucardo | customname_id_seq | sequence | bucardo bucardo | db | table | bucardo bucardo | db_connlog | table | bucardo bucardo | dbgroup | table | bucardo bucardo | dbmap | table | bucardo bucardo | dbrun | table | bucardo bucardo | goat | table | bucardo bucardo | goat_id_seq | sequence | bucardo bucardo | herd | table | bucardo bucardo | herdmap | table | bucardo bucardo | sync | table | bucardo bucardo | syncrun | table | bucardo bucardo | upgrade_log | table | bucardo (24 rows) bucardo=# \df List of functions Schema | Name | Result data type | Argument data types | Type ---------+----------------------------+------------------+---------------------+--------- bucardo | bucardo_delete_sync | trigger | | trigger bucardo | bucardo_log_message_notify | trigger | | trigger bucardo | bucardo_tablename_maker | text | text | normal bucardo | check_bucardo_config | trigger | | trigger bucardo | db_change | trigger | | trigger bucardo | db_getconn | text | text | normal bucardo | db_testconn | text | text | normal bucardo | find_unused_goats | SETOF text | | normal bucardo | herdcheck | trigger | | trigger bucardo | magic_update | text | | normal bucardo | plperlu_test | text | | normal bucardo | table_exists | boolean | text, text | normal bucardo | validate_all_syncs | integer | | normal bucardo | validate_all_syncs | integer | integer | normal bucardo | validate_goat | trigger | | trigger bucardo | validate_sync | trigger | | trigger bucardo | validate_sync | text | text | normal bucardo | validate_sync | text | text, integer | normal (18 rows) bucardo=# \dL List of languages Name | Owner | Trusted | Description ---------+----------+---------+------------------------------ plperlu | bucardo | f | plpgsql | postgres | t | PL/pgSQL procedural language (2 rows)
3.3 修改环境变量
# su - postgres $ vi .bash_profile export PATH=/opt/bucardo:/opt/pg93/bin:$PATH:$HOME/bin export PGDATA=/opt/pg93/data export LD_LIBRARY_PATH=/opt/pg93/lib
四、测试
4.1 Master-Slave
在本地创建两个测试库：
$ createdb db1 $ createdb db2
使用pgbench初始化产生测试数据：
$ pgbench -i db1 $ pgbench -i db2
添加复制数据库：
[postgres@bucardo ~]$ bucardo add db masterdb dbhost=127.0.0.1 dbuser=postgres dbport=5432 dbname=db1 Added database "masterdb" [postgres@bucardo ~]$ bucardo add db slavedb dbhost=127.0.0.1 dbuser=postgres dbport=5432 dbname=db2 Added database "slavedb"
添加后查看：
bucardo=# select * from db; name | dbtype | dbhost | dbport | dbname | dbuser | dbpass | dbconn | dbservice | pgpass | status | server_side_prepares | cda te ----------+----------+-----------+--------+--------+----------+--------+--------+-----------+--------+--------+----------------------+---------------- --------------- masterdb | postgres | 127.0.0.1 | 5432 | db1 | postgres | | | | | active | t | 2014-04-30 03:1 6:05.68223+08 slavedb | postgres | 127.0.0.1 | 5432 | db2 | postgres | | | | | active | t | 2014-04-30 03:1 6:16.158017+08 (2 rows) bucardo=# select * from db_connlog ; db | conndate | connstring | status | version ----------+-------------------------------+----------------------------------------------------------+--------+--------- masterdb | 2014-04-30 03:16:05.68223+08 | dbi:Pg:dbname=db1;host=127.0.0.1;port=5432 user=postgres | good | 90304 slavedb | 2014-04-30 03:16:16.158017+08 | dbi:Pg:dbname=db2;host=127.0.0.1;port=5432 user=postgres | good | 90304 (2 rows)
添加需要同步的表：
[postgres@bucardo ~]$ bucardo add all tables db=masterdb -T pgbench_history --herd=herd1 --verbose Creating relgroup: herd1 Added table public.pgbench_tellers to relgroup herd1 Added table public.pgbench_branches to relgroup herd1 Added table public.pgbench_accounts to relgroup herd1 New tables: public.pgbench_accounts public.pgbench_branches public.pgbench_tellers New tables added: 3
查看herd：
bucardo=# select * from herd; name | about | cdate -------+-------+------------------------------- herd1 | | 2014-04-30 03:17:53.433387+08 (1 rows)
查看herd映射记录：
bucardo=# select * from herdmap; herd | goat | priority | cdate -------+------+----------+------------------------------- herd1 | 1 | 0 | 2014-04-30 03:17:53.433387+08 herd1 | 2 | 0 | 2014-04-30 03:17:53.433387+08 herd1 | 3 | 0 | 2014-04-30 03:17:53.433387+08 (3 rows)
添加同步：
[postgres@bucardo ~]$ bucardo add sync sync1 relgroup=herd1 dbs=masterdb,slavedb Created database group "sync1" Added sync "sync1" [postgres@bucardo ~]$ bucardo list syncs Sync: sync1 Relgroup: herd1 DB group sync1: masterdb (source) slavedb (target) [Active] [postgres@bucardo ~]$ bucardo list dbs Database: masterdb Status: active Conn: psql -p 5432 -U postgres -d db1 -h 127.0.0.1 Database: slavedb Status: active Conn: psql -p 5432 -U postgres -d db2 -h 127.0.0.1 [postgres@bucardo ~]$ bucardo list herds[relgroups/herd/relgroup] Relgroup: herd1 DB: masterdb Members: public.pgbench_accounts, public.pgbench_branches, public.pgbench_tellers Used in syncs: sync1 Relgroup: herd2 DB: masterdb Members: public.pgbench_history [postgres@bucardo ~]$ bucardo list tables[table] 3. Table: public.pgbench_accounts DB: masterdb PK: aid (int4) 2. Table: public.pgbench_branches DB: masterdb PK: bid (int4) 4. Table: public.pgbench_history DB: masterdb PK: none 1. Table: public.pgbench_tellers DB: masterdb PK: tid (int4)
启动bucardo：
[postgres@bucardo bucardo]$ bucardo start Checking for existing processes Removing /var/run/bucardo/fullstopbucardo Starting Bucardo [postgres@bucardo bucardo]$ ps -ef | grep Bucardo | grep -v "grep" postgres 4745 1 0 06:03 ? 00:00:00 Bucardo Master Control Program v4.99.11. Active syncs: sync1 postgres 4751 4745 0 06:03 ? 00:00:00 Bucardo VAC. postgres 4754 4745 0 06:03 ? 00:00:00 Bucardo Controller. Sync "sync1" for relgroup "herd1" to dbs "sync1" postgres 4758 4754 0 06:03 ? 00:00:00 Bucardo Kid. Sync "sync1"
查看数据：
[postgres@bucardo ~]$ psql db1 -c "select * from pgbench_tellers where tid=10" tid | bid | tbalance | filler -----+-----+----------+-------- 10 | 1 | 0 | (1 row) [postgres@bucardo ~]$ psql db2 -c "select * from pgbench_tellers where tid=10" tid | bid | tbalance | filler -----+-----+----------+-------- 10 | 1 | 0 | (1 row)
更改masterdb中的数据：
$ psql db1 -c "update pgbench_tellers set bid=10 where tid=10" UPDATE 1
查看改后情况：
[postgres@bucardo ~]$ psql db1 -c "select * from pgbench_tellers where tid=10" tid | bid | tbalance | filler -----+-----+----------+-------- 10 | 10 | 0 | (1 row) [postgres@bucardo ~]$ psql db2 -c "select * from pgbench_tellers where tid=10" tid | bid | tbalance | filler -----+-----+----------+-------- 10 | 10 | 0 | (1 row)
在masterdb中删除数据：
[postgres@bucardo ~]$ psql db1 -c "delete from pgbench_tellers where tid=10" DELETE 1
查看删后情况：
[postgres@bucardo ~]$ psql db2 -c "select * from pgbench_tellers where tid=10" tid | bid | tbalance | filler -----+-----+----------+-------- (0 rows)
在masterdb中插入数据：
[postgres@bucardo ~]$ psql db1 -c "insert into pgbench_tellers values(10,1,0)" INSERT 0 1
查看插入后情况：
[postgres@bucardo ~]$ psql db2 -c "select * from pgbench_tellers where tid=10" tid | bid | tbalance | filler -----+-----+----------+-------- 10 | 1 | 0 | (1 row)
[postgres@bucardo ~]$ bucardo status PID of Bucardo MCP: 4745 Name State Last good Time Last I/D Last bad Time =======+========+============+========+===========+===========+======= sync1 | Good | 06:17:28 | 6m 15s | 0/1 | none | [postgres@bucardo ~]$ bucardo status sync1 ====================================================================== Last good : Apr 30, 2014 06:17:28 (time to run: 1s) Rows deleted/inserted : 0 / 1 Sync name : sync1 Current state : Good Source relgroup/database : herd1 / masterdb Tables in sync : 3 Status : Active Check time : None Overdue time : 00:00:00 Expired time : 00:00:00 Stayalive/Kidsalive : Yes / Yes Rebuild index : No Autokick : Yes Onetimecopy : No Post-copy analyze : Yes Last error: : ======================================================================
4.2 Master-Master
重新初始化一个库:
$ initdb $ pg_ctl start $ ./bucardo install -U postgres -d postgres
在本地创建两个测试库：
$ createdb db1 $ createdb db2
使用pgbench初始化产生测试数据：
$ pgbench -i db1 $ pgbench -i db2
添加复制数据库：
[postgres@bucardo ~]$ bucardo add db masterdb1 dbhost=127.0.0.1 dbuser=postgres dbport=5432 dbname=db1 Added database "masterdb1" [postgres@bucardo ~]$ bucardo add db masterdb2 dbhost=127.0.0.1 dbuser=postgres dbport=5432 dbname=db2 Added database "masterdb2"
添加需要同步的表：
[postgres@bucardo ~]$ bucardo add all tables db=masterdb1 -T pgbench_history --herd=herd1 --verbose Creating relgroup: herd1 Added table public.pgbench_tellers to relgroup herd1 Added table public.pgbench_branches to relgroup herd1 Added table public.pgbench_accounts to relgroup herd1 New tables: public.pgbench_accounts public.pgbench_branches public.pgbench_tellers New tables added: 3
添加同步：
[postgres@bucardo ~]$ bucardo add sync sync1 relgroup=herd1 dbs=masterdb1:source,masterdb2:source Created database group "sync1" Added sync "sync1"
启动bucardo：
[postgres@bucardo ~]$ cd /opt/bucardo/ [postgres@bucardo bucardo]$ bucardo start
验证同步：
db1=# select * from pgbench_tellers where tid=1; tid | bid | tbalance | filler -----+-----+----------+-------- 1 | 1 | 0 | (1 row) db1=# update pgbench_tellers set bid=2 where tid=1; UPDATE 1 db2=# select * from pgbench_tellers where tid=1; tid | bid | tbalance | filler -----+-----+----------+-------- 1 | 2 | 0 | (1 row) db2=# update pgbench_tellers set bid=1 where tid=1; UPDATE 1 db1=# select * from pgbench_tellers where tid=1; tid | bid | tbalance | filler -----+-----+----------+-------- 1 | 1 | 0 | (1 row)
五、研究
在规定复制表、复制方式后会生成一些辅助表、触发器、函数、序列、存储过程等。如下，为复制表创建了触发器：
db1=# \d pgbench_accounts
Table "public.pgbench_accounts"
Column | Type | Modifiers
----------+---------------+-----------
aid | integer | not null
bid | integer |
abalance | integer |
filler | character(84) |
Indexes:
"pgbench_accounts_pkey" PRIMARY KEY, btree (aid)
Triggers:
bucardo_delta AFTER INSERT OR DELETE OR UPDATE ON pgbench_accounts FOR EACH ROW EXECUTE PROCEDURE bucardo.delta_public_pgbench_accounts()
bucardo_note_trunc_sync1 AFTER TRUNCATE ON pgbench_accounts FOR EACH STATEMENT EXECUTE PROCEDURE bucardo.bucardo_note_truncation('sync1')
Triggers firing always:
bucardo_kick_sync1 AFTER INSERT OR DELETE OR UPDATE OR TRUNCATE ON pgbench_accounts FOR EACH STATEMENT EXECUTE PROCEDURE bucardo.bucardo_kick_sync1()
db1=# \df
List of functions
Schema | Name | Result data type | Argument data types | Type
---------+-------------------------------+------------------+---------------------+---------
bucardo | bucardo_compress_delta | SETOF text | | normal
bucardo | bucardo_compress_delta | text | oid | normal
bucardo | bucardo_compress_delta | text | text | normal
bucardo | bucardo_compress_delta | text | text, text | normal
bucardo | bucardo_delta_check | SETOF text | text, text | normal
bucardo | bucardo_delta_names_helper | trigger | | trigger
bucardo | bucardo_kick_sync1 | trigger | | trigger
bucardo | bucardo_note_truncation | trigger | | trigger
bucardo | bucardo_purge_delta | text | text | normal
bucardo | bucardo_purge_delta | text | text, text | normal
bucardo | bucardo_tablename_maker | text | text | normal
bucardo | bucardo_tablename_maker | text | text, text | normal
bucardo | delta_public_pgbench_accounts | trigger | | trigger
bucardo | delta_public_pgbench_branches | trigger | | trigger
bucardo | delta_public_pgbench_tellers | trigger | | trigger
(15 rows)
查看具体定义：
db1=# select proname,prosrc from pg_proc where proname='delta_public_pgbench_accounts'; proname | prosrc -------------------------------+--------------------------------------------------------------------------- delta_public_pgbench_accounts | + | BEGIN + | IF (TG_OP = 'INSERT') THEN + | INSERT INTO bucardo.delta_public_pgbench_accounts VALUES (NEW."aid"); + | ELSIF (TG_OP = 'UPDATE') THEN + | INSERT INTO bucardo.delta_public_pgbench_accounts VALUES (OLD."aid"); + | IF (OLD."aid" <> NEW."aid") THEN + | INSERT INTO bucardo.delta_public_pgbench_accounts VALUES (NEW."aid");+ | END IF; + | ELSE + | INSERT INTO bucardo.delta_public_pgbench_accounts VALUES (OLD."aid"); + | END IF; + | RETURN NULL; + | END; + | (1 row) db1=# select proname,prosrc from pg_proc where proname='bucardo_note_truncation'; proname | prosrc -------------------------+----------------------------------------------------------------------------------------- bucardo_note_truncation | + | DECLARE + | mytable TEXT; + | myst TEXT; + | BEGIN + | INSERT INTO bucardo.bucardo_truncate_trigger(tablename,sname,tname,sync) + | VALUES (TG_RELID, TG_TABLE_SCHEMA, TG_TABLE_NAME, TG_ARGV[0]); + | SELECT INTO mytable + | bucardo.bucardo_tablename_maker(TG_TABLE_SCHEMA || '.' || TG_TABLE_NAME, 'delta_');+ | myst = 'TRUNCATE TABLE bucardo.' || mytable; + | EXECUTE myst; + | SELECT INTO mytable + | bucardo.bucardo_tablename_maker(TG_TABLE_SCHEMA || '.' || TG_TABLE_NAME, 'track_');+ | myst = 'TRUNCATE TABLE bucardo.' || mytable; + | EXECUTE myst; + | -- Not strictly necessary, but nice to have a clean slate + | SELECT INTO mytable + | bucardo.bucardo_tablename_maker(TG_TABLE_SCHEMA || '.' || TG_TABLE_NAME, 'stage_');+ | myst = 'TRUNCATE TABLE bucardo.' || mytable; + | EXECUTE myst; + | RETURN NEW; + | END; + | (1 row) db1=# select proname,prosrc from pg_proc where proname='bucardo_kick_sync1'; proname | prosrc --------------------+------------------------------------------------------ bucardo_kick_sync1 | + | BEGIN + | EXECUTE $nn$NOTIFY bucardo, 'kick_sync_sync1'$nn$;+ | RETURN NEW; + | END; + | (1 row)
六、总结
1. bucardo不仅支持Master/Slaves模式，而且支持双主复制，即两个节点无论哪个节点发生变更都会同步到另一个节点上。同步冲突处理方式由用户指定。 2. 提供负载均衡。 3. Bucardo仅提供异步复制，也不支持ddl复制，因为其同步原理是基于触发器实现，类似Slony。 4. 多主复制仅支持两个Master，即双主复制。同时支持双主多备。(在5版本中已经解除了该限制，能够支持更多的主/备数据库) 5. 新版本支持不同数据库系统间的数据同步，如Drizzle, MariaDB, MongoDB, MySQL, Oracle, Redis, SQLite, as well as "flat files"。
七、参考文献
主页： Wiki： Github: https://github.com/bucardo/bucardo http://bucardo.org/wiki/Bucardo/add_sync http://justatheory.com/computers/databases/postgresql/bootstrap-bucardo-mulitmaster.html
八、license
BSD
九、问题
Q1
问题描述：
# /opt/bucardo/bucardo_ctl --help Can't locate Time/HiRes.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/bucardo/bucardo_ctl line 18. BEGIN failed--compilation aborted at /opt/bucardo/bucardo_ctl line 18.
解决方式：
yum install perl-devel perl-CPAN perl -MCPAN -e shell cpan[2]> install Time::HiRes cpan[3]> exit
Q2
问题：
# ./configure --prefix=/opt/pg93/ --with-perl checking for Perl privlibexp... /usr/share/perl5 checking for Perl useshrplib... true checking for flags to link embedded Perl... Can't locate ExtUtils/Embed.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .). BEGIN failed--compilation aborted. no configure: error: could not determine flags for linking embedded Perl. This probably means that ExtUtils::Embed or ExtUtils::MakeMaker is not
解决方式：
# yum install perl-ExtUtils-Embed
Q3
问题描述：
$ ./bucardo install Can't locate Encode/Locale.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./bucardo line 24. BEGIN failed--compilation aborted at ./bucardo line 24.
解决方式：
# yum install perl-Encode-Locale
Q4
问题：
[postgres@bucardo ~]$ bucardo start Checking for existing processes Can't locate Bucardo.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/bucardo/bucardo line 767.
解决方式：
到/opt/bucardo目录下再执行start
Q5
问题：
[postgres@bucardo bucardo]$ bucardo start Checking for existing processes Can't locate boolean.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Bucardo.pm line 34. BEGIN failed--compilation aborted at Bucardo.pm line 34. Compilation failed in require at /opt/bucardo/bucardo line 767.
解决方式：
# yum install perl-boolean
Q6
问题：
[postgres@bucardo bucardo]$ bucardo start Checking for existing processes Removing /var/run/bucardo/fullstopbucardo Starting Bucardo Could not append to "/var/log/bucardo/log.bucardo": No such file or directory
解决方式：
# mkdir /var/log/bucardo # chmod 777 /var/log/bucardo/