postgresql双机热备、高可用方案（采用pacemaker+corosync实现）
PostgreSQL高可用
需求描述
我们有两台centos7的数据库主机A、B。要对A、B实现双机热备，A作为数据库master主机，对外提供读写服务，B作为slave主机能实时同步A的数据。当A发生故障时，B从slave状态切换到master状态，以保证pg数据库能正常对外提供服务。
经过调研，最终我们给出的解决方案是采用pacemaker和corosync组件来实现postgresql的高可用。
所有部署脚本可以下github中找到：https://github.com/qinguanri/pg-ha
环境说明
- 操作系统centos7：
[root@10 ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@10 ~]# uname -a
Linux 10.16.93.179 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linuxpostgresql版本：9.2.15
- 2台主机：
| 角色 | IP |
|---|---|
| node1 | 10.16.93.179 |
| node2 | 10.16.93.182 |
- 分配2个虚IP。我们的数据库对外提供服务时，暴露的是虚IP：
vip-master: 10.16.93.180
vip-slave: 10.16.93.181
配置Linux集群环境
注意：请严格按照下文所给的顺序进行配置
1.安装Pacemaker和Corosync包
需要在所有节点执行：
$ yum install -y pacemaker pcs psmisc policycoreutils-python postgresql-server2.禁用防火墙
在所有节点执行：
$ systemctl disable firewalld.service
$ systemctl stop firewalld.service3.启用pcs
在所有节点执行：
$ systemctl start pcsd.service
$ systemctl enable pcsd.service
ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'
$ echo hacluster | passwd hacluster --stdin
Changing password for user hacluster.
Changing password for user hacluster.
passwd: all authentication tokens updated successfully.4.集群认证
在任何一个节点上执行，这里选择node1：
$ pcs cluster auth -u hacluster -p hacluster 10.16.93.179 10.16.93.182
10.16.93.179: Authorized
10.16.93.182: Authorized5.同步配置
在node1上执行：
$ pcs cluster setup --last_man_standing=1 --name pgcluster 10.16.93.179 10.16.93.182
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
10.16.93.179: Succeeded
10.16.93.182: Succeeded6.启动集群
在node1上执行。（有时需要稍等十几秒，才能看到命令的输出结果）：
$ pcs cluster start --all
172.17.5.90: Starting Cluster...
172.17.5.91: Starting Cluster...
172.17.5.92: Starting Cluster...7.检验
1）检验corosync
在node1上执行：
$ pcs status corosync
Membership information
----------------------
Nodeid Votes Name
1 1 10.16.93.179 (local)
2 1 10.16.93.1822）检验pacemaker
root@10 data]# pcs status
Cluster name: pgcluster
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Mon Apr 4 23:45:48 2016 Last change: Fri Apr 1 09:17:03 2016 by root via crm_attribute on tsung_test
Stack: corosync
Current DC: tsung_test (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
2 nodes and 5 resources configured
Online: [ 10.16.93.179 tsung_test ]
Full list of resources:
Master/Slave Set: pgsql-cluster [pgsql]
Masters: [ tsung_test ]
Slaves: [ 10.16.93.179 ]
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started tsung_test
Resource Group: slave-group
vip-slave (ocf::heartbeat:IPaddr2): Started 10.16.93.179
PCSD Status:
10.16.93.179: Online
tsung_test (10.16.93.182): Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled注意：当vip-master和vip-slave都是Started状态时，集群节点配置正确。否则，请检查你的配置是否有误
安装和配置PostgreSQL
1.创建数据库目录
需要在所有节点上执行：
$ mkdir -p /data/postgresql/{data,xlog_archive}
$ chown -R postgres:postgres /data/postgresql/
$ chmod 0700 /data/postgresql/data2.初始化db
在node1上执行：
$ su - postgres
$ initdb -D /data/postgresql/data/
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
fixing permissions on existing directory /data/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /data/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
postgres -D /data/postgresql/data
or
pg_ctl -D /data/postgresql/data -l logfile start3.修改配置文件
在node1上执行：
$ vim /data/postgresql/data/postgresql.conf
listen_addresses = '*'
wal_level = hot_standby
synchronous_commit = on
archive_mode = on
archive_command = 'cp %p /data/postgresql/xlog_archive/%f'
max_wal_senders=5
wal_keep_segments = 32
hot_standby = on
restart_after_crash = off
replication_timeout = 5000
wal_receiver_status_interval = 2
max_standby_streaming_delay = -1
max_standby_archive_delay = -1
synchronous_commit = on
restart_after_crash = off
hot_standby_feedback = on
$ vim /data/postgresql/data/pg_hba.conf
local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
host all all 10.16.0.0/16 md5
host all all 10.18.0.0/16 md5
host replication all 10.16.0.0/16 md5注意：pg_hba.conf文件用于配置postgresql的访问控制，例如：host all all 10.16.0.0/16 md5,表示允许10.16.0.0/16网段的主机访问所有用户、所有数据库，密码使用md5密文的方式。你需要根据你的实际网段进行设置。当出现访问权限错误时，请检查你的pg_hba.conf文件
4.启动
启动数据库，创建replicator用户。将用户postgres的密码修改为postgres（pg初始安装时，postgres用户的登录密码默认为空）
在node1上执行：
$ pg_ctl -D /data/postgresql/data/ start
server starting
[ 2015-10-16 08:51:31.451 UTC 53158 5620ba93.cfa6 1 0]LOG: redirecting log output to logging collector process
[ 2015-10-16 08:51:31.451 UTC 53158 5620ba93.cfa6 2 0]HINT: Future log output will appear in directory "pg_log".
$ psql -U postgres
psql (9.2.13)
Type "help" for help.
postgres=# create role replicator with login replication password '8d5e9531-3817-460d-a851-659d2e51ca99';
CREATE ROLE
postgres=# alter user postgres with password 'postgres';
postgres=# \q5.制作slave
在node2上执行：
$ su - postgres
$ pg_basebackup -h 10.16.93.179 -U postgres -D /data/postgresql/data/ -X stream -P
could not change directory to "/home/wenhang.pan"
20127/20127 kB (100%), 1/1 tablespace
$ vim /data/postgresql/data/recovery.conf
standby_mode = 'on'
primary_conninfo = 'host=10.16.93.179 port=5432 user=replicator application_name=zhaopin-5-92 password=8d5e9531-3817-460d-a851-659d2e51ca99 keepalives_idle=60 keepalives_in
terval=5 keepalives_count=5'
restore_command = 'cp /data/postgresql/xlog_archive/%f %p'
recovery_target_timeline = 'latest'6.启动slave
在node2上执行：
$ pg_ctl -D /data/postgresql/data/ start
pg_ctl: another server might be running; trying to start server anyway
server starting
-bash-4.2$ LOG: database system was interrupted while in recovery at log time 2015-10-16 08:19:07 GMT
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: entering standby mode
LOG: redo starts at 0/3000020
LOG: consistent recovery state reached at 0/30000E0
LOG: database system is ready to accept read only connections
LOG: streaming replication successfully connected to primary7.查看集群状态
在node1上执行一下命令。这个时候，可看到node2是一个replication。同步状态为sync
[root@10 ~]# psql -U postgres
psql (9.2.15)
Type "help" for help.
postgres=# select * from pg_stat_replication ;
pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start
| state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
-------+----------+------------+------------------+--------------+-----------------+-------------+-----------------------
--------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
17349 | 16384 | replicator | 10.16.93.182 | 10.16.93.182 | | 22690 | 2016-04-14 14:07:25.18
1297+08 | streaming | 0/9014508 | 0/9014508 | 0/9014508 | 0/9014508 | 1 | sync
(1 row)
8.停止PostgreSQL服务
在node1、node2上执行：
$ pg_ctl -D /data/postgresql/data/ -mi stop
waiting for server to shut down.... done
server stopped注意：在配置自动切换前，一定要先把node1、node2上的PostgreSQL服务停掉
配置自动切换
上面的配置完成了数据库的初始化。下面进行自动切换相关的配置。当发生故障时，master/slave身份实现自动切换。
1.配置
在node1执行：
1）配置项较多，我们先将配置步骤写到脚本里
[root@10 ~]# vim cluster_setup.sh
# 将cib配置保存到文件
pcs cluster cib pgsql_cfg
# 在pacemaker级别忽略quorum
pcs -f pgsql_cfg property set no-quorum-policy="ignore"
# 禁用STONITH
pcs -f pgsql_cfg property set stonith-enabled="false"
# 设置资源粘性，防止节点在故障恢复后发生迁移
pcs -f pgsql_cfg resource defaults resource-stickiness="INFINITY"
# 设置多少次失败后迁移
pcs -f pgsql_cfg resource defaults migration-threshold="3"
# 设置master节点虚ip
pcs -f pgsql_cfg resource create vip-master IPaddr2 ip="10.16.93.180" cidr_netmask="24" op start timeout="60s" interval="0s" on-fail="restart" op monitor timeout="60s" interval="10s" on-fail="restart" op stop timeout="60s" interval="0s" on-fail="block"
# 设置slave节点虚ip
pcs -f pgsql_cfg resource create vip-slave IPaddr2 ip="10.16.93.181" cidr_netmask="24" op start timeout="60s" interval="0s" on-fail="restart" op monitor timeout="60s" interval="10s" on-fail="restart" op stop timeout="60s" interval="0s" on-fail="block"
# 设置pgsql集群资源
# pgctl、psql、pgdata和config等配置根据自己的环境修改,node list填写节点的hostname，master_ip填写虚master_ip
pcs -f pgsql_cfg resource create pgsql pgsql pgctl="/usr/bin/pg_ctl" psql="/usr/bin/psql" pgdata="/data/postgresql/data/" config="/data/postgresql/data/postgresql.conf" rep_mode="sync" node_list="tsung_test 10.16.93.179" master_ip="10.16.93.180" repuser="replicator" primary_conninfo_opt="password=8d5e9531-3817-460d-a851-659d2e51ca99 keepalives_idle=60 keepalives_interval=5 keepalives_count=5" restore_command="cp /data/postgresql/xlog_archive/%f %p" restart_on_promote='true' op start timeout="60s" interval="0s" on-fail="restart" op monitor timeout="60s" interval="4s" on-fail="restart" op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" op promote timeout="60s" interval="0s" on-fail="restart" op demote timeout="60s" interval="0s" on-fail="stop" op stop timeout="60s" interval="0s" on-fail="block"
# 设置master/slave模式，clone-max=2，两个节点
pcs -f pgsql_cfg resource master pgsql-cluster pgsql master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# 配置master ip组
pcs -f pgsql_cfg resource group add master-group vip-master
# 配置slave ip组
pcs -f pgsql_cfg resource group add slave-group vip-slave
# 配置master ip组绑定master节点
pcs -f pgsql_cfg constraint colocation add master-group with master pgsql-cluster INFINITY
# 配置启动master节点
pcs -f pgsql_cfg constraint order promote pgsql-cluster then start master-group symmetrical=false score=INFINITY
# 配置停止master节点
pcs -f pgsql_cfg constraint order demote pgsql-cluster then stop master-group symmetrical=false score=0
# 配置slave ip组绑定slave节点
pcs -f pgsql_cfg constraint colocation add slave-group with slave pgsql-cluster INFINITY
# 配置启动slave节点
pcs -f pgsql_cfg constraint order promote pgsql-cluster then start slave-group symmetrical=false score=INFINITY
# 配置停止slave节点
pcs -f pgsql_cfg constraint order demote pgsql-cluster then stop slave-group symmetrical=false score=0
# 把配置文件push到cib
pcs cluster cib-push pgsql_cfg2）执行操作文件
$ sh cluster_setup.sh2.查看状态
1）查看cluster状态
在node1上执行：
[root@10 ~]# pcs status
Cluster name: pgcluster
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Last updated: Mon Apr 4 23:15:56 2016 Last change: Fri Apr 1 09:17:03 2016 by root via crm_attribute on tsung_test
Stack: corosync
Current DC: tsung_test (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
2 nodes and 5 resources configured
Online: [ 10.16.93.179 tsung_test ]
Full list of resources:
Master/Slave Set: pgsql-cluster [pgsql]
Masters: [ tsung_test ]
Slaves: [ 10.16.93.179 ]
Resource Group: master-group
vip-master (ocf::heartbeat:IPaddr2): Started tsung_test
Resource Group: slave-group
vip-slave (ocf::heartbeat:IPaddr2): Started 10.16.93.179
PCSD Status:
10.16.93.179: Online
tsung_test (10.16.93.182): Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled2）查看PostgreSQL集群状态
在node1上执行：
[root@tsung_test data]# psql -U postgres
psql (9.2.15)
Type "help" for help.
postgres=# select * from pg_stat_replication ;
pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | state | sent_location | write_locatio
n | flush_location | replay_location | sync_priority | sync_state
-------+----------+------------+------------------+--------------+-----------------+-------------+-------------------------------+-----------+---------------+--------------
--+----------------+-----------------+---------------+------------
19415 | 16384 | replicator | 10.16.93.182 | 10.16.93.182 | | 18289 | 2016-04-01 09:17:29.037197-04 | streaming | 0/40F0ED8 | 0/40F0ED8
| 0/40F0ED8 | 0/40F0ED8 | 0 | async
(1 row)验证自动切换
参考文献
- http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster
- http://m.oschina.net/blog/518928
- http://m.oschina.net/blog/519458?mType=Group