greenplum 如何恢复Master standby

环境信息：

Red Hat Enterprise Linux Server release 7.5 (Maipo)

Greenplum Database 5.16.0

[gpadmin@rhmdw gpadata]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.15.201  rhmdw
192.168.15.202  rhsdw1
192.168.15.203  rhsdw2

 

之前配置master standby后，重启主机后gp集群异常，部分节点无法启动。

报错信息如下：

[gpadmin@rhmdw pg_log]$ gpstate 
20190418:15:11:30:002103 gpstate:rhmdw:gpadmin-[INFO]:-Starting gpstate with args: 
20190418:15:11:30:002103 gpstate:rhmdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:15:11:31:002103 gpstate:rhmdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190418:15:11:31:002103 gpstate:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:15:11:31:002103 gpstate:rhmdw:gpadmin-[INFO]:-Gathering data from segments...
.. 
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-Greenplum instance status summary
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Master instance                                           = Active
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Master standby                                            = rhsdw1
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Standby master state                                      = Standby host DOWN   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total segment instance count from metadata                = 4
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Primary Segment Status
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total primary segments                                    = 2
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 1
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total primary segment failures (at master)                = 1                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number postmaster processes found                   = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Mirror Segment Status
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total mirror segments                                     = 2
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 2
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total mirror segment failures (at master)                 = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 2                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number postmaster processes found                   = 0
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments   = 1                   <<<<<<<<
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 1
20190418:15:11:33:002103 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------

重启gp集群，依然无法启动。

清除master standy后，重新添加恢复正常

[gpadmin@rhmdw pg_log]$ gpinitstandby -r rhsdw1
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:------------------------------------------------------
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Warm master standby removal parameters
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:------------------------------------------------------
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum master hostname               = rhmdw
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum master data directory         = /gp/gpdata/master/gpseg-1
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum master port                   = 5432
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master hostname       = rhsdw1
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master port           = 5432
20190418:15:13:36:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master data directory = /gp/gpdata/master/gpseg-1
Do you want to continue with deleting the standby master? Yy|Nn (default=N):
> y
20190418:15:13:39:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Removing standby master from catalog...
20190418:15:13:39:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Database catalog updated successfully.
20190418:15:13:40:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Removing filespace directories on standby master...
20190418:15:13:40:002213 gpinitstandby:rhmdw:gpadmin-[INFO]:-Successfully removed standby master
[gpadmin@rhmdw pg_log]$ gpstate -f
20190418:15:13:47:002248 gpstate:rhmdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:-Standby master instance not configured
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:--pg_stat_replication
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:-No entries found.
20190418:15:13:48:002248 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------

重新配置standby

[gpadmin@rhmdw ~]$ gpinitstandby -s rhsdw1
20190418:15:26:36:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
20190418:15:26:37:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Checking for filespace directory /gp/gpdata/master/gpseg-1 on rhsdw1
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:------------------------------------------------------
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master initialization parameters
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:------------------------------------------------------
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum master hostname               = rhmdw
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum master data directory         = /gp/gpdata/master/gpseg-1
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum master port                   = 5432
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master hostname       = rhsdw1
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master port           = 5432
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum standby master data directory = /gp/gpdata/master/gpseg-1
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Greenplum update system catalog         = On
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:------------------------------------------------------
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:- Filespace locations
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:------------------------------------------------------
20190418:15:26:38:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-pg_system -> /gp/gpdata/master/gpseg-1
Do you want to continue with standby master initialization? Yy|Nn (default=N):
> y
20190418:15:26:40:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby
20190418:15:26:42:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-The packages on rhsdw1 are consistent.
20190418:15:26:42:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Adding standby master to catalog...
20190418:15:26:42:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Database catalog updated successfully.
20190418:15:26:42:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Updating pg_hba.conf file...
20190418:15:26:44:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-pg_hba.conf files updated successfully.
20190418:15:26:49:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Updating filespace flat files...
20190418:15:26:49:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Filespace flat file updated successfully.
20190418:15:26:49:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Starting standby master
20190418:15:26:49:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Checking if standby master is running on host: rhsdw1  in directory: /gp/gpdata/master/gpseg-1
20190418:15:26:54:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files...
20190418:15:26:55:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully.
20190418:15:26:55:004709 gpinitstandby:rhmdw:gpadmin-[INFO]:-Successfully created standby master on rhsdw1
[gpadmin@rhmdw ~]$ gpstate -f
20190418:15:28:20:004864 gpstate:rhmdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20190418:15:28:20:004864 gpstate:rhmdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:15:28:20:004864 gpstate:rhmdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190418:15:28:20:004864 gpstate:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-Standby master details
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-----------------------
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-   Standby address          = rhsdw1
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-   Standby data directory   = /gp/gpdata/master/gpseg-1
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-   Standby port             = 5432
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-   Standby PID              = 5744
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:-   Standby status           = Standby host passive
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--pg_stat_replication
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--WAL Sender State: streaming
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--Sync state: sync
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--Sent Location: 0/14000000
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--Flush Location: 0/14000000
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--Replay Location: 0/14000000
20190418:15:28:22:004864 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------