MySql GR启动时节点状态卡在RECOVERING

问题描述:

root@localhost:(none) 04:47:06>SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 574d6e9a-fc15-11e7-9fcf-000c29879d51 | centos129   |        3306 | ONLINE       |
| group_replication_applier | b376058d-5762-11e7-baa0-000c29e6b568 | centos128   |        3306 | RECOVERING   |
| group_replication_applier | e87b5168-fc15-11e7-9fcf-000c29879d51 | centos130   |        3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+

并且错误日志显示内容如下:

2018-05-15T08:08:18.041338Z 54 [ERROR] Error reading packet from server for channel 'group_replication_recovery': The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires. (server_errno=1236)
2018-05-15T08:08:18.041384Z 54 [ERROR] Slave I/O for channel 'group_replication_recovery': Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.', Error_code: 1236

2018-05-15T08:10:18.511895Z 18 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='centos129', master_port= 3306, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2018-05-15T08:10:18.523029Z 18 [ERROR] Plugin group_replication reported: 'Maximum number of retries when trying to connect to a donor reached. Aborting group replication recovery.'
2018-05-15T08:10:18.523049Z 18 [Note] Plugin group_replication reported: 'Terminating existing group replication donor connection and purging the corresponding logs.'
2018-05-15T08:10:18.531944Z 18 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.
2018-05-15T08:10:18.547788Z 18 [ERROR] Plugin group_replication reported: 'Fatal error during the Recovery process of Group Replication. The server will leave the group.'
2018-05-15T08:10:18.549152Z 0 [Note] Plugin group_replication reported: 'getstart group_id 4317e324'
2018-05-15T08:10:22.045063Z 0 [Note] Plugin group_replication reported: 'state 4330 action xa_terminate'
2018-05-15T08:10:22.045482Z 0 [Note] Plugin group_replication reported: 'new state x_start'
2018-05-15T08:10:22.045511Z 0 [Note] Plugin group_replication reported: 'state 4257 action xa_exit'
2018-05-15T08:10:22.045634Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'
2018-05-15T08:10:22.045655Z 0 [Note] Plugin group_replication reported: 'new state x_start'

查看centos128的状态:

root@localhost:qn 05:02:15>select * from performance_schema.replication_group_member_stats \G ;
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15263657103992362:28
                         MEMBER_ID: b376058d-5762-11e7-baa0-000c29e6b568
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 0
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 
    LAST_CONFLICT_FREE_TRANSACTION:

  此时没有显示centos128上的gtid信息。查看其它节点状态:

centos129:

root@localhost:qn 04:47:47>select * from performance_schema.replication_group_member_stats \G ;
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15263657103992362:28
                         MEMBER_ID: 574d6e9a-fc15-11e7-9fcf-000c29879d51
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 1
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-151,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678
    LAST_CONFLICT_FREE_TRANSACTION: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:142
1 row in set (0.00 sec)

centos130:

root@localhost:(none) 05:05:06>select * from performance_schema.replication_group_member_stats \G ;
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15263657103992362:33
                         MEMBER_ID: e87b5168-fc15-11e7-9fcf-000c29879d51
       COUNT_TRANSACTIONS_IN_QUEUE: 5
        COUNT_TRANSACTIONS_CHECKED: 1
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-151,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678
    LAST_CONFLICT_FREE_TRANSACTION: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:142

查看节点的GTID状态:

centos128:

root@localhost:(none) 04:48:02>show global variables like '%gtid%' ;
+---------------------------------------------------+---------+
| Variable_name                                     | Value   |
+---------------------------------------------------+---------+
| binlog_gtid_simple_recovery                       | ON      |
| enforce_gtid_consistency                          | ON      |
| group_replication_allow_local_disjoint_gtids_join | OFF     |
| group_replication_gtid_assignment_block_size      | 1000000 |
| gtid_executed                                     |         |
| gtid_executed_compression_period                  | 1000    |
| gtid_mode                                         | ON      |
| gtid_owned                                        |         |
| gtid_purged                                       |         |
| session_track_gtids                               | OFF     |
+---------------------------------------------------+---------+

centos129和centos130一样:

root@localhost:qn 04:44:32>show global variables like '%gtid%' ;
+---------------------------------------------------+------------------------------------------------------------------------------------------+
| Variable_name                                     | Value                                                                                    |
+---------------------------------------------------+------------------------------------------------------------------------------------------+
| binlog_gtid_simple_recovery                       | ON                                                                                       |
| enforce_gtid_consistency                          | ON                                                                                       |
| group_replication_allow_local_disjoint_gtids_join | OFF                                                                                      |
| group_replication_gtid_assignment_block_size      | 1000000                                                                                  |
| gtid_executed                                     | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-149,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678 |
| gtid_executed_compression_period                  | 1000                                                                                     |
| gtid_mode                                         | ON                                                                                       |
| gtid_owned                                        |                                                                                          |
| gtid_purged                                       | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-135,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678 |
| session_track_gtids                               | OFF                                    

根据centos129与centos130的gtid_purged ,更改centos128的gtid_purged使其按gtid_purged开始复制:

root@localhost:(none) 04:53:03>set global gtid_purged='aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-135,b376058d-5762-11e7-baa0-000c29e6b568:1-47678' ;
Query OK, 0 rows affected (0.24 sec)

root@localhost:(none) 04:53:17>show global variables like '%gtid%' ;
+---------------------------------------------------+------------------------------------------------------------------------------------------+
| Variable_name                                     | Value                                                                                    |
+---------------------------------------------------+------------------------------------------------------------------------------------------+
| binlog_gtid_simple_recovery                       | ON                                                                                       |
| enforce_gtid_consistency                          | ON                                                                                       |
| group_replication_allow_local_disjoint_gtids_join | OFF                                                                                      |
| group_replication_gtid_assignment_block_size      | 1000000                                                                                  |
| gtid_executed                                     | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-135,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678 |
| gtid_executed_compression_period                  | 1000                                                                                     |
| gtid_mode                                         | ON                                                                                       |
| gtid_owned                                        |                                                                                          |
| gtid_purged                                       | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-135,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678 |
| session_track_gtids                               | OFF                                                                                      |
+---------------------------------------------------+------------------------------------------------------------------------------------------+

开启centos128的GR:

root@localhost:(none) 04:53:22>CHANGE MASTER TO MASTER_USER='rpl_user', MASTER_PASSWORD='rpl_pass' FOR CHANNEL 'group_replication_recovery';
Query OK, 0 rows affected, 2 warnings (0.02 sec)

root@localhost:(none) 04:54:17>show global variables like '%gtid%' ;
+---------------------------------------------------+------------------------------------------------------------------------------------------+
| Variable_name                                     | Value                                                                                    |
+---------------------------------------------------+------------------------------------------------------------------------------------------+
| binlog_gtid_simple_recovery                       | ON                                                                                       |
| enforce_gtid_consistency                          | ON                                                                                       |
| group_replication_allow_local_disjoint_gtids_join | OFF                                                                                      |
| group_replication_gtid_assignment_block_size      | 1000000                                                                                  |
| gtid_executed                                     | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-135,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678 |
| gtid_executed_compression_period                  | 1000                                                                                     |
| gtid_mode                                         | ON                                                                                       |
| gtid_owned                                        |                                                                                          |
| gtid_purged                                       | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-135,
b376058d-5762-11e7-baa0-000c29e6b568:1-47678 |
| session_track_gtids                               | OFF                                                                                      |
+---------------------------------------------------+------------------------------------------------------------------------------------------+
10 rows in set (0.00 sec)

root@localhost:(none) 04:54:21>START GROUP_REPLICATION; 
Query OK, 0 rows affected (1.93 sec)

root@localhost:(none) 04:54:35>SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 574d6e9a-fc15-11e7-9fcf-000c29879d51 | centos129   |        3306 | ONLINE       |
| group_replication_applier | b376058d-5762-11e7-baa0-000c29e6b568 | centos128   |        3306 | ONLINE       |
| group_replication_applier | e87b5168-fc15-11e7-9fcf-000c29879d51 | centos130   |        3306 | ONLINE       |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)

你可能感兴趣的:(mysql架构)