测试基于gtid模式主从复制下数据不一致的处理

1.基本信息

主库：
 IP：10.16.24.107  port:3377
 server-id = 1073377
 data_dir:/data/MySQL/mysql3377/data/
 base_dir:/usr/local/mysql
 版本：mysql  5.6.29-log
 binlog_format:ROW

从库：
 IP：10.16.24.108  port:3377
 server-id = 1083377
 data_dir:/data/mysql/mysql3377/data/
 base_dir:/usr/local/mysql
 版本：mysql  5.6.29-log
 binlog_format:ROW

2.在主库和从库上准备数据

主库上操作：
 use lots
  create table test (id int,name varchar(20))
  engine=innodb DEFAULT CHARSET=utf8;
  insert into test values (1,'zeng1'),(2,'zeng2'),(3,'zeng3');

从库上操作:

 update test set name='zeng3-1' where name='zeng3';

查看主库test表上数据：
(product)root@localhost [lots]>  select * from test;
+------+-------+
| id   | name  |
+------+-------+
|    1 | zeng1 |
|    2 | zeng2 |
|    3 | zeng3 |
+------+-------+
3 rows in set (0.00 sec)

查看从库test表数据：
(product)root@localhost [lots]> select * from test;
+------+---------+
| id   | name    |
+------+---------+
|    1 | zeng1   |
|    2 | zeng2   |
|    3 | zeng3-1 |
+------+---------+
3 rows in set (0.00 sec)

3.模拟数据不一致报错

删除主库上id=3的数据：

(product)root@localhost [lots]>  delete from test where id=3;
Query OK, 1 row affected (0.00 sec)

(product)root@localhost [lots]> select * from test;
+------+-------+
| id   | name  |
+------+-------+
|    1 | zeng1 |
|    2 | zeng2 |
+------+-------+
2 rows in set (0.00 sec)

从库上查看test表的数据：
(product)root@localhost [lots]> select * from test;
+------+---------+
| id   | name    |
+------+---------+
|    1 | zeng1   |
|    2 | zeng2   |
|    3 | zeng3-1 |
+------+---------+
3 rows in set (0.00 sec)

从库test表上数据没有发生变化，主是什么原因？

查看从库状态：
(product)root@localhost [lots]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.16.24.107
                  Master_User: repl
                  Master_Port: 3377
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000017
          Read_Master_Log_Pos: 2366
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 2278
        Relay_Master_Log_File: mysql-bin.000017
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1032
                   Last_Error: Worker 3 failed executing transaction 'ae2d957d-f682-11e5-94fa-0050568a0bcb:8' at master log mysql-bin.000017, end_log_pos 2335; Could

not execute Delete_rows event on table lots.test; Can't find record in 'test', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-

bin.000017, end_log_pos 2335
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2068
              Relay_Log_Space: 2774
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1032
               Last_SQL_Error: Worker 3 failed executing transaction 'ae2d957d-f682-11e5-94fa-0050568a0bcb:8' at master log mysql-bin.000017, end_log_pos 2335; Could

not execute Delete_rows event on table lots.test; Can't find record in 'test', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-

bin.000017, end_log_pos 2335
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1073377
                  Master_UUID: ae2d957d-f682-11e5-94fa-0050568a0bcb
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp: 160428 20:06:42
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: ae2d957d-f682-11e5-94fa-0050568a0bcb:1-8
            Executed_Gtid_Set: 1a3f7187-0b93-11e6-9e55-0050568a6e64:1-3,
ae2d957d-f682-11e5-94fa-0050568a0bcb:1-7
                Auto_Position: 1
1 row in set (0.00 sec)

4.问题分析

上面发现问题如下：

a.从库数据没有同步
b.从库SQL线程状态为no
c.从库SQL线程报错

主库状态：
(product)root@localhost [lots]> show master status\G

*************************** 1. row ***************************
             File: mysql-bin.000017
         Position: 2366
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: ae2d957d-f682-11e5-94fa-0050568a0bcb:1-8
1 row in set (0.00 sec)

若是生产环境的操作，可能不知道删除前的数据是什么样的？这时就要去分析Binlog，检查删除的数据：
分析mysql-bin.000017:
[mysql@mvxl0782 logs]$ mysqlbinlog -vv mysql-bin.000017 --stop-position=2366

#160428 20:06:42 server id 1073377  end_log_pos 2335 CRC32 0x6c2da148   Delete_rows: table id 71 flags: STMT_END_F

BINLOG '
0vwhVx3hYBAAMwAAAL8IAACAABtkZWxldGUgZnJvbSB0ZXN0IHdoZXJlIGlkPTOwAvoB
0vwhVxPhYBAAMgAAAPEIAAAAAEcAAAAAAAEABGxvdHMABHRlc3QAAgMPAjwAA9DQyls=
0vwhVyDhYBAALgAAAB8JAAAAAEcAAAAAAAEAAgAC//wDAAAABXplbmczSKEtbA==
'/*!*/;
### DELETE FROM `lots`.`test`
### WHERE
###   @1=3 /* INT meta=0 nullable=1 is_null=0 */
###   @2='zeng3' /* VARSTRING(60) meta=60 nullable=1 is_null=0 */
# at 2335
#160428 20:06:42 server id 1073377  end_log_pos 2366 CRC32 0xe509dd97   Xid = 56
COMMIT/*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog *//*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;

上面日志看到，有删除了delete from test where @1=3 and  @2='zeng3' ，@1是id列，@2是name列，而从库上id=3的name值是zeng3-1，说明主从id=3的name值不一致，又由于主库同步到从

库的日志，在从库上面是一行行匹配，数据不致，所以从库的sql线程报错。

5.问题处理

从库上执行如下操作：
stop slave;
set gtid_next="ae2d957d-f682-11e5-94fa-0050568a0bcb:8";   ---跳过报错
begin;commit;                                             ---执行一个空事务代替报错事务
set gtid_next="AUTOMATIC"; 
start slave;

查看从库状态，显示正常。
(product)root@localhost [lots]> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.16.24.107
                  Master_User: repl
                  Master_Port: 3377
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000017
          Read_Master_Log_Pos: 2366
               Relay_Log_File: relay-bin.000003
                Relay_Log_Pos: 448
        Relay_Master_Log_File: mysql-bin.000017
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2366
              Relay_Log_Space: 3071
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1073377
                  Master_UUID: ae2d957d-f682-11e5-94fa-0050568a0bcb
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: ae2d957d-f682-11e5-94fa-0050568a0bcb:1-8
            Executed_Gtid_Set: 1a3f7187-0b93-11e6-9e55-0050568a6e64:1-3,
ae2d957d-f682-11e5-94fa-0050568a0bcb:1-8
                Auto_Position: 1
1 row in set (0.00 sec)

再手工删除从库上test表id=3的记录：
(product)root@localhost [lots]> delete from test where id=3;

Query OK, 1 row affected (0.00 sec)

最后查看从库test表的记录，已与主库记录保持一致：
(product)root@localhost [lots]> select * from test;
+------+-------+
| id   | name  |
+------+-------+
|    1 | zeng1 |
|    2 | zeng2 |
+------+-------+
2 rows in set (0.00 sec)