mysql slave复制1032错误解决方法

http://hi.baidu.com/dba_hui/item/a4b23a60ae1d6882c4d2497c

查看slave复制状态，发现SQL线程停止应用：

mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.9.61
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql-bin.000034
Read_Master_Log_Pos: 742553356
Relay_Log_File: relay-log.000004
Relay_Log_Pos: 41832895
Relay_Master_Log_File: mysql-bin.000034
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table: eoc.%,test.%,mysql.%
Replicate_Wild_Ignore_Table:
Last_Errno: 1032
Last_Error: Could not execute Delete_rows event on table eoc.eoc_course_rank; Can't find record in 'eoc_course_rank', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000034, end_log_pos 684728241
Skip_Counter: 0
Exec_Master_Log_Pos: 684727611
Relay_Log_Space: 257830925
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Delete_rows event on table eoc.eoc_course_rank; Can't find record in 'eoc_course_rank', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000034, end_log_pos 684728241
1 row in set (0.00 sec)

错误日志如下：
120719 1:01:07 [ERROR] Slave SQL: Could not execute Delete_rows event on table eoc.eoc_course_rank; Can't find record in 'eoc_course_rank', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000034, end_log_pos 684728241, Error_code: 1032
120719 1:01:07 [Warning] Slave: Can't find record in 'eoc_course_rank' Error_code: 1032
120719 1:01:07 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000034' position 684727611

从上面的情况可以看出，在slave执行从master复制过来的delete语句因为slave找不到要删除的数据，导致SQL线程报错，停止应用。也就是说master和slave上数据不一致导致的该错误。
遇到这种情况，一般都是找出错误，然后修改slave上的数据，然后启动slave，继续应用。
比如上面，要删除的数据在slave上找不到，我们就想知道是删除哪条数据呢？但是show slave status中又没有明确给出错误的语句，因此需要分析binary log来查看是哪条语句。

binary log的格式是mixed，初步使用mysqlbinlog查看，发现该条语句是RBR，而不是SBR,因此需要使用--base64-output=decode-rows -v -v选项，如下：
使用binary log：
[root@awstats usr]# /usr/local/mysql5157_2/bin/mysqlbinlog --no-defaults --base64-output=decode-rows -v -v mysql-bin.000034 | grep -A 200 '684728241' > zhh.txt
或者
使用relay log：
[root@mysqldb var]# /usr/local/mysql51572/bin/mysqlbinlog --no-defaults --base64-output=decode-rows -v -v relay-log.000004 | grep -A 200 '684728241' > zhh.txt

查看文件，发现以下内容：
#120719 1:01:03 server id 1 end_log_pos 684728241 Delete_rows: table id 16854 flags: STMT_END_F
### DELETE FROM eoc.eoc_course_rank
### WHERE
### @1=2 /* TINYINT meta=0 nullable=0 is_null=0 */
### @2=392 /* SHORTINT meta=0 nullable=0 is_null=0 */
### @3=574 /* INT meta=0 nullable=0 is_null=0 */
### @4='xxx' /* VARSTRING(192) meta=192 nullable=0 is_null=0 */
### @5='yyy' /* VARSTRING(1500) meta=1500 nullable=1 is_null=0 */
### @6='20111202-2df9eef5c0e33cfd643927cb1bb048fa17a40dfa.jpg' /* VARSTRING(600) meta=600 nullable=1 is_null=0 */
### @7=3 /* INT meta=0 nullable=1 is_null=0 */
### @8=256 /* INT meta=0 nullable=1 is_null=0 */
# at 684728241

即出错误的语句为：
delete from eoc.eoc_course_rank
where
第一列等于2
第二列等于392
......
根据该表的主键在slave查询，发现where条件中有一个字段值不同，因此在slave上找不到要删除的数据。

mysql> use eoc;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> SELECT * FROM eoc_course_rank
-> WHERE course_id=574 \G;
*************************** 1. row ***************************
rank_type: 2
rank_id: 385
course_id: 574
course_name: xxx

detail_info: yyy

brief_image: 20111202-2df9eef5c0e33cfd643927cb1bb048fa17a40dfa.jpg
cp_id: 3
grade_id: 256
1 row in set (0.00 sec)

ERROR:
No query specified

mysql> UPDATE eoc_course_rank SET rank_id= 392 WHERE course_id=574;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> stop slave;
Query OK, 0 rows affected (0.00 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.9.61
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql-bin.000034
Read_Master_Log_Pos: 746616004
Relay_Log_File: relay-log.000004
Relay_Log_Pos: 95495411
Relay_Master_Log_File: mysql-bin.000034
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table: eoc.%,test.%,mysql.%
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 738390127
Relay_Log_Space: 261893867
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 4791
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)

mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.9.61
Master_User: repl
Master_Port: 3312
Connect_Retry: 60
Master_Log_File: mysql-bin.000034
Read_Master_Log_Pos: 747488636
Relay_Log_File: relay-log.000005
Relay_Log_Pos: 904068
Relay_Master_Log_File: mysql-bin.000034
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table: eoc.%,test.%,mysql.%
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 747488636
Relay_Log_Space: 104594214
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)

上面说了，产生该错误的根本原因是主从数据不一致。为什么主从数据不一致呢？
从报错的时间点分析，发现该错误数据是一个event执行后产生的，查看该event，发现有存在结果集不确定的语句。修改该event的sql，再次执行该event调用的stored procedure，状态仍然为双YES。

另外有一点不明白：

my.cnf中设置的binary log是mixed格式：在未修改该sql之前，该行数据binary log记录的是RBR格式；修改后记录的是SBR格式。