刚洗完澡准备睡觉，同事给我打电话说团购数据库主从不同步了，尼玛，咋就这么苦逼呢，好吧，速度开电脑拨×××解决。
1、登录从库查看主从同步状态，确实是否不同步
mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Master_Host: 10.10.0.236 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.001501 Read_Master_Log_Pos: 144001931 Relay_Log_File: mysql-relay-bin.000296 Relay_Log_Pos: 144002077 Relay_Master_Log_File: mysql-bin.001501 Slave_IO_Running: No Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: mysql,information_schema,performance_schema,test Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 144001931 Relay_Log_Space: 144002276 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 1236 Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position' Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2 1 row in set (0.00 sec)
看来尼玛确实出问题了，查看从库错误日志：
[root@tg_0_zjc tuan]# tail -n 30 logs/mysqld.log 130619 22:23:36 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001343' at position 107, relay log './mysqld-relay-bin.000001' position: 4 130619 22:23:36 [Note] Slave I/O thread: connected to master '[email protected]:3306',replication started in log 'mysql-bin.001343' at position 107 130620 14:04:21 [Note] Slave: received end packet from server, apparent master shutdown: 130620 14:04:21 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.001343' at position 2544563 130620 14:04:21 [ERROR] Slave I/O: error reconnecting to master '[email protected]:3306' - retry-time: 60 retries: 86400, Error_code: 2003 130620 14:13:45 [Note] Slave: connected to master '[email protected]:3306',replication resumed in log 'mysql-bin.001343' at position 2544563 130623 22:04:17 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 130623 22:04:17 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.001402' at position 121870679 130623 22:04:20 [ERROR] Slave I/O: error reconnecting to master '[email protected]:3306' - retry-time: 60 retries: 86400, Error_code: 2003 130623 22:08:29 [Note] Slave: connected to master '[email protected]:3306',replication resumed in log 'mysql-bin.001402' at position 121870679 130623 22:08:29 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236) 130623 22:08:29 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position', Error_code: 1236 130623 22:08:29 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.001402', position 121870679 130623 22:26:29 [Note] Error reading relay log event: slave SQL thread was killed 130623 22:26:34 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='10.10.0.236', master_port='3306', master_log_file='mysql-bin.001402', master_log_pos='121870679'. New state master_host='10.10.0.236', master_port='3306', master_log_file='mysql-bin.001403', master_log_pos='107'. 130623 22:26:42 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001403' at position 107, relay log './mysqld-relay-bin.000001' position: 4 130623 22:26:42 [Note] Slave I/O thread: connected to master '[email protected]:3306',replication started in log 'mysql-bin.001403' at position 107 130629 12:04:21 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013) 130629 12:04:21 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.001501' at position 144001931 130629 12:04:24 [ERROR] Slave I/O: error reconnecting to master '[email protected]:3306' - retry-time: 60 retries: 86400, Error_code: 2003 130629 12:08:33 [Note] Slave: connected to master '[email protected]:3306',replication resumed in log 'mysql-bin.001501' at position 144001931 130629 12:08:33 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236) 130629 12:08:33 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position', Error_code: 1236 130629 12:08:33 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.001501', position 144001931 130629 22:10:08 [Note] Error reading relay log event: slave SQL thread was killed 130629 22:10:12 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.001501' at position 144001931, relay log './mysqld-relay-bin.000296' position: 144002077 130629 22:10:12 [Note] Slave I/O thread: connected to master '[email protected]:3306',replication started in log 'mysql-bin.001501' at position 144001931 130629 22:10:12 [ERROR] Error reading packet from server: Client requested master to start replication from impossible position ( server_errno=1236) 130629 22:10:12 [ERROR] Slave I/O: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position', Error_code: 1236 130629 22:10:12 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.001501', position 144001931
尝试更改position位置恢复：
mysql> slave stop; Query OK, 0 rows affected (0.01 sec) mysql> change master to master_host='10.10.0.236', master_Port=3306, master_user='slave', master_password='123456', master_log_file='mysql-bin.001501', master_log_pos=144001931; Query OK, 0 rows affected (0.05 sec) mysql> slave start; Query OK, 0 rows affected (0.00 sec) mysql> show slave status \G *************************** 1. row *************************** Slave_IO_State: Master_Host: 10.10.0.236 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.001501 Read_Master_Log_Pos: 144001931 Relay_Log_File: mysqld-relay-bin.000001 Relay_Log_Pos: 4 Relay_Master_Log_File: mysql-bin.001501 Slave_IO_Running: No Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: mysql,information_schema,performance_schema,test,mysql,information_schema,performance_schema,test Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 144001931 Relay_Log_Space: 107 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 1236 Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from impossible position' Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2 1 row in set (0.00 sec)
坑爹啊，还是不行，查看主库的binlog日志，按照错误点的标记去主库的日志中查找：
[root@tg_0_zq ~]# /usr/local/mysql/bin/mysqlbinlog --start-position=144001931 /data/mysql/mysql-bin.001501 /*!40010 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 40019 #130629 12:00:50 server id 2 end_log_pos 107 Start: binlog v 4, server v 5.5.15-log created 130629 12:00:50 # Warning: this binlog is either in use or was not closed properly. BINLOG ' 8lvOUQ8CAAAAZwAAAGsAAAABAAQANS41LjE1LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAVAAEGggAAAAICAgCAA== '/*!*/; DELIMITER ; # End of log file ROLLBACK /* added by mysqlbinlog */; /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
找到和出错最近的那个点，重新change master，把position止到这个位置：
mysql> change master to master_host='10.10.0.236', master_Port=3306, master_user='slave', master_password='123456', master_log_file='mysql-bin.001501', master_log_pos=40019; Query OK, 0 rows affected (0.00 sec) mysql> slave start; Query OK, 0 rows affected (0.00 sec) mysql> show slave status \G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 10.10.0.236 Master_User: slave Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.001502 Read_Master_Log_Pos: 204293576 Relay_Log_File: mysqld-relay-bin.000003 Relay_Log_Pos: 28179632 Relay_Master_Log_File: mysql-bin.001502 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB:尼玛，终于好使了，关机睡觉。