1、MySQL主库发生内存溢出,并自动重启,通过如下日志可以看出:
[chenzx@MySQL02 ~]$tail /var/log/messages
Aug 28 20:23:33 MySQL02 kernel: Out of memory: Kill process 26033 (mysqld) score 751 or sacrifice child
2、由于管理员在外面吃饭,21:20才登陆到数据库服务器定位原因,意识到内存溢出,就在线调整了主库innodb_buffer_pool的大小,从100G缩小为90G:
mysql>set global innodb_buffer_pool_size=90G;
3、过了几分钟,管理员就收到nagios告警,说两个slave库停止复制了,登陆两个slave库一看,均报错如下:
slave1:
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.19.20.111
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: binglog.001503
Read_Master_Log_Pos: 834920825
Relay_Log_File: relaylog.000283
Relay_Log_Pos: 321413
Relay_Master_Log_File: binglog.001501
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1032
Last_Error: Could not execute Delete_rows event on table condb.rundetails; Can't find record in 'rundetails', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binglog.001501, end_log_pos 321522
Skip_Counter: 0
Exec_Master_Log_Pos: 321204
Relay_Log_Space: 9694583059
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Delete_rows event on table condb.rundetails; Can't find record in 'rundetails', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binglog.001501, end_log_pos 321522
Replicate_Ignore_Server_Ids:
Master_Server_Id: 111
Master_UUID: a8beebcf-cb3f-11e7-858b-005056b85344
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 180829 08:41:14
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
slave2:
mysql> show slave status \G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 172.19.20.111
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: binglog.001503
Read_Master_Log_Pos: 739719811
Relay_Log_File: relaylog.004380
Relay_Log_Pos: 321413
Relay_Master_Log_File: binglog.001501
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1032
Last_Error: Could not execute Delete_rows event on table condb.rundetails; Can't find record in 'rundetails', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binglog.001501, end_log_pos 321522
Skip_Counter: 0
Exec_Master_Log_Pos: 321204
Relay_Log_Space: 9599382409
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1032
Last_SQL_Error: Could not execute Delete_rows event on table condb.rundetails; Can't find record in 'rundetails', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log binglog.001501, end_log_pos 321522
Replicate_Ignore_Server_Ids:
Master_Server_Id: 111
Master_UUID: a8beebcf-cb3f-11e7-858b-005056b85344
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 180828 20:28:04
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
4、我的MySQL版本是5.7.22的,请问是什么原因导致的我两个备库数据和主库不一致的。谢谢大家。