在查看cactiez发现登录不上去,登录上去发现磁盘空间不足了,最后发现mysql报错日志高达10G,通过日志发现:

[root@HTD-CATIT bin]# tail -f /var/log/mysqld.log
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired

而且不断快速的增长,通过日志发现是cacti的这张表出现了错误,经过,排查经过修复表结构,报错结束,以下是修复过程。
通过使用 mysqlcheck --auto-repair --databases cacti -u root -p 来进行修复。

[root@HTD-CATIT bin]#  mysqlcheck --auto-repair --databases cacti -u root -p
Enter password: 
cacti.cdef                                         OK
cacti.cdef_items                                   OK
cacti.colors                                       OK
cacti.data_input                                   OK
cacti.data_input_data                              OK
cacti.data_input_fields                            OK
cacti.data_local                                   OK
cacti.data_template                                OK
cacti.data_template_data                           OK
cacti.data_template_data_rra                       OK
cacti.data_template_rrd                            OK
cacti.graph_local                                  OK
cacti.graph_template_input                         OK
cacti.graph_template_input_defs                    OK
cacti.graph_templates                              OK
cacti.graph_templates_gprint                       OK
cacti.graph_templates_graph                        OK
cacti.graph_templates_item                         OK
cacti.graph_tree                                   OK
cacti.graph_tree_items                             OK
cacti.host                                         OK
cacti.host_graph                                   OK
cacti.host_snmp_cache                              OK
cacti.host_snmp_query                              OK
cacti.host_template                                OK
cacti.host_template_graph                          OK
cacti.host_template_snmp_query                     OK
cacti.plugin_config                                OK
cacti.plugin_db_changes                            OK
cacti.plugin_hooks                                 OK
cacti.plugin_realms                                OK
cacti.poller                                       OK
cacti.poller_command                               OK
cacti.poller_item
warning  : Table is marked as crashed
warning  : 2 clients are using or haven't closed the table properly
error    : Unexpected byte: 0 at link: 95080
error    : Corrupt
cacti.poller_output                                OK
cacti.poller_reindex                               OK
cacti.poller_time                                  OK
cacti.rra                                          OK
cacti.rra_cf                                       OK
cacti.settings                                     OK
cacti.settings_graphs                              OK
cacti.settings_tree                                OK
cacti.snmp_query                                   OK
cacti.snmp_query_graph                             OK
cacti.snmp_query_graph_rrd                         OK
cacti.snmp_query_graph_rrd_sv                      OK
cacti.snmp_query_graph_sv                          OK
cacti.user_auth                                    OK
cacti.user_auth_perms                              OK
cacti.user_auth_realm                              OK
cacti.user_log                                     OK
cacti.version                                      OK
cacti.weathermap_auth                              OK
cacti.weathermap_data                              OK
cacti.weathermap_groups                            OK
cacti.weathermap_maps                              OK
cacti.weathermap_settings                          OK

Repairing tables
cacti.poller_item
info     : Wrong bytesec:   0-  0-  0 at 20912; Skipped
info     : Found block with too small length at 87732; Skipped
info     : Wrong bytesec:   0-  0-  0 at 21096; Skipped
info     : Wrong bytesec:  57- 48- 57 at 21164; Skipped
info     : Wrong bytesec:  46- 49- 48 at 28040; Skipped
info     : Found block that points outside data file at 92696
info     : Wrong bytesec:   0-  0-  0 at 20884; Skipped
info     : Found block with too small length at 104116; Skipped
info     : Wrong bytesec:   0-  0-  0 at 27264; Skipped
info     : Wrong bytesec:   0-  0-  0 at 28208; Skipped
info     : Wrong bytesec:   0-  0-  0 at 27980; Skipped
info     : Wrong block with wrong total length starting at 108504
info     : Found block with too small length at 109952; Skipped
info     : Wrong block with wrong total length starting at 112812
info     : Wrong bytesec:   0-  0-  0 at 21132; Skipped
info     : Wrong block with wrong total length starting at 114852
info     : Wrong bytesec:  46- 49- 48 at 28012; Skipped
info     : Found block with too small length at 117068; Skipped
info     : Wrong block with wrong total length starting at 117292
info     : Wrong bytesec:   0-  0-  0 at 9320; Skipped
info     : Wrong bytesec:  49- 46- 49 at 28360; Skipped
info     : Wrong block with wrong total length starting at 120876
info     : Wrong block with wrong total length starting at 121468
info     : Wrong bytesec:  46- 49- 46 at 28304; Skipped
info     : Wrong block with wrong total length starting at 124488
info     : Wrong block with wrong total length starting at 125820
warning  : Number of rows changed from 652 to 624
status   : OK

再次查看日志:

root@HTD-CATIT bin]# tail -f /var/log/mysqld.log
2018-03-02 16:01:53 1824 [Note] InnoDB: Waiting for purge to start
2018-03-02 16:01:53 1824 [Note] InnoDB: 5.6.37 started; log sequence number 1498263109644
2018-03-02 16:01:53 1824 [Note] Server hostname (bind-address): '*'; port: 3306
2018-03-02 16:01:53 1824 [Note] IPv6 is available.
2018-03-02 16:01:53 1824 [Note]   - '::' resolves to '::';
2018-03-02 16:01:53 1824 [Note] Server socket created on IP: '::'.
2018-03-02 16:01:53 1824 [Note] Event Scheduler: Loaded 0 events
2018-03-02 16:01:53 1824 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.37'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2018-03-02 16:01:54 1824 [Warning] IP address '172.20.66.26' could not be resolved: Name or service not known

错误得到解决了。