在查看cactiez发现登录不上去,登录上去发现磁盘空间不足了,最后发现mysql报错日志高达10G,通过日志发现:
[root@HTD-CATIT bin]# tail -f /var/log/mysqld.log
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
2018-03-02 15:54:17 31645 [ERROR] /usr/sbin/mysqld: Table './cacti/poller_item' is marked as crashed and should be repaired
而且不断快速的增长,通过日志发现是cacti的这张表出现了错误,经过,排查经过修复表结构,报错结束,以下是修复过程。
通过使用 mysqlcheck --auto-repair --databases cacti -u root -p 来进行修复。
[root@HTD-CATIT bin]# mysqlcheck --auto-repair --databases cacti -u root -p
Enter password:
cacti.cdef OK
cacti.cdef_items OK
cacti.colors OK
cacti.data_input OK
cacti.data_input_data OK
cacti.data_input_fields OK
cacti.data_local OK
cacti.data_template OK
cacti.data_template_data OK
cacti.data_template_data_rra OK
cacti.data_template_rrd OK
cacti.graph_local OK
cacti.graph_template_input OK
cacti.graph_template_input_defs OK
cacti.graph_templates OK
cacti.graph_templates_gprint OK
cacti.graph_templates_graph OK
cacti.graph_templates_item OK
cacti.graph_tree OK
cacti.graph_tree_items OK
cacti.host OK
cacti.host_graph OK
cacti.host_snmp_cache OK
cacti.host_snmp_query OK
cacti.host_template OK
cacti.host_template_graph OK
cacti.host_template_snmp_query OK
cacti.plugin_config OK
cacti.plugin_db_changes OK
cacti.plugin_hooks OK
cacti.plugin_realms OK
cacti.poller OK
cacti.poller_command OK
cacti.poller_item
warning : Table is marked as crashed
warning : 2 clients are using or haven't closed the table properly
error : Unexpected byte: 0 at link: 95080
error : Corrupt
cacti.poller_output OK
cacti.poller_reindex OK
cacti.poller_time OK
cacti.rra OK
cacti.rra_cf OK
cacti.settings OK
cacti.settings_graphs OK
cacti.settings_tree OK
cacti.snmp_query OK
cacti.snmp_query_graph OK
cacti.snmp_query_graph_rrd OK
cacti.snmp_query_graph_rrd_sv OK
cacti.snmp_query_graph_sv OK
cacti.user_auth OK
cacti.user_auth_perms OK
cacti.user_auth_realm OK
cacti.user_log OK
cacti.version OK
cacti.weathermap_auth OK
cacti.weathermap_data OK
cacti.weathermap_groups OK
cacti.weathermap_maps OK
cacti.weathermap_settings OK
Repairing tables
cacti.poller_item
info : Wrong bytesec: 0- 0- 0 at 20912; Skipped
info : Found block with too small length at 87732; Skipped
info : Wrong bytesec: 0- 0- 0 at 21096; Skipped
info : Wrong bytesec: 57- 48- 57 at 21164; Skipped
info : Wrong bytesec: 46- 49- 48 at 28040; Skipped
info : Found block that points outside data file at 92696
info : Wrong bytesec: 0- 0- 0 at 20884; Skipped
info : Found block with too small length at 104116; Skipped
info : Wrong bytesec: 0- 0- 0 at 27264; Skipped
info : Wrong bytesec: 0- 0- 0 at 28208; Skipped
info : Wrong bytesec: 0- 0- 0 at 27980; Skipped
info : Wrong block with wrong total length starting at 108504
info : Found block with too small length at 109952; Skipped
info : Wrong block with wrong total length starting at 112812
info : Wrong bytesec: 0- 0- 0 at 21132; Skipped
info : Wrong block with wrong total length starting at 114852
info : Wrong bytesec: 46- 49- 48 at 28012; Skipped
info : Found block with too small length at 117068; Skipped
info : Wrong block with wrong total length starting at 117292
info : Wrong bytesec: 0- 0- 0 at 9320; Skipped
info : Wrong bytesec: 49- 46- 49 at 28360; Skipped
info : Wrong block with wrong total length starting at 120876
info : Wrong block with wrong total length starting at 121468
info : Wrong bytesec: 46- 49- 46 at 28304; Skipped
info : Wrong block with wrong total length starting at 124488
info : Wrong block with wrong total length starting at 125820
warning : Number of rows changed from 652 to 624
status : OK
再次查看日志:
root@HTD-CATIT bin]# tail -f /var/log/mysqld.log
2018-03-02 16:01:53 1824 [Note] InnoDB: Waiting for purge to start
2018-03-02 16:01:53 1824 [Note] InnoDB: 5.6.37 started; log sequence number 1498263109644
2018-03-02 16:01:53 1824 [Note] Server hostname (bind-address): '*'; port: 3306
2018-03-02 16:01:53 1824 [Note] IPv6 is available.
2018-03-02 16:01:53 1824 [Note] - '::' resolves to '::';
2018-03-02 16:01:53 1824 [Note] Server socket created on IP: '::'.
2018-03-02 16:01:53 1824 [Note] Event Scheduler: Loaded 0 events
2018-03-02 16:01:53 1824 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.37' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
2018-03-02 16:01:54 1824 [Warning] IP address '172.20.66.26' could not be resolved: Name or service not known
错误得到解决了。