思路

1. 查看数据库日志

2. 查看慢查询

3. 查看系统日志

4. 查看监控

 

2017-01-03 03:03:48 0 170103 03:03:45 mysqld_safe Number of processes running now: 0

170103 03:03:45 mysqld_safe mysqld restarted   --异常重启

[Note] /usr/sbin/mysqld (mysqld 5.6.25-log) starting as process 104767 ...

2017-01-03 03:03:48 7f9bcd3f4720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the opt

ion innodb_use_sys_malloc and with the InnoDB's internal memory allocator.

2017-01-03 03:03:48 104767 [Note] InnoDB: Using atomics to ref count buffer pool pages

2017-01-03 03:03:48 104767 [Note] InnoDB: The InnoDB memory heap is disabled

2017-01-03 03:03:48 104767 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2017-01-03 03:03:48 104767 [Note] InnoDB: Memory barrier is not used

2017-01-03 03:03:48 104767 [Note] InnoDB: Compressed tables use zlib 1.2.3

2017-01-03 03:03:48 104767 [Note] InnoDB: Using Linux native AIO

2017-01-03 03:03:48 104767 [Note] InnoDB: Using CPU crc32 instructions

2017-01-03 03:03:48 104767 [Note] InnoDB: Initializing buffer pool, size = 4.0G

2017-01-03 03:03:48 104767 [Note] InnoDB: Completed initialization of buffer pool

2017-01-03 03:03:48 104767 [Note] InnoDB: Highest supported file format is Barracuda.

2017-01-03 03:03:48 104767 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3021323914038

2017-01-03 03:03:48 104767 [Note] InnoDB: Database was not shutdown normally!

2017-01-03 03:03:48 104767 [Note] InnoDB: Starting crash recovery.

 

通过查看open-falcon 监控，发现数据库重启的时间内存使用率100%。mysqld_safe具有安全保护机制，对于异常的关闭数据库，mysqld_safe 进程可以把mysqld 进程拉起来。我们当时的情况是，数据库被拉起来了，但是主库参数设置的是只读，影响了部分业务。后续调整了参数，把主库的只读参数设置为read_only=off。当然罪魁祸首是因为慢查询引起的。找到慢查询，提交给开发同事，修改sql之后，稳定下来，不过内存也确实太紧张了，没法再挺着了，准备扩一点内存，这个费用没法再节省了。