MySQL General、Slow日志介绍，参数及配置

MySQL General log

MySQL General log是记录mysqld(MySQLserver进程)收到的所有请求的日志，我们可以通过这个命令查看格式：

SHOW CREATE TABLE mysql.general_log

可以看出general_log的格式为：

CREATE TABLE `general_log` (
  `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
  `user_host` mediumtext NOT NULL,
  `thread_id` bigint(21) unsigned NOT NULL,
  `server_id` int(10) unsigned NOT NULL,
  `command_type` varchar(64) NOT NULL,
  `argument` mediumblob NOT NULL
) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='General log'

这个日志包括客户端连接事件记录，客户端发送来的SQL查询记录等等。其中客户端连接还包括连接类型等等，目前MySQL支持的连接类型有TCP/IP, SSL/TLS , Socket (unix文件链接), Named Pipe (windows管道连接), or Shared Memory (共享内存)

客户端发送来的SQL查询记录也许和最后执行的顺序不同，这个只是收到请求的时间。

配置

查看参数

mysql> show variables like '%general%';

+------------------+----------------------------------------------------------------+
| Variable_name    | Value                                                          |
+------------------+----------------------------------------------------------------+
| general_log      | OFF                                                            |
| general_log_file | D:\mysql-5.7.21-winx64\mysql-5.7.21-winx64\data\BIH-D-6885.log |
+------------------+----------------------------------------------------------------+
2 rows in set

临时开启通用日志查询：set global general_log = on;
临时关闭通用日志查询：set global general_log = off;

日志格式示例：

mysqld, Version: 5.7.21 (MySQL Community Server (GPL)). started with:
TCP Port: 3306, Named Pipe: MySQL
Time                 Id Command    Argument
2018-03-30T03:19:50.409295Z     4 Connect   root@localhost on  using TCP/IP
2018-03-30T03:19:50.409295Z     4 Query SET NAMES utf8mb4
2018-03-30T03:19:50.409295Z     4 Init DB   test
2018-03-30T03:19:50.434300Z     4 Query SELECT @@character_set_database, @@collation_database
2018-03-30T03:19:50.448303Z     4 Query SHOW FULL TABLES WHERE Table_type != 'VIEW'
2018-03-30T03:19:53.385890Z     2 Query SHOW CHARACTER SET
2018-03-30T03:19:53.386890Z     2 Query SHOW COLLATION
2018-03-30T03:19:53.389891Z     4 Query SHOW ENGINES
2018-03-30T03:20:41.493510Z     4 Query SHOW FULL TABLES WHERE Table_type != 'VIEW'
2018-03-30T03:20:41.496510Z     4 Query SHOW FULL TABLES WHERE Table_type = 'VIEW'
2018-03-30T03:20:41.543520Z     4 Query CREATE TABLE `test` (
`id`  int NOT NULL AUTO_INCREMENT ,
`name`  varchar(255) NULL ,
`info`  varchar(255) NULL ,
PRIMARY KEY (`id`)
)

如果想永久配置，则在配置文件中加入：

general_log = ON
general_log_file=D:\mysql-5.7.21-winx64\mysql-5.7.21-winx64\data\BIH-D-6885.log

可以以文件和表格（就是上面提到的mysql.general_log表）两种方式存储：
查看当前格式

mysql> show variables like '%log_output%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_output    | FILE  |
+---------------+-------+
1 row in set

设置通用日志输出为表方式：

set global log_output = 'file';

设置通用日志输出为表和文件方式：

set global log_output = 'file,table';

利用mysql general log日志找出查询次数最多的SQL句子

analysis-general-log.py：

#!/usr/bin/python

# sort and count mysql general log
# Author: Jason
# Created: UTC 2015-02-15 17:51:53

import re
import sys
import os

if len(sys.argv) == 2:
    logPath = sys.argv[1]
    if not os.path.exists(logPath):
        print ("file " + logPath + " does not exists.")
        sys.exit(1)
else:
    print ("Usage: " + sys.argv[0] + " logPath")
    sys.exit(1)

logFo = open(logPath)
match = 0

for line in logFo:
    line = re.sub(r"＼n","",line)
    if match == 0:
        # match line begin with numbers
        lineMatch = re.match(r"＼s+[0-9]+＼s+.*",line,flags=re.I)
        if lineMatch:
            lineTmp = lineMatch.group(0)
            match = match + 1
            continue

    elif match == 1:
        # match line begin with numbers
        lineMatch = re.match(r"＼s+[0-9]+＼s+.*",line,flags=re.I)
        if lineMatch:
            # match only query
            lineMatchQuery = re.match(r".*Query＼s+(.*)",lineTmp,flags=re.I)
            if lineMatchQuery:
                lineTmp = lineMatchQuery.group(1)
                # remove extra space
                lineTmp = re.sub(r"＼s+", " ",lineTmp)
                # replace values (value) to values (x)
                lineTmp = re.sub(r"values＼s*＼(.*?＼)", "values (x)",lineTmp,flags=re.I)
                # replace filed = 'value' to filed = 'x'
                lineTmp = re.sub(r"(=|>|<|>=|<=)＼s*('|＼").*?＼2","＼＼1 'x'",lineTmp)
                # replace filed = value to filed = x
                lineTmp = re.sub(r"(=|>|<|>=|<=)＼s*[0-9]+","＼＼1 x",lineTmp)
                # replace like 'value' to like 'x'
                lineTmp = re.sub(r"like＼s+('|＼").*?＼1","like 'x'",lineTmp,flags=re.I)
                # replace in (value) to in (x)
                lineTmp = re.sub(r"in＼s+＼(.*?＼)","in (x)",lineTmp,flags=re.I)
                # replace limit x,y to limit
                lineTmp = re.sub(r"limit.*","limit",lineTmp,flags=re.I)

                print (lineTmp)

            match = 1
            lineTmp = lineMatch.group(0)
        else:   
            lineTmp += line
            match = 1

logFo.close()

使用方法：

analysis-general-log.py general.log | sort | uniq -c | sort -nr

示例输出：

1032 SELECT * FROM wp_comments WHERE ( comment_approved = 'x' OR comment_approved = 'x' ) AND comment_post_ID = x ORDER BY comment_date_gmt DESC
653 SELECT post_id, meta_key, meta_value FROM wp_postmeta WHERE post_id in (x) ORDER BY meta_id ASC
527 SELECT FOUND_ROWS()
438 SELECT t.*, tt.* FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id WHERE tt.taxonomy = 'x' AND t.term_id = x limit
341 SELECT option_value FROM wp_options WHERE option_name = 'x' limit
329 SELECT t.*, tt.*, tr.object_id FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON tt.term_id = t.term_id INNER JOIN wp_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy in (x) AND tr.object_id in (x) ORDER BY t.name ASC
311 SELECT wp_posts.* FROM wp_posts WHERE 1= x AND wp_posts.ID in (x) AND wp_posts.post_type = 'x' AND ((wp_posts.post_status = 'x')) ORDER BY wp_posts.post_date DESC
219 SELECT wp_posts.* FROM wp_posts WHERE ID in (x)
218 SELECT tr.object_id FROM wp_term_relationships AS tr INNER JOIN wp_term_taxonomy AS tt ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy in (x) AND tt.term_id in (x) ORDER BY tr.object_id ASC
217 SELECT wp_posts.* FROM wp_posts WHERE 1= x AND wp_posts.ID in (x) AND wp_posts.post_type = 'x' AND ((wp_posts.post_status = 'x')) ORDER BY wp_posts.menu_order ASC
202 SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1= x AND wp_posts.post_type = 'x' AND (wp_posts.post_status = 'x') ORDER BY wp_posts.post_date DESC limit
118 SET NAMES utf8
115 SET SESSION sql_mode= 'x'
115 SELECT @@SESSION.sql_mode
112 SELECT option_name, option_value FROM wp_options WHERE autoload = 'x'
111 SELECT user_id, meta_key, meta_value FROM wp_usermeta WHERE user_id in (x) ORDER BY umeta_id ASC
108 SELECT YEAR(min(post_date_gmt)) AS firstdate, YEAR(max(post_date_gmt)) AS lastdate FROM wp_posts WHERE post_status = 'x'
108 SELECT t.*, tt.* FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id WHERE tt.taxonomy in (x) AND tt.count > x ORDER BY tt.count DESC limit
107 SELECT t.*, tt.* FROM wp_terms AS t INNER JOIN wp_term_taxonomy AS tt ON t.term_id = tt.term_id WHERE tt.taxonomy in (x) AND t.term_id in (x) ORDER BY t.name ASC
107 SELECT * FROM wp_users WHERE ID = 'x'
106 SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1= x AND wp_posts.post_type = 'x' AND (wp_posts.post_status = 'x') AND post_date > 'x' ORDER BY wp_posts.post_date DESC limit
106 SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1= x AND wp_posts.post_type = 'x' AND (wp_posts.post_status = 'x') AND post_date > 'x' ORDER BY RAND() DESC limit
105 SELECT SQL_CALC_FOUND_ROWS wp_posts.ID FROM wp_posts WHERE 1= x AND wp_posts.post_type = 'x' AND (wp_posts.post_status = 'x') AND post_date > 'x' ORDER BY wp_posts.comment_count DESC limit

慢查询日志

获取锁的时间并不算在慢查询时间内，这个记录是在语句执行完后写入，所以也可能和执行顺序不一样。

Slow log 大概格式

CREATE TABLE `slow_log` (
  `start_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
  `user_host` mediumtext NOT NULL,
  `query_time` time(6) NOT NULL,
  `lock_time` time(6) NOT NULL,
  `rows_sent` int(11) NOT NULL,
  `rows_examined` int(11) NOT NULL,
  `db` varchar(512) NOT NULL,
  `last_insert_id` int(11) NOT NULL,
  `insert_id` int(11) NOT NULL,
  `server_id` int(10) unsigned NOT NULL,
  `sql_text` mediumblob NOT NULL,
  `thread_id` bigint(21) unsigned NOT NULL
) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='Slow log'

配置

查看相关参数：

mysql> show variables like '%quer%';
+----------------------------------------+---------------------------------------------------------------------+
| Variable_name                          | Value                                                               |
+----------------------------------------+---------------------------------------------------------------------+
| binlog_rows_query_log_events           | OFF                                                                 |
| ft_query_expansion_limit               | 20                                                                  |
| have_query_cache                       | YES                                                                 |
| log_queries_not_using_indexes          | OFF                                                                 |
| log_throttle_queries_not_using_indexes | 0                                                                   |
| long_query_time                        | 10.000000                                                           |
| query_alloc_block_size                 | 8192                                                                |
| query_cache_limit                      | 1048576                                                             |
| query_cache_min_res_unit               | 4096                                                                |
| query_cache_size                       | 1048576                                                             |
| query_cache_type                       | OFF                                                                 |
| query_cache_wlock_invalidate           | OFF                                                                 |
| query_prealloc_size                    | 8192                                                                |
| slow_query_log                         | OFF                                                                 |
| slow_query_log_file                    | D:\mysql-5.7.21-winx64\mysql-5.7.21-winx64\data\BIH-D-6885-slow.log |
+----------------------------------------+---------------------------------------------------------------------+
15 rows in set

1. log_queries_not_using_indexes:如果该值是ON，则会记录所有没有利用索引来进行查询的语句，前提是slow_query_log 的值也是ON,否则，不会记录; 和log_throttle_queries_not_using_indexes配合使用，设定每分钟记录到日志的未使用索引的语句数目，超过这个数目后只记录语句数量和花费的总时间
2. long_query_time 指定了慢查询的阈值，即执行语句的时间若超过这个值则为慢查询语句

这两种日志的维护

不像binlog那样，只要flush一下，就从新文件开始写。对于Generallog还有slowlog只是关闭原有文件再打开下。为了让这些日志在新的文件中写入，可以在flush之前，将文件重命名。这样在执行flush的时候，会重新生成一个文件：

shell> cd mysql-data-directory
shell> mv mysql.log mysql.old
shell> mv mysql-slow.log mysql-slow.old
shell> mysqladmin flush-logs