第七章节 日志管理
1. 错误日志
1.1 作用
记录了从数据库启动、工作、关闭期间,状态、警告、错误。
1.2 配置方法
默认就开启,存放在数据目录下,名字为:HOSTNAME.err
mysql> select @@datadir;
+------------------+
| @@datadir |
+------------------+
| /data/3306/data/ |
+------------------+
1 row in set (0.00 sec)
mysql> system hostname;
db01
mysql> select @@log_error;
+-------------+
| @@log_error |
+-------------+
| ./db01.err |
+-------------+
1 row in set (0.00 sec)
mysql>
如何修改:
[root@db01 ~]# mkdir -p /data/3306/errorlog/
[root@db01 ~]# touch /data/3306/errorlog/mysql.log
[root@db01 ~]# chown -R mysql.mysql /data/*
vim /etc/my.cnf
log_error=/data/3306/errorlog/mysql.log
1.3 如何使用
[root@db01 errorlog]# chown -R root.root /data/3306/data/ibdata*
[root@db01 errorlog]# /etc/init.d/mysqld restart
Shutting down MySQL.. SUCCESS!
Starting MySQL.. ERROR! The server quit without updating PID file (/data/3306/data/db01.pid).
2020-07-09T06:47:42.282201Z 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2020-07-09T06:47:42.282234Z 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
2020-07-09T06:47:42.282247Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-07-09T06:47:42.888995Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-07-09T06:47:42.889063Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-07-09T06:47:42.889073Z 0 [ERROR] Failed to initialize builtin plugins.
2020-07-09T06:47:42.889090Z 0 [ERROR] Aborting
[root@db01 errorlog]# vim /data/3306/errorlog/mysql.log
[root@db01 errorlog]# ll -l /data/3306/data/ibdata*
-rw-r----- 1 root root 104857600 Jul 9 14:47 /data/3306/data/ibdata1
-rw-r----- 1 root root 104857600 Jul 8 15:04 /data/3306/data/ibdata2
-rw-r----- 1 root root 104857600 Jul 8 15:04 /data/3306/data/ibdata3
[root@db01 errorlog]# chown -R mysql.mysql /data/3306/data/ibdata*
[root@db01 errorlog]# /etc/init.d/mysqld restart
ERROR! MySQL server PID file could not be found!
Starting MySQL.. SUCCESS!
[root@db01 errorlog]#
2. binlog 二进制日志
2.1 作用
记录数据库发生的变化的操作(DDL、DCL、DML...)的逻辑日志(SQL)。
数据恢复。
主从复制。
2.2 配置
8.0之前,默认没有打开。需要手工开启。
基本参数:
server_id=6
log_bin=/data/3306/binlog/mysql-bin ---> mysql-bin.000001
创建目录和授权 :
mkdir -p /data/3306/binlog/
chown -R mysql.mysql /data/*
2.3 二进制日志记录格式
a. 事件方式记录。
b. 事件内容格式:
DDL和DCL语句: 内容为语句本身。SQL Statement 方式。
DML语句:
1. RBR : 默认模式,行模式。数据行的变化。
2. SBR :语句模式。记录SQL语句
3. MBR :混合模式。一般不用。
补充: 只记录提交的事务日志。
c. 面试题: RBR和SBR的区别?
update t1 set name='zs' where id<100; 100行更新。
RBR记录: 每行的变化日志。
SBR记录: update t1 set name='zs' where id<100;
RBR: 日志量大。可读性差。记录准确。高级特性依赖于RBR。
SBR: 日志量小。可读性强。记录不准确(函数类操作时)。
d. 格式参数:
mysql> select @@binlog_format;
+-----------------+
| @@binlog_format |
+-----------------+
| ROW |
+-----------------+
2.4 二进制日志查看
2.4.1 查看配置信息
mysql> show variables like '%log_bin%';
+---------------------------------+-----------------------------------+
| Variable_name | Value |
+---------------------------------+-----------------------------------+
| log_bin | ON |
| log_bin_basename | /data/3306/binlog/mysql-bin
2.4.2 查看存放位置
[root@db01 binlog]# ll /data/3306/binlog/
total 8
-rw-r----- 1 mysql mysql 154 Jul 9 15:06 mysql-bin.000001
-rw-r----- 1 mysql mysql 35 Jul 9 15:06 mysql-bin.index
2.4.3 查看所有二进制日志
mysql> flush logs;
Query OK, 0 rows affected (0.01 sec)
mysql> show binary logs;
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| mysql-bin.000001 | 201 |
| mysql-bin.000002 | 201 |
| mysql-bin.000003 | 154 |
+------------------+-----------+
2.4.4 查看二进制正在使用的文件
mysql> show master status ;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000003 | 319 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
2.4.5 查看二进制日志事件
mysql> create database oldguo;
Query OK, 1 row affected (0.00 sec)
mysql> show binlog events in 'mysql-bin.000003';
+------------------+-----+----------------+-----------+-------------+---------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+-----+----------------+-----------+-------------+---------------------------------------+
| mysql-bin.000003 | 4 | Format_desc | 6 | 123 | Server ver: 5.7.28-log, Binlog ver: 4 |
| mysql-bin.000003 | 123 | Previous_gtids | 6 | 154 | |
| mysql-bin.000003 | 154 | Anonymous_Gtid | 6 | 219 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| mysql-bin.000003 | 219 | Query | 6 | 319 | create database oldguo |
+------------------+-----+----------------+-----------+-------------+---------------------------------------+
4 rows in set (0.00 sec)
Pos : 事件开始的position号
End_log_pos :事件结束的position号
Info :日志内容信息。
2.5 二进制日志应用
2.5.1 日志截取命令: mysqlbinlog
a.日志内容基本查看:
[root@db01 ~]# mysqlbinlog /data/3306/binlog/mysql-bin.000003
......省略若干行
# at 154
...
# at 219
#200709 15:28:51 server id 6 end_log_pos 319
.....省略若干行
create database oldguo
...
# at 319
....省略若干行
# at 843
#200709 15:35:36 server id 6 end_log_pos 917 CRC32 0x31444dd7 Query thread_id=3 exec_time=0 error_code=0
BEGIN
# at 917
#200709 15:35:36 server id 6 end_log_pos 964 CRC32 0xaf030924 Table_map: `oldguo`.`t1` mapped to number 108
# at 964
#200709 15:35:36 server id 6 end_log_pos 1004 CRC32 0x9303b113 Write_rows: table id 108 flags: STMT_END_FBINLOG'yMgGXxMGAAAALwAAAMQDAAAAAGwAAAAAAAEABm9sZGd1bwACdDEAAQMAASQJA68=yMgGXx4GAAAAKAAAAOwDAAAAAGwAAAAAAAEAAgAB//4BAAAAE7EDkw=='
# at 1004
...... 省略若干行
b. 解析row的DML操作
[root@db01 ~]# mysqlbinlog --base64-output=decode-rows -vvv /data/3306/binlog/mysql-bin.000003
### INSERT INTO `oldguo`.`t1`
### SET
### @1=1 /* INT meta=0 nullable=1 is_null=0 */
c. 截取oldboy库下的日志
[root@db01 ~]# mysqlbinlog -d oldboy --base64-output=decode-rows -vvv /data/3306/binlog/mysql-bin.000003
d.按时间截取日志信息
--start-datetime='2020-07-09 16:00:00'
--stop-datetime=''
注意: 最多精确到秒。做精确截取会有问题。
mysqlbinlog --start-datetime='2020-07-09 16:00:00' /data/3306/binlog/mysql-bin.000003
e. 按position号码截取日志
--start-position
--stop-position
mysqlbinlog --start-position=1701 --stop-position=1801 /data/3306/binlog/mysql-bin.000003
2.5.2 通过截取命令恢复数据故障演练
a. 准备环境
mysql>
mysql> flush logs;
Query OK, 0 rows affected (0.02 sec)
mysql> show master status ;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000004 | 154 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)
mysql> create database bindb;
Query OK, 1 row affected (0.00 sec)
mysql> use bindb;
Database changed
mysql> create table t1 (id int);
Query OK, 0 rows affected (0.02 sec)
mysql> insert into t1 values(1),(2),(3);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into t1 values(11),(22),(33);
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
b. 搞破坏
mysql> drop database bindb;
c. 通过二进制日志恢复
截取思路:
起点: 建库操作
终点:删库之前
mysql> show master status ;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000004 | 1171 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> show binlog events in 'mysql-bin.000004';
.....
mysql-bin.000004 | 219 | Query | 6 | 316 | create database bindb
.....
| mysql-bin.000004 | 1076 | Query | 6 | 1171 | drop database bindb
起点: 219
终点: 1076
d. 截取日志 并 恢复
[root@db01 ~]# mysqlbinlog --start-position=219 --stop-position=1076 /data/3306/binlog/mysql-bin.000004 >/tmp/1.sql
mysql> set sql_log_bin=0;
mysql> source /tmp/1.sql;
mysql> set sql_log_bin=1;
mysql> use bindb;
mysql> select *from t1;
+------+
| id |
+------+
| 1 |
| 2 |
| 3 |
| 11 |
| 22 |
| 33 |
+------+
2.5.3 思考问题 : 以上如果是在生产环境中,binlog恢复会有哪些痛点?
a. 假如bindb是两年前已经创建了。
解决方案:数据备份+binlog进行恢复。
b. 跨多个文件该怎么截取。
000001 700 ~ end
000002 start ~ end
000003 start ~ 2000
解决方案:
1. 分段截取 ,追加方式存储。 √
2. 按时间截取,但有可能不准确。 X
3. GTID √
c. 如果是一张表被误删除。
000003 200 10000
解决方案:
1. 自己写grep命令
2. 工具:binlog2sql
d. binlog文件特别大怎么办
mysql> pager grep "bindb";
PAGER set to 'grep "bindb"'
mysql> show binlog events in 'mysql-bin.000004';
mysql> pager less
mysql> show binlog events in 'mysql-bin.000004';
[root@db01 binlog]# mysql -e "show binlog events in 'mysql-bin.000004';"|grep bindb;
2.6 gtid 模式 binlog管理
2.6.1 什么是GTID ?
全局事务ID。
5.6+版本以上具备。5.7+ 增强,强烈建议使用(主要针对主从复制)。
GTID具备幂等性。
2.6.2 如何构成?
server_uuid:N
[root@db01 data]# cat auto.cnf
[auto]
server-uuid=4d98ed45-c0e9-11ea-8dd7-000c295bb94f
2.6.3 开启GTID
gtid_mode=on
enforce_gtid_consistency=on
log_slave_updates=on
2.6.4 查看GTID信息
mysql> create database gtdb;
Query OK, 1 row affected (0.00 sec)
mysql> use gtdb;
Database changed
mysql> create table t1 (id int);
Query OK, 0 rows affected (0.01 sec)
mysql> insert into t1 values(1),(2),(3);
Query OK, 3 rows affected (0.01 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> commit;
Query OK, 0 rows affected (0.01 sec)
mysql> show master status ;
+------------------+----------+--------------+------------------+------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------+
| mysql-bin.000005 | 739 | | | 4d98ed45-c0e9-11ea-8dd7-000c295bb94f:1-3 |
+------------------+----------+--------------+------------------+------------------------------------------+
1 row in set (0.00 sec)
mysql> select @@server_uuid;
+--------------------------------------+
| @@server_uuid |
+--------------------------------------+
| 4d98ed45-c0e9-11ea-8dd7-000c295bb94f |
+--------------------------------------+
1 row in set (0.00 sec)
mysql> show binlog events in 'mysql-bin.000005';
+------------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+------------------+-----+----------------+-----------+-------------+-------------------------------------------------------------------+
| mysql-bin.000005 | 4 | Format_desc | 6 | 123 | Server ver: 5.7.28-log, Binlog ver: 4 |
| mysql-bin.000005 | 123 | Previous_gtids | 6 | 154 | |
| mysql-bin.000005 | 154 | Gtid | 6 | 219 | SET @@SESSION.GTID_NEXT= '4d98ed45-c0e9-11ea-8dd7-000c295bb94f:1' |
| mysql-bin.000005 | 219 | Query | 6 | 313 | create database gtdb |
| mysql-bin.000005 | 313 | Gtid | 6 | 378 | SET @@SESSION.GTID_NEXT= '4d98ed45-c0e9-11ea-8dd7-000c295bb94f:2' |
| mysql-bin.000005 | 378 | Query | 6 | 476 | use `gtdb`; create table t1 (id int) |
| mysql-bin.000005 | 476 | Gtid | 6 | 541 | SET @@SESSION.GTID_NEXT= '4d98ed45-c0e9-11ea-8dd7-000c295bb94f:3' |
| mysql-bin.000005 | 541 | Query | 6 | 613 | BEGIN |
| mysql-bin.000005 | 613 | Table_map | 6 | 658 | table_id: 108 (gtdb.t1) |
| mysql-bin.000005 | 658 | Write_rows | 6 | 708 | table_id: 108 flags: STMT_END_F |
| mysql-bin.000005 | 708 | Xid | 6 | 739 | COMMIT /* xid=10 */ |
+------------------+-----+----------------+-----------+-------------+--------------------------------
2.6.5 GTID方式截取和恢复日志
mysql> drop database gtdb;
Query OK, 1 row affected (0.01 sec)
a. 确认GTID起点终点
4d98ed45-c0e9-11ea-8dd7-000c295bb94f:1-3
b. 截取和恢复
错误截取的方法:
[root@db01 data]# mysqlbinlog --include-gtids='4d98ed45-c0e9-11ea-8dd7-000c295bb94f:1-3' /data/3306/binlog/mysql-bin.000005 >/tmp/1.sql
mysql> set sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)
mysql> source /tmp/1.sql;
正确的方法:
mysqlbinlog --skip-gtids --include-gtids='4d98ed45-c0e9-11ea-8dd7-000c295bb94f:1-3' /data/3306/binlog/mysql-bin.000005 >/tmp/2.sql
--exclude-gtids 参数说明:
--include-gtids='4d98ed45-c0e9-11ea-8dd7-000c295bb94f:1-10' --exclude-gtids='4d98ed45-c0e9-11ea-8dd7-000c295bb94f:6'
2.7 binlog的其他操作
2.7.1 binlog 滚动
mysql> flush logs; ---》 备份时会手工触发
[root@db01 binlog]# mysqladmin flush-logs;
[root@db01 binlog]# /etc/init.d/mysqld restart
[root@db01 binlog]# mysqldump -F -A >/tmp/full.sql
mysql> select @@max_binlog_size/1024/1024 as MB;
+---------------+
| MB |
+---------------+
| 1024.00000000 |
+---------------+
2.7.2 binlog的删除
a. 过期时间
mysql> select @@expire_logs_days;
mysql> set global expire_logs_days=15;
Query OK, 0 rows affected (0.00 sec)
mysql> select @@expire_logs_days;
+--------------------+
| @@expire_logs_days |
+--------------------+
| 15 |
+--------------------+
1 row in set (0.00 sec)
说明: 最小值,一轮全备时间+1 ,生产建议至少两轮全备时间+1
b. 手工清理
Examples:
PURGE BINARY LOGS TO 'mysql-bin.010';
PURGE BINARY LOGS BEFORE '2019-04-02 22:46:26';
c. 重置二进制日志
mysql> reset master;
Query OK, 0 rows affected (0.00 sec)
mysql> show master status ;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 | 154 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
说明: 数据恢复后,会使用。
2.7.3 远程实时备份binlog
mkdir -p /data/backup/binlog
cd /data/backup/binlog
mysqlbinlog -R --host=10.0.0.51 --user=root --password=123 --raw --stop-never mysql-bin.000001 &
3. slowlog 慢日志
3.1 作用
记录MySQL中执行较慢的SQL。SQL优化的工具日志。
3.2 配置
默认: 没有开。
mysql> select @@slow_query_log;
mysql> select @@slow_query_log_file;
mysql> select @@long_query_time;
mysql> select @@log_queries_not_using_indexes;
设定方法:
vim /etc/my.cnf
slow_query_log=ON
slow_query_log_file=/data/3306/slowlog/slow.log
long_query_time=0.01
log_queries_not_using_indexes
mkdir -p /data/3306/slowlog/
chown -R mysql.mysql /data/*
[root@db01 3306]# /etc/init.d/mysqld restart
Shutting down MySQL............ SUCCESS!
Starting MySQL.. SUCCESS!
[root@db01 3306]#
在线调整:
set global slow_query_log=ON/OFF
set global long_query_time=0.01
3.3 自行模拟一些慢语句
3.4 mysqldumpslow 语句分析慢日志
[root@db01 slowlog]# mysqldumpslow -s c -t 3 slow.log
Reading mysql slow query log from slow.log
Count: 11 Time=0.00s (0s) Lock=0.00s (0s) Rows=100.0 (1100), root[root]@localhost
select * from t100w where k1 <'S' limit N
Count: 8 Time=0.71s (5s) Lock=0.00s (0s) Rows=10.0 (80), root[root]@localhost
select group_concat(k1),k2,sum(num) from t100w where k2<'S' group by k2 limit N
Count: 7 Time=1.17s (8s) Lock=0.00s (0s) Rows=36.0 (252), root[root]@localhost
select group_concat(k1),k2,sum(num) from t100w group by k2 limit N
[root@db01 slowlog]#
其他的分析工具:
pt-query-digest