mysql的闪回工具binlog2sql试用

  数据库难免会被误操作，比如delete,drop等有害的语句被执行后，该如何恢复对dba是一个考验，如果备份可用的话，可以从备份中恢复数据来，不过这样做一般的成本有些大。

  binlog2sql工具是大众点评团队用python工具写的一个工具，通过解析binlog的sql，将sql语句反解析达到恢复数据的目的。如果没有这个工具，当然我们可以通过复杂的sed命令来手动生成恢复的语句，不过binglog2sql将这一步操作做了简便处理。

这篇blog是通过手动生成恢复语句：http://www.cnblogs.com/gomysql/p/3582058.html

binlog2sql官方文档：https://github.com/danfengcao/binlog2sql

下面进行安装和试用：

1.先配置好yum后安装git,yum的配置参考：https://www.cnblogs.com/yizhichun/p/6339742.html

2.安装binlogs2sql之前的准备工作

   首先需要安装好git和pip工具

    git的安装比较简单，网上有很多资料

[root@qht131 yum.repos.d]# yum install git-core

    不过在安装pip时出现了各种问题，好在最后安装成功了。

        官方安装文档：https://pip.pypa.io/en/stable/installing/

        安装pip之前需要先升级python文档：https://blog.csdn.net/see_you_see_me/article/details/78550977

        如果安装过程中出错了处理方法参考：http://www.zhimengzhe.com/linux/251710.html

[root@qht131 binlog2sql]# python -V
Python 2.7.14

[root@qht131 binlog2sql]# git --version
git version 1.7.1

[root@qht131 binlog2sql]# pip -V
pip 10.0.1 from /usr/local/python2.7/lib/python2.7/site-packages/pip (python 2.7)

3.安装binglog2sql工具

    shell> git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql

我在用git下载原文件的时候出错，通过安装curl及curl-devel来解决的

    yum install curl curl-devel  

[root@qht131 binlog2sql]# pip install -r requirements.txt

4.设置参数及帐户权限

[root@qht131 binlog2sql]# cat /etc/my.cnf
...
server-id=10000
log_bin = /u01/mysql/mysql_bin
max_binlog_size=1G
binlog_format = row
binlog_row_image = full

mysql> create user btwos@'localhost' identified by '123456';
Query OK, 0 rows affected (0.06 sec)

mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* to btwos@localhost;
Query OK, 0 rows affected (0.00 sec)

准备测试数据 

mysql> create table test (
    -> id int  not null auto_increment,
    -> name varchar(20) not null,
    -> create_time datetime not null,
    -> primary key (id));
Query OK, 0 rows affected (0.07 sec)

mysql> insert into test(name,create_time) values ('Xiaolin','2015-10-1'),('WangMac','2016-08-25'),('Jackli','2017-04-10');
Query OK, 3 rows affected (0.21 sec)
Records: 3  Duplicates: 0  Warnings: 0

mysql> select * from test;
+----+---------+---------------------+
| id | name    | create_time         |
+----+---------+---------------------+
|  1 | Xiaolin | 2015-10-01 00:00:00 |
|  2 | WangMac | 2016-08-25 00:00:00 |
|  3 | Jackli  | 2017-04-10 00:00:00 |
+----+---------+---------------------+
3 rows in set (0.02 sec)

5.测试delete以及update操作

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql_bin.000040 |     1212 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)

mysql> update test set create_time='2016-10-10' where name='WangMac';
Query OK, 1 row affected (0.03 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> delete from test where name='Jackli';
Query OK, 1 row affected (0.00 sec)

mysql> commit;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from test;
+----+---------+---------------------+
| id | name    | create_time         |
+----+---------+---------------------+
|  1 | Xiaolin | 2015-10-01 00:00:00 |
|  2 | WangMac | 2016-10-10 00:00:00 |
+----+---------+---------------------+
2 rows in set (0.00 sec)

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql_bin.000040 |     1772 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

修改了一笔数据，以及删除了最后一笔数据。

先查看一下mysqlbinlog解析出来的日志，从日志可以看到从建表到update和delete的所有操作。

[root@qht131 mysql]# mysqlbinlog --no-defaults -v -v mysql_bin.000040 > 000040.log

[root@qht131 mysql]# cat 000040.log
# at 706
#180426 11:01:35 server id 10000  end_log_pos 906 CRC32 0x6f60578e      Query   thread_id=3   exec_time=0     error_code=0
use `l5m`/*!*/;
SET TIMESTAMP=1524711695/*!*/;
create table test (
id int  not null auto_increment,
name varchar(20) not null,
create_time datetime not null,
primary key (id))
/*!*/;
# at 906
#180426 11:03:43 server id 10000  end_log_pos 971 CRC32 0x91159a2a      Anonymous_GTIDlast_committed=3        sequence_number=4       rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 971
#180426 11:03:43 server id 10000  end_log_pos 1042 CRC32 0xb26843e4     Query   thread_id=3   exec_time=0     error_code=0
SET TIMESTAMP=1524711823/*!*/;
BEGIN
/*!*/;
# at 1042
#180426 11:03:43 server id 10000  end_log_pos 1093 CRC32 0x72c273fe     Table_map: `l5m`.`test` mapped to number 108
# at 1093
#180426 11:03:43 server id 10000  end_log_pos 1181 CRC32 0xb0a6183b     Write_rows: table id 108 flags: STMT_END_F

BINLOG '
j0HhWhMQJwAAMwAAAEUEAAAAAGwAAAAAAAEAA2w1bQAEdGVzdAADAw8SAzwAAAD+c8Jy
j0HhWh4QJwAAWAAAAJ0EAAAAAGwAAAAAAAEAAgAD//gBAAAAB1hpYW9saW6Zl0IAAPgCAAAAB1dh
bmdNYWOZmjIAAPgDAAAABkphY2tsaZmcVAAAOximsA==
'/*!*/;
### INSERT INTO `l5m`.`test`
### SET
###   @1=1 /* INT meta=0 nullable=0 is_null=0 */
###   @2='Xiaolin' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */
###   @3='2015-10-01 00:00:00' /* DATETIME(0) meta=0 nullable=0 is_null=0 */
### INSERT INTO `l5m`.`test`
### SET
###   @1=2 /* INT meta=0 nullable=0 is_null=0 */
###   @2='WangMac' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */
###   @3='2016-08-25 00:00:00' /* DATETIME(0) meta=0 nullable=0 is_null=0 */
### INSERT INTO `l5m`.`test`
### SET
###   @1=3 /* INT meta=0 nullable=0 is_null=0 */
###   @2='Jackli' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */
###   @3='2017-04-10 00:00:00' /* DATETIME(0) meta=0 nullable=0 is_null=0 */
# at 1181
#180426 11:03:43 server id 10000  end_log_pos 1212 CRC32 0x9e3baa9e     Xid = 21
COMMIT/*!*/;
# at 1212
#180426 11:18:57 server id 10000  end_log_pos 1277 CRC32 0x7d82517c     Anonymous_GTIDlast_committed=4        sequence_number=5       rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 1277
#180426 11:18:57 server id 10000  end_log_pos 1348 CRC32 0x0ebea2ac     Query   thread_id=3   exec_time=0     error_code=0
SET TIMESTAMP=1524712737/*!*/;
BEGIN
/*!*/;
# at 1348
#180426 11:18:57 server id 10000  end_log_pos 1399 CRC32 0x86e375ff     Table_map: `l5m`.`test` mapped to number 108
# at 1399
#180426 11:18:57 server id 10000  end_log_pos 1471 CRC32 0xfb2db159     Update_rows: table id 108 flags: STMT_END_F

BINLOG '
IUXhWhMQJwAAMwAAAHcFAAAAAGwAAAAAAAEAA2w1bQAEdGVzdAADAw8SAzwAAAD/deOG
IUXhWh8QJwAASAAAAL8FAAAAAGwAAAAAAAEAAgAD///4AgAAAAdXYW5nTWFjmZoyAAD4AgAAAAdX
YW5nTWFjmZqUAABZsS37
'/*!*/;
### UPDATE `l5m`.`test`
### WHERE
###   @1=2 /* INT meta=0 nullable=0 is_null=0 */
###   @2='WangMac' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */
###   @3='2016-08-25 00:00:00' /* DATETIME(0) meta=0 nullable=0 is_null=0 */
### SET
###   @1=2 /* INT meta=0 nullable=0 is_null=0 */
###   @2='WangMac' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */
###   @3='2016-10-10 00:00:00' /* DATETIME(0) meta=0 nullable=0 is_null=0 */
# at 1471
#180426 11:18:57 server id 10000  end_log_pos 1502 CRC32 0x0f213682     Xid = 25
COMMIT/*!*/;
# at 1502
#180426 11:19:11 server id 10000  end_log_pos 1567 CRC32 0x15dd429a     Anonymous_GTIDlast_committed=5        sequence_number=6       rbr_only=yes
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 1567
#180426 11:19:11 server id 10000  end_log_pos 1638 CRC32 0x926675d7     Query   thread_id=3   exec_time=0     error_code=0
SET TIMESTAMP=1524712751/*!*/;
BEGIN
/*!*/;
# at 1638
#180426 11:19:11 server id 10000  end_log_pos 1689 CRC32 0x83a25989     Table_map: `l5m`.`test` mapped to number 108
# at 1689
#180426 11:19:11 server id 10000  end_log_pos 1741 CRC32 0xbb2e56b2     Delete_rows: table id 108 flags: STMT_END_F

BINLOG '
L0XhWhMQJwAAMwAAAJkGAAAAAGwAAAAAAAEAA2w1bQAEdGVzdAADAw8SAzwAAACJWaKD
L0XhWiAQJwAANAAAAM0GAAAAAGwAAAAAAAEAAgAD//gDAAAABkphY2tsaZmcVAAAslYuuw==
'/*!*/;
### DELETE FROM `l5m`.`test`
### WHERE
###   @1=3 /* INT meta=0 nullable=0 is_null=0 */
###   @2='Jackli' /* VARSTRING(60) meta=60 nullable=0 is_null=0 */
###   @3='2017-04-10 00:00:00' /* DATETIME(0) meta=0 nullable=0 is_null=0 */

下面使用binlog2sql来解析：

[root@qht131 binlog2sql]# python binlog2sql.py -h127.0.0.1 -P3306 -ubtwos -p '123456' -dl5m -ttest --start-file='mysql_bin.000040'
CREATE USER 'btwos'@'localhost' IDENTIFIED WITH 'mysql_native_password' AS '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'btwos'@'localhost';
USE l5m;
create table test (
id int  not null auto_increment,
name varchar(20) not null,
create_time datetime not null,
primary key (id));
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2015-10-01 00:00:00', 1, 'Xiaolin'); #start 906 end 1181 time 2018-04-26 11:03:43
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2016-08-25 00:00:00', 2, 'WangMac'); #start 906 end 1181 time 2018-04-26 11:03:43
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2017-04-10 00:00:00', 3, 'Jackli'); #start 906 end 1181 time 2018-04-26 11:03:43
UPDATE `l5m`.`test` SET `create_time`='2016-10-10 00:00:00', `id`=2, `name`='WangMac' WHERE `create_time`='2016-08-25 00:00:00' AND `id`=2 AND `name`='WangMac' LIMIT 1; #start 1212 end 1471 time 2018-04-26 11:18:57
DELETE FROM `l5m`.`test` WHERE `create_time`='2017-04-10 00:00:00' AND `id`=3 AND `name`='Jackli' LIMIT 1; #start 1502 end 1741 time 2018-04-26 11:19:11

完美呈现了所有以test的操作，并且还备注了在binlog中的position以及执行的时间

下面重点来了，解析出回滚的sql

[root@qht131 binlog2sql]# python binlog2sql.py --flashback -h127.0.0.1 -P3306 -ubtwos -p '123456' -dl5m -ttest --start-file='mysql_bin.000040'
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2017-04-10 00:00:00', 3, 'Jackli'); #start 1502 end 1741 time 2018-04-26 11:19:11
UPDATE `l5m`.`test` SET `create_time`='2016-08-25 00:00:00', `id`=2, `name`='WangMac' WHERE `create_time`='2016-10-10 00:00:00' AND `id`=2 AND `name`='WangMac' LIMIT 1; #start 1212 end 1471 time 2018-04-26 11:18:57
DELETE FROM `l5m`.`test` WHERE `create_time`='2017-04-10 00:00:00' AND `id`=3 AND `name`='Jackli' LIMIT 1; #start 906 end 1181 time 2018-04-26 11:03:43
DELETE FROM `l5m`.`test` WHERE `create_time`='2016-08-25 00:00:00' AND `id`=2 AND `name`='WangMac' LIMIT 1; #start 906 end 1181 time 2018-04-26 11:03:43
DELETE FROM `l5m`.`test` WHERE `create_time`='2015-10-01 00:00:00' AND `id`=1 AND `name`='Xiaolin' LIMIT 1; #start 906 end 1181 time 2018-04-26 11:03:43

解析出来的sql是根据时间戳倒序的，delete语句被flashback成了insert语句，update语句也可以成功将数据恢复到之前的状态，insert语句被flashback成了delete语句。

如果需要恢复数据的话只需要直接执行这些语句就可以了。

6.测试delete,truncate以及drop操作

mysql> flush logs;
Query OK, 0 rows affected (0.01 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql_bin.000041 |      154 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

mysql> use l5m
Database changed
mysql> delete from test where id<3;
Query OK, 2 rows affected (0.03 sec)

mysql> select * from test;
Empty set (0.00 sec)

mysql> insert into test (name,create_time) values('ChunkA','2019-10-08');
Query OK, 1 row affected (0.00 sec)

mysql> select * from test;
+----+--------+---------------------+
| id | name   | create_time         |
+----+--------+---------------------+
|  4 | ChunkA | 2019-10-08 00:00:00 |
+----+--------+---------------------+
1 row in set (0.00 sec)

mysql> truncate table test;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from test;
Empty set (0.00 sec)

mysql> drop table test;
Query OK, 0 rows affected (0.02 sec)

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql_bin.000041 |     1043 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

直接用binlog2sql来解析binlog

[root@qht131 binlog2sql]# python binlog2sql.py -h127.0.0.1 -P3306 -ubtwos -p '123456' -dl5m -ttest --start-file='mysql_bin.000041'
USE l5m;
truncate table test;
USE l5m;
DROP TABLE `test` /* generated by server */;

发现解析不出来truncate之前delete和insert操作，应该是由于information_schema.tables已没有test表的信息，导致解析不出来。

那如果我重新把test表建立好之后再用binlog2sql来解析行不行？

mysql> create table test (
    -> id int  not null auto_increment,
    -> name varchar(20) not null,
    -> create_time datetime not null,
    -> primary key (id));
Query OK, 0 rows affected (0.03 sec)

[root@qht131 binlog2sql]# python binlog2sql.py -h127.0.0.1 -P3306 -ubtwos -p '123456' -dl5m -ttest --start-file='mysql_bin.000041'
DELETE FROM `l5m`.`test` WHERE `create_time`='2015-10-01 00:00:00' AND `id`=1 AND `name`='Xiaolin' LIMIT 1; #start 4 end 412 time 2018-04-26 11:55:27
DELETE FROM `l5m`.`test` WHERE `create_time`='2016-10-10 00:00:00' AND `id`=2 AND `name`='WangMac' LIMIT 1; #start 4 end 412 time 2018-04-26 11:55:27
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2019-10-08 00:00:00', 4, 'ChunkA'); #start 443 end 682 time 2018-04-26 11:56:10
USE l5m;
truncate table test;
USE l5m;
DROP TABLE `test` /* generated by server */;
USE l5m;
create table test (
id int  not null auto_increment,
name varchar(20) not null,
create_time datetime not null,
primary key (id));

没有问题，binlog2sql倒序着把binlog的所有操作都解析出来了。

接着就好flashback了生成回滚sql了：

[root@qht131 binlog2sql]# python binlog2sql.py --flashback -h127.0.0.1 -P3306 -ubtwos -p '123456' -dl5m -ttest --start-file='mysql_bin.000041'
DELETE FROM `l5m`.`test` WHERE `create_time`='2019-10-08 00:00:00' AND `id`=4 AND `name`='ChunkA' LIMIT 1; #start 443 end 682 time 2018-04-26 11:56:10
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2016-10-10 00:00:00', 2, 'WangMac'); #start 4 end 412 time 2018-04-26 11:55:27
INSERT INTO `l5m`.`test`(`create_time`, `id`, `name`) VALUES ('2015-10-01 00:00:00', 1, 'Xiaolin'); #start 4 end 412 time 2018-04-26 11:55:27

可以看到，此时批量 delete 和 insert 的语句被解析出来了，并且能够成功生成回滚的 SQL。

但是，DDL 语句，在整个测试过程中都是无法被回滚的，这也在我们的预期中，至此，MySQL 闪回工具 -- binlog2sql 的基本测试就告一段落了。

参考：https://www.cnblogs.com/glon/p/6856192.html

         https://github.com/danfengcao/binlog2sql