MySQL 运维管理工具--pt(Percona-toolkit)

文章目录

  • 一.检查和安装与Perl相关的模块
  • 二.下载和安装percona toolkit的包
    • 2.1 下载和安装percona toolkit的包
    • 2.2 安装percona-toolkit工具包
    • 2.3 percona-toolkit工具介绍
      • 2.3.1 pt-archiver(归档表)
      • 2.3.2 pt-online-schema-change(在线修改表结构)
      • 2.3.3 pt-table-checksum
      • 2.3.4 pt-table-sync
      • 2.3.5 pt-deadlock-logger(死锁检测)
      • 2.3.6 pt-duplicate-key-checker(主键冲突检测)
      • 2.3.7 pt-kill(杀进程)
        • 2.3.7.1 测试kill空闲链接
        • 2.3.7.2 kill查询时间超过20秒的进程
        • 2.3.7.3 Kill掉 select IFNULl.*语句开头的SQL
  • 参考:

一.检查和安装与Perl相关的模块

PT工具是使用Perl语言编写和执行的,所以需要系统中有Perl环境。
依赖包检查命令为:

yum -y install perl-DBI perl-DBD-MySQL perl-Time-HiRes perl-IO-Socket-SSL 

二.下载和安装percona toolkit的包

2.1 下载和安装percona toolkit的包

参照:https://www.percona.com/doc/percona-toolkit/3.0/installation.html#installing-percona-toolkit-on-red-hat-or-centos

yum -y install http://www.percona.com/downloads/percona-release/redhat/0.1-4/percona-release-0.1-4.noarch.rpm

查看可以安装的包

[root@10-31-1-119 src]# yum list | grep percona-toolkit
percona-toolkit.noarch                      2.2.20-1                   percona-release-noarch
percona-toolkit.x86_64                      3.2.1-1.el7                percona-release-x86_64
percona-toolkit-debuginfo.x86_64            3.0.13-1.el7               percona-release-x86_64
[root@10-31-1-119 src]# 

2.2 安装percona-toolkit工具包

yum install percona-toolkit

遇到报错

[root@10-31-1-119 src]# yum install percona-toolkit -y
已加载插件:fastestmirror, langpacks
Loading mirror speeds from cached hostfile
正在解决依赖关系
--> 正在检查事务
---> 软件包 percona-toolkit.x86_64.0.3.2.1-1.el7 将被 安装
--> 正在处理依赖关系 perl(Digest::MD5),它被软件包 percona-toolkit-3.2.1-1.el7.x86_64 需要
--> 正在检查事务
---> 软件包 perl-Digest-MD5.x86_64.0.2.52-3.el7 将被 安装
--> 正在处理依赖关系 perl(Digest::base) >= 1.00,它被软件包 perl-Digest-MD5-2.52-3.el7.x86_64 需要
--> 正在检查事务
---> 软件包 perl-Digest.noarch.0.1.17-245.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

============================================================================================================================================================================================================
 Package                                           架构                                     版本                                             源                                                        大小
============================================================================================================================================================================================================
正在安装:
 percona-toolkit                                   x86_64                                   3.2.1-1.el7                                      percona-release-x86_64                                    17 M
为依赖而安装:
 perl-Digest                                       noarch                                   1.17-245.el7                                     base                                                      23 k
 perl-Digest-MD5                                   x86_64                                   2.52-3.el7                                       base                                                      30 k

事务概要
============================================================================================================================================================================================================
安装  1 软件包 (+2 依赖软件包)

总下载量:17 M
安装大小:17 M
Downloading packages:
(1/3): perl-Digest-MD5-2.52-3.el7.x86_64.rpm                                                                                                                                         |  30 kB  00:00:00     
(2/3): perl-Digest-1.17-245.el7.noarch.rpm                                                                                                                                           |  23 kB  00:00:00     
percona-toolkit-3.2.1-1.el7.x8 FAILED                                                   51% [=======================================-                                     ]  4.8 B/s | 8.6 MB 487:30:28 ETA 
http://repo.percona.com/release/7/RPMS/x86_64/percona-toolkit-3.2.1-1.el7.x86_64.rpm: [Errno 12] Timeout on http://repo.percona.com/release/7/RPMS/x86_64/percona-toolkit-3.2.1-1.el7.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
正在尝试其它镜像。
percona-toolkit-3.2.1-1.el7.x8 FAILED                                                   96% [==========================================================================   ]  5.7 B/s |  16 MB  31:35:32 ETA 
http://repo.percona.com/release/7/RPMS/x86_64/percona-toolkit-3.2.1-1.el7.x86_64.rpm: [Errno 12] Timeout on http://repo.percona.com/release/7/RPMS/x86_64/percona-toolkit-3.2.1-1.el7.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
正在尝试其它镜像。
warning: /var/cache/yum/x86_64/7/percona-release-x86_64/packages/percona-toolkit-3.2.1-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 8507efa5: NOKEY==========-] 3.0 kB/s |  17 MB  00:00:01 ETA 
percona-toolkit-3.2.1-1.el7.x86_64.rpm 的公钥尚未安装
(3/3): percona-toolkit-3.2.1-1.el7.x86_64.rpm                                                                                                                                        |  17 MB  00:01:59     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
总计                                                                                                                                                                        4.8 kB/s |  17 MB  00:59:47     
从 file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Percona 检索密钥
导入 GPG key 0xCD2EFD2A:
 用户ID     : "Percona MySQL Development Team "
 指纹       : 430b df5c 56e7 c94e 848e e60c 1c4c bdcd cd2e fd2a
 软件包     : percona-release-0.1-4.noarch (@/percona-release-0.1-4.noarch)
 来自       : /etc/pki/rpm-gpg/RPM-GPG-KEY-Percona


percona-toolkit-3.2.1-1.el7.x86_64.rpm 的公钥尚未安装


 失败的软件包是:percona-toolkit-3.2.1-1.el7.x86_64
 GPG  密钥配置为:file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Percona

解决方案

[root@10-31-1-119 src]# yum update percona-release
已加载插件:fastestmirror, langpacks
Loading mirror speeds from cached hostfile
正在解决依赖关系
--> 正在检查事务
---> 软件包 percona-release.noarch.0.0.1-4 将被 升级
---> 软件包 percona-release.noarch.0.1.0-24 将被 更新
--> 解决依赖关系完成

依赖关系解决

============================================================================================================================================================================================================
 Package                                             架构                                       版本                                       源                                                          大小
============================================================================================================================================================================================================
正在更新:
 percona-release                                     noarch                                     1.0-24                                     percona-release-noarch                                      19 k

事务概要
============================================================================================================================================================================================================
升级  1 软件包

总下载量:19 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for percona-release-noarch
percona-release-1.0-24.noarch.rpm                                                                                                                                                    |  19 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在更新    : percona-release-1.0-24.noarch                                                                                                                                                           1/2 
* Enabling the Percona Original repository
<*> All done!
* Enabling the Percona Release repository
<*> All done!
The percona-release package now contains a percona-release script that can enable additional repositories for our newer products.

For example, to enable the Percona Server 8.0 repository use:

  percona-release setup ps80

Note: To avoid conflicts with older product versions, the percona-release setup command may disable our original repository for some products.

For more information, please visit:
  https://www.percona.com/doc/percona-repo-config/percona-release.html

  清理        : percona-release-0.1-4.noarch                                                                                                                                                            2/2 
  验证中      : percona-release-1.0-24.noarch                                                                                                                                                           1/2 
  验证中      : percona-release-0.1-4.noarch                                                                                                                                                            2/2 

更新完毕:
  percona-release.noarch 0:1.0-24                                                                                                                                                                           

完毕!
[root@10-31-1-119 src]# 
[root@10-31-1-119 src]# yum install percona-toolkit -y
已加载插件:fastestmirror, langpacks
Loading mirror speeds from cached hostfile
percona-release-noarch                                                                                                                                                               | 2.9 kB  00:00:00     
percona-release-x86_64                                                                                                                                                               | 2.9 kB  00:00:00     
prel-release-x86_64                                                                                                                                                                  | 2.9 kB  00:00:00     
prel-release-x86_64/7/primary_db                                                                                                                                                     | 1.1 kB  00:00:00     
正在解决依赖关系
--> 正在检查事务
---> 软件包 percona-toolkit.x86_64.0.3.2.1-1.el7 将被 安装
--> 正在处理依赖关系 perl(Digest::MD5),它被软件包 percona-toolkit-3.2.1-1.el7.x86_64 需要
--> 正在检查事务
---> 软件包 perl-Digest-MD5.x86_64.0.2.52-3.el7 将被 安装
--> 正在处理依赖关系 perl(Digest::base) >= 1.00,它被软件包 perl-Digest-MD5-2.52-3.el7.x86_64 需要
--> 正在检查事务
---> 软件包 perl-Digest.noarch.0.1.17-245.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

============================================================================================================================================================================================================
 Package                                           架构                                     版本                                             源                                                        大小
============================================================================================================================================================================================================
正在安装:
 percona-toolkit                                   x86_64                                   3.2.1-1.el7                                      percona-release-x86_64                                    17 M
为依赖而安装:
 perl-Digest                                       noarch                                   1.17-245.el7                                     base                                                      23 k
 perl-Digest-MD5                                   x86_64                                   2.52-3.el7                                       base                                                      30 k

事务概要
============================================================================================================================================================================================================
安装  1 软件包 (+2 依赖软件包)

总计:17 M
安装大小:17 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/percona-release-x86_64/packages/percona-toolkit-3.2.1-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 8507efa5: NOKEY
从 file:///etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY 检索密钥
导入 GPG key 0x8507EFA5:
 用户ID     : "Percona MySQL Development Team (Packaging key) "
 指纹       : 4d1b b29d 63d9 8e42 2b21 13b1 9334 a25f 8507 efa5
 软件包     : percona-release-1.0-24.noarch (@percona-release-noarch)
 来自       : /etc/pki/rpm-gpg/PERCONA-PACKAGING-KEY
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在安装    : perl-Digest-1.17-245.el7.noarch                                                                                                                                                         1/3 
  正在安装    : perl-Digest-MD5-2.52-3.el7.x86_64                                                                                                                                                       2/3 
  正在安装    : percona-toolkit-3.2.1-1.el7.x86_64                                                                                                                                                      3/3 
  验证中      : perl-Digest-1.17-245.el7.noarch                                                                                                                                                         1/3 
  验证中      : percona-toolkit-3.2.1-1.el7.x86_64                                                                                                                                                      2/3 
  验证中      : perl-Digest-MD5-2.52-3.el7.x86_64                                                                                                                                                       3/3 

已安装:
  percona-toolkit.x86_64 0:3.2.1-1.el7                                                                                                                                                                      

作为依赖被安装:
  perl-Digest.noarch 0:1.17-245.el7                                                                   perl-Digest-MD5.x86_64 0:2.52-3.el7                                                                  

完毕!
[root@10-31-1-119 src]# 

2.3 percona-toolkit工具介绍

有的32个命令,可以分为7大类

工具类别 工具命令 工具作用 备注
开发类 pt-duplicate-key-checker 列出并删除重复的索引和外键
开发类 pt-online-schema-change 在线修改表结构
开发类 pt-query-advisor 分析查询语句,并给出建议,有bug 已废弃
开发类 pt-show-grants 规范化和打印权限
开发类 pt-upgrade 在多个服务器上执行查询,并比较不同
性能类 pt-index-usage 分析日志中索引使用情况,并出报告
性能类 pt-pmp 为查询结果跟踪,并汇总跟踪结果
性能类 pt-visual-explain 格式化执行计划
性能类 pt-table-usage 分析日志中查询并分析表使用情况
配置类 pt-config-diff 比较配置文件和参数
配置类 pt-mysql-summary 对mysql配置和status进行汇总
配置类 pt-variable-advisor 分析参数,并提出建议
监控类 pt-deadlock-logger 提取和记录mysql死锁信息
监控类 pt-fk-error-logger 提取和记录外键信息
监控类 pt-mext 并行查看status样本信息
监控类 pt-query-digest 分析查询日志,并产生报告 常用命令
监控类 pt-trend 按照时间段读取slow日志信息 已废弃
复制类 pt-heartbeat 监控mysql复制延迟
复制类 pt-slave-delay 设定从落后主的时间
复制类 pt-slave-find 查找和打印所有mysql复制层级关系
复制类 pt-slave-restart 监控salve错误,并尝试重启salve
复制类 pt-table-checksum 校验主从复制一致性
复制类 pt-table-sync 高效同步表数据
系统类 pt-diskstats 查看系统磁盘状态
系统类 pt-fifo-split 模拟切割文件并输出
系统类 pt-summary 收集和显示系统概况
系统类 pt-stalk 出现问题时,收集诊断数据
系统类 pt-sift 浏览由pt-stalk创建的文件
系统类 pt-ioprofile 查询进程IO并打印一个IO活动表
实用类 pt-archiver 将表数据归档到另一个表或文件中
实用类 pt-find 查找表并执行命令
实用类 pt-kill Kill掉符合条件的sql 常用命令
实用类 pt-align 对齐其他工具的输出
实用类 pt-fingerprint 将查询转成密文

2.3.1 pt-archiver(归档表)

# 重要参数
--limit 100       每次取100行数据用pt-archive处理    
--txn-size  100   设置100行为一个事务提交一次,    
--where 'id<3000'  设置操作条件    
--progress 5000     每处理5000行输出一次处理信息    
--statistics       输出执行过程及最后的操作统计。(只要不加上--quiet,默认情况下pt- archive都会输出执行过程的)    
--charset=UTF8     指定字符集为UTF8—这个最后加上不然可能出现乱码。    
--bulk-delete      批量删除source上的旧数据(例如每次1000行的批量删除操作)
使用案例:
1.归档到数据库

pt-archiver --source h=10.0.0.11,D=world,t=city,u=root,p=123 --dest h=10.0.0.11,D=world,t=city2,u=root,p=123 --where 'id<1000' --no-check-charset --no-delete --limit=100 --commit-each --progress 200 --statistics

2.只清理数据

pt-archiver --source h=127.0.0.1,D=world,t=city2,u=root,p=123 --where 'id<100' --purge --limit=1 --no-check-charset

3.只把数据导出到外部文件,但是不删除源表里的数据
pt-archiver --source h=10.0.0.11,D=world,t=city,u=root,p=123 --where '1=1' --no-check-charset --no-delete --file="/tmp/archiver.dat" 

2.3.2 pt-online-schema-change(在线修改表结构)

MySQL 5.6之后支持online DDL之后,可以直接在线修改表结构

pt-osc工作流程:
1、检查更改表是否有主键或唯一索引,是否有触发器
2、检查修改表的表结构,创建一个临时表,在新表上执行ALTER TABLE语句
3、在源表上创建三个触发器分别对于INSERT UPDATE DELETE操作
4、从源表拷贝数据到临时表,在拷贝过程中,对源表的更新操作会写入到新建表中
5、将临时表和源表rename(需要元数据修改锁,需要短时间锁表)
6、删除源表和触发器,完成表结构的修改。

##=====================================================##
pt-osc工具限制
1、源表必须有主键或唯一索引,如果没有工具将停止工作
2、如果线上的复制环境过滤器操作过于复杂,工具将无法工作
3、如果开启复制延迟检查,但主从延迟时,工具将暂停数据拷贝工作
4、如果开启主服务器负载检查,但主服务器负载较高时,工具将暂停操作
5、当表使用外键时,如果未使用--alter-foreign-keys-method参数,工具将无法执行
6、只支持Innodb存储引擎表,且要求服务器上有该表1倍以上的空闲空间。

pt-osc之alter语句限制
1、不需要包含alter table关键字,可以包含多个修改操作,使用逗号分开,如"drop clolumn c1, add column c2 int"
2、不支持rename语句来对表进行重命名操作
3、不支持对索引进行重命名操作
4、如果删除外键,需要对外键名加下划线,如删除外键fk_uid, 修改语句为"DROP FOREIGN KEY _fk_uid"

pt-osc之命令模板
## --execute表示执行
## --dry-run表示只进行模拟测试
## 表名只能使用参数t来设置,没有长参数
pt-online-schema-change \
--host="127.0.0.1" \
--port=3358 \
--user="root" \
--password="root@root" \
--charset="utf8" \
--max-lag=10 \
--check-salve-lag='xxx.xxx.xxx.xxx' \
--recursion-method="hosts" \
--check-interval=2 \
--database="testdb1" \
 t="tb001" \
--alter="add column c4 int" \
--execute

例子:
pt-online-schema-change --user=root --password=123 --host=10.0.0.11 --alter "add column age int default 0" D=test,t=t1 --print --execute

2.3.3 pt-table-checksum

创建数据库
Create database pt CHARACTER SET utf8;
创建用户checksum并授权
GRANT ALL ON *.* TO 'checksum'@'10.0.0.%' IDENTIFIED BY 'checksum';
flush privileges;

--[no]check-replication-filters:是否检查复制的过滤器,默认是yes,建议启用不检查模式。
--databases | -d:指定需要被检查的数据库,多个库之间可以用逗号分隔。
--[no]check-binlog-format:是否检查binlog文件的格式,默认值yes。建议开启不检查。因为在默认的row格式下会出错。
--replicate`:把checksum的信息写入到指定表中。
--replicate-check-only:只显示不同步信息

pt-table-checksum --nocheck-replication-filters --no-check-binlog-format --replicate=pt.checksums --create-replicate-table --databases=test --tables=t1 h=10.0.0.11,u=checksum,p=checksum,P=3306

#!/bin/bash
date >> /root/db/checksum.log
pt-table-checksum --nocheck-binlog-format --nocheck-plan
--nocheck-replication-filters --replicate=pt.checksums --set-vars
innodb_lock_wait_timeout=120 --databases UAR_STATISTIC -u'checksum' -p'checksum'
-h'10.0.0.11' >> /root/db/checksum.log
date >> /root/db/checksum.log

用途: 可以用来检测主、 从数据库中数据的一致性。
原理: 在主库上运行, 对同步的表进行checksum, 记录下来。 然后对比主从中各个表的checksum是否一致, 从而判断数据是否一致。

测试
数据准备

-- 120
mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
|  4 | d    |
+----+------+
4 rows in set (0.00 sec)
-- 121 
mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
|  4 | d    |
+----+------+
4 rows in set (0.00 sec)

测试

[root@10-31-1-119 ~]# pt-table-checksum --nocheck-replication-filters --no-check-binlog-format --replicate=pt.checksums --create-replicate-table --databases=test --tables=zqs h=10.31.1.120,u=root,p=abc123,P=3306
Checking if all tables can be checksummed ...
Starting checksum ...
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 possibly with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-table-checksum line 332.
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 possibly with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-table-checksum line 332.

# A software update is available:
            TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
08-31T11:53:32      0      0        4          0       1       0   0.044 test.zqs
[root@10-31-1-119 ~]# 
-- 121
mysql> delete from zqs where id = 4;
Query OK, 1 row affected (0.01 sec)

mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
+----+------+
3 rows in set (0.00 sec)

重新检查

[root@10-31-1-119 ~]# pt-table-checksum --nocheck-replication-filters --no-check-binlog-format --replicate=pt.checksums --create-replicate-table --databases=test --tables=zqs h=10.31.1.120,u=root,p=abc123,P=3306
Checking if all tables can be checksummed ...
Starting checksum ...
            TS ERRORS  DIFFS     ROWS  DIFF_ROWS  CHUNKS SKIPPED    TIME TABLE
08-31T11:56:09      0      1        4          1       1       0   0.040 test.zqs

2.3.4 pt-table-sync

主要参数介绍
--replicate :指定通过pt-table-checksum得到的表.
--databases : 指定执行同步的数据库。
--tables :指定执行同步的表,多个用逗号隔开。
--sync-to-master :指定一个DSN,即从的IP,他会通过show processlist或show slave status 去自动的找主。
h= :服务器地址,命令里有2个ip,第一次出现的是Master的地址,第2次是Slave的地址。
u= :帐号。
p= :密码。
--print :打印,但不执行命令。
--execute :执行命令。
pt-table-sync --replicate=pt.checksums h=10.0.0.11,u=root,p=123,P=3306 --print

使用对两个库不一致的数据进行同步,他能够自动发现两个实例间不一致的数据,然后进行sync操作,pt-table-sync无法同步表结构,和索引等对象,只能同步数据。

数据准备

-- 120主库
mysql> create table zqs(id int primary key,name varchar(100));
Query OK, 0 rows affected (0.02 sec)

mysql> insert into zqs values (1,'a'),(2,'b'),(3,'c'),(4,'d');
Query OK, 4 rows affected (0.05 sec)
Records: 4  Duplicates: 0  Warnings: 0

mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
|  4 | d    |
+----+------+
4 rows in set (0.00 sec)

-- 121 从库
mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
|  4 | d    |
+----+------+
4 rows in set (0.00 sec)

mysql> delete from zqs where id = 4;
Query OK, 1 row affected (0.00 sec)

mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
+----+------+
3 rows in set (0.00 sec)

测试记录

[root@10-31-1-119 ~]# pt-table-sync --charset=utf8 --ignore-databases=mysql,sys --databases=test --tables=zqs --no-check-slave dsn=u=root,p=abc123,h=10.31.1.120 dsn=u=root,p=abc123,h=10.31.1.121 --execute --print
INSERT INTO `test`.`zqs`(`id`, `name`) VALUES ('4', 'd') /*percona-toolkit src_db:test src_tbl:zqs src_dsn:A=utf8,h=10.31.1.120,p=... dst_db:test dst_tbl:zqs dst_dsn:A=utf8,h=10.31.1.121,p=... lock:0 transaction:1 changing_src:0 replicate:0 bidirectional:0 pid:20478 user:root host:10-31-1-119*/;
[root@10-31-1-119 ~]# 

查看121从库上的数据

mysql> select * from zqs;
+----+------+
| id | name |
+----+------+
|  1 | a    |
|  2 | b    |
|  3 | c    |
|  4 | d    |
+----+------+
4 rows in set (0.00 sec)

mysql> 

也可以同步库和同步整个实例,简直不要太方便

-- 同步除mysql、sys两个库外的所有数据库
pt-table-sync --charset=utf8 --ignore-databases=mysql,sys --no-check-slave dsn=u=root,p=abc123,h=10.31.1.120 dsn=u=root,p=abc123,h=10.31.1.121 --execute --print
-- 同步整个test数据库
pt-table-sync --charset=utf8 --ignore-databases=mysql,sys --databases=test --no-check-slave dsn=u=root,p=abc123,h=10.31.1.120 dsn=u=root,p=abc123,h=10.31.1.121 --execute --print

2.3.5 pt-deadlock-logger(死锁检测)

参数

--create-dest-table :创建指定的表。
--dest              :创建存储死锁信息的表。
--database          :-D,指定链接的数据库。
--table             :-t,指定存储的表名。
--log               :指定死锁日志信息写入到文件。
--run-time          :运行次数,默认永久
--interval          :运行间隔时间,默认30s。
u,p,h,P           :链接数据库的信息。

用法
模拟一个死锁的场景

-- 关闭死锁检测
mysql> show variables like '%deadlock%';
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| innodb_deadlock_detect     | ON    |
| innodb_print_all_deadlocks | OFF   |
+----------------------------+-------+
2 rows in set (0.00 sec)

mysql> 
mysql> 
mysql> set global innodb_deadlock_detect = OFF;
Query OK, 0 rows affected (0.00 sec)

mysql> show variables like '%deadlock%';
+----------------------------+-------+
| Variable_name              | Value |
+----------------------------+-------+
| innodb_deadlock_detect     | OFF   |
| innodb_print_all_deadlocks | OFF   |
+----------------------------+-------+
2 rows in set (0.00 sec)
session A session B 描述
create table t1(id int,name varchar(100));
insert into t1 values (1,‘a’),(2,‘b’),(3,‘c’);
begin;
update t1 set name=‘ttt’ where id = 2;
begin;
update t1 set name = ‘ttt’ where id = 3;
update t1 set name = ‘www’ where id = 3; 锁住
update t1 set name=‘xxx’ where id = 2; 锁住
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction 未开启死锁检测,超时
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction 未开启死锁检测,超时
pt-deadlock-logger --ask-pass --run-time=10 --interval=3 --create-dest-table --dest D=test,t=deadlocks u=root,P=3306,h=10.31.1.120

输出信息

[root@10-31-1-120 ~]# pt-deadlock-logger --ask-pass --run-time=10 --interval=3 --create-dest-table --dest D=test,t=deadlocks u=root,P=3306,h=10.31.1.120
Enter MySQL password: 
Enter MySQL password: 
server ts thread txn_id txn_time user hostname ip db tbl idx lock_type lock_mode wait_hold victim query
10.31.1.120 2020-08-28T17:17:39 7448 0 32 root localhost  test t1 idx_id RECORD X w 1 update t1 set name='xxx' where id = 2
10.31.1.120 2020-08-28T17:17:39 7457 0 43 root localhost  test t1 idx_id RECORD X w 0 update t1 set name = 'www' where id = 3
[root@10-31-1-120 ~]# 
[root@10-31-1-120 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7486
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| deadlocks      |
| t1             |
+----------------+
2 rows in set (0.00 sec)

mysql> select * from deadlocks;
+-------------+---------------------+--------+--------+----------+------+-----------+----+------+-----+--------+-----------+-----------+-----------+--------+-----------------------------------------+
| server      | ts                  | thread | txn_id | txn_time | user | hostname  | ip | db   | tbl | idx    | lock_type | lock_mode | wait_hold | victim | query                                   |
+-------------+---------------------+--------+--------+----------+------+-----------+----+------+-----+--------+-----------+-----------+-----------+--------+-----------------------------------------+
| 10.31.1.120 | 2020-08-28 17:17:39 |   7448 |      0 |       32 | root | localhost |    | test | t1  | idx_id | RECORD    | X         | w         |      1 | update t1 set name='xxx' where id = 2   |
| 10.31.1.120 | 2020-08-28 17:17:39 |   7457 |      0 |       43 | root | localhost |    | test | t1  | idx_id | RECORD    | X         | w         |      0 | update t1 set name = 'www' where id = 3 |
+-------------+---------------------+--------+--------+----------+------+-----------+----+------+-----+--------+-----------+-----------+-----------+--------+-----------------------------------------+
2 rows in set (0.00 sec)

mysql> 

2.3.6 pt-duplicate-key-checker(主键冲突检测)

pt-duplicate-key-checker --database=world h='127.0.0.1' --user=root --password=123456

2.3.7 pt-kill(杀进程)

常用参数说明
--daemonize  放在后台以守护进程的形式运行;
--interval  多久运行一次,单位可以是s,m,h,d等默认是s –不加这个默认是5秒
--victims 默认是oldest,只杀最古老的查询。这是防止被查杀是不是真的长时间运行的查询,他们只是长期等待 这种种匹配按时间查询,杀死一个时间最高值。
--all 杀掉所有满足的线程
--kill-query      只杀掉连接执行的语句,但是线程不会被终止
--print               打印满足条件的语句
--busy-time 批次查询已运行的时间超过这个时间的线程;
--idle-time 杀掉sleep 空闲了多少时间的连接线程,必须在--match-command sleep时才有效—也就是匹配使用 -- –match-command 匹配相关的语句。
----ignore-command 忽略相关的匹配。 这两个搭配使用一定是ignore-commandd在前 match-command在后,
--match-db cdelzone 匹配哪个库
command有:Query、Sleep、Binlog Dump、Connect、Delayed insert、Execute、Fetch、Init DB、Kill、Prepare、Processlist、Quit、Reset stmt、Table Dump

例子:
---杀掉空闲链接sleep 5秒的 SQL 并把日志放到/home/pt-kill.log文件中
/usr/bin/pt-kill  --user=用户名 --password=密码 --match-command Sleep  --idle-time 5 --victim all --interval 5 --kill --daemonize -S /tmp/mysql.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

---查询SELECT 超过1分钟路

/usr/bin/pt-kill --user=用户名 --password=密码 --busy-time 60  --match-info "SELECT|select" --victim all --interval 5 --kill --daemonize -S  -S /tmp/mysql.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

--- Kill掉 select IFNULl.*语句开头的SQL

pt-kill --user=用户名 --password=密码 --victims all --busy-time=0 --match-info="select IFNULl.*" --interval 1 -S /tmp/mysqld.sock --kill --daemonize --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

----kill掉state Locked

/usr/bin/pt-kill --user=用户名 --password=密码  --victims all --match-state='Locked' --victim all --interval 5 --kill --daemonize -S /tmp/mysqld.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

---kill掉 a库,web为10.0.0.11的链接

pt-kill  --user=用户名 --password=密码 --victims all  --match-db='a' --match-host='10.0.0.11' --kill --daemonize --interval 10  -S /tmp/mysqld.sock  --pid=/tmp/ptkill.pid --print-log=/tmp/pt-kill.log &

---指定哪个用户kill

pt-kill   --user=用户名 --password=密码 --victims all --match-user='root' --kill  --daemonize --interval 10 -S /home/zb/data/my6006/socket/mysqld.sock --pid=/tmp/ptkill.pid --print --log=/home/pt-kill.log &

---查询SELECT 超过1分钟路

pt-kill  --user=用户名 --password=密码 --busy-time 60 --match-info "SELECT|select" --victim all  --interval 5 --kill --daemonize -S /tmp/mysqld.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &


----kill掉 command query | Execute

pt-kill --user=用户名 --password=密码 --victims all  --match-command= "query|Execute" --interval 5 --kill --daemonize -S /tmp/mysqld.sock --pid=/tmp/ptkill.pid --print --log=/home/pt-kill.log &

2.3.7.1 测试kill空闲链接

模拟两个空闲实例

-- session 1
[root@10-31-1-120 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14177
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

-- session 2
[root@10-31-1-120 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14180
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 
mysql> 
mysql> 
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| deadlocks      |
| t1             |
+----------------+
2 rows in set (0.00 sec)

mysql> begin;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from t1;
+------+------+
| id   | name |
+------+------+
|    1 | a    |
|    2 | b    |
|    3 | c    |
+------+------+
3 rows in set (0.00 sec)
-- 等待10秒,再执行这个语句
/usr/bin/pt-kill  --user=root --password=abc123 --match-command Sleep  --idle-time 5 --victim all --interval 5 --kill --daemonize -S /var/lib/mysql/mysql.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

测试记录

[root@10-31-1-119 ~]# /usr/bin/pt-kill  --user=root --password=abc123 --host=10.31.1.120 --match-command Sleep  --idle-time 5 --victim all --interval 5 --kill --daemonize -S /var/lib/mysql/mysql.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &
[2] 13170
[root@10-31-1-119 ~]# Overwriting PID file /tmp/ptkill.pid because the PID that it contains, 13000, is not running at /usr/bin/pt-kill line 2420.
Overwriting PID file /tmp/ptkill.pid because the PID that it contains, 13000, is not running at /usr/bin/pt-kill line 2420.

[2]+  完成                  /usr/bin/pt-kill --user=root --password=abc123 --host=10.31.1.120 --match-command Sleep --idle-time 5 --victim all --interval 5 --kill --daemonize -S /var/lib/mysql/mysql.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log
[root@10-31-1-119 ~]# 
[root@10-31-1-119 ~]# ps -ef | grep pt-kill
root     13171     1  0 10:35 ?        00:00:00 perl /usr/bin/pt-kill --user=root --password=abc123 --host=10.31.1.120 --match-command Sleep --idle-time 5 --victim all --interval 5 --kill --daemonize -S /var/lib/mysql/mysql.sock --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log
root     13193 31644  0 10:35 pts/1    00:00:00 grep --color=auto pt-kill
[root@10-31-1-119 ~]# 
[root@10-31-1-119 ~]# more /tmp/pt-kill.log
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 possibly with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-kill line 5761.
# 2020-08-31T10:35:03 KILL 14177 (Sleep 233 sec) NULL
# 2020-08-31T10:35:03 KILL 14180 (Sleep 167 sec) NULL
[root@10-31-1-119 ~]# 

查看2个session是否被kill

-- session 1
[root@10-31-1-120 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14177
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 
mysql> select sysdate;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    14188
Current database: *** NONE ***

ERROR 1054 (42S22): Unknown column 'sysdate' in 'field list'
mysql> 
mysql> 
-- session 2
[root@10-31-1-120 ~]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14180
Server version: 5.7.31-log MySQL Community Server (GPL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 
mysql> 
mysql> 
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| deadlocks      |
| t1             |
+----------------+
2 rows in set (0.00 sec)

mysql> begin;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from t1;
+------+------+
| id   | name |
+------+------+
|    1 | a    |
|    2 | b    |
|    3 | c    |
+------+------+
3 rows in set (0.00 sec)

mysql> select current_timestamp();
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    14189
Current database: test

+---------------------+
| current_timestamp() |
+---------------------+
| 2020-08-31 10:35:54 |
+---------------------+
1 row in set (0.00 sec)

可以看到,未完成的事务因为空闲5秒,也被kill了,生产环境需谨慎

2.3.7.2 kill查询时间超过20秒的进程

数据准备

mysql> select count(*) from t1;
+----------+
| count(*) |
+----------+
|  5889738 |
+----------+
1 row in set (30.54 sec)

kill 语句准备

/usr/bin/pt-kill --user=lepus --password=lepus --host=10.31.1.112 --busy-time 20  --match-info "SELECT|select" --victim all --interval 5 --kill --daemonize -S  --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

测试记录

[root@10-31-1-119 ~]# /usr/bin/pt-kill --user=lepus --password=lepus --host=10.31.1.112 --busy-time 20  --match-info "SELECT|select" --victim all --interval 5 --kill --daemonize -S  --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &
[2] 15172
[root@10-31-1-119 ~]# 
[2]+  完成                  /usr/bin/pt-kill --user=lepus --password=lepus --host=10.31.1.112 --busy-time 20 --match-info "SELECT|select" --victim all --interval 5 --kill --daemonize -S --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log
[root@10-31-1-119 ~]# 
[root@10-31-1-119 ~]# ps -ef | grep pt-kill
root     15173     1  0 10:51 ?        00:00:00 perl /usr/bin/pt-kill --user=lepus --password=lepus --host=10.31.1.112 --busy-time 20 --match-info SELECT|select --victim all --interval 5 --kill --daemonize -S --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log
root     15208 31644  0 10:52 pts/1    00:00:00 grep --color=auto pt-kill
[root@10-31-1-119 ~]# 
mysql> select count(*) from t1 order by name;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 

2.3.7.3 Kill掉 select IFNULl.*语句开头的SQL

kill语句

pt-kill --user=lepus --password=lepus --host=10.31.1.112 --victims all --busy-time=0 --match-info="select IFNULl.*" --interval 1  --kill --daemonize --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &

测试记录

[root@10-31-1-119 ~]# pt-kill --user=lepus --password=lepus --host=10.31.1.112 --victims all --busy-time=0 --match-info="select IFNULl.*" --interval 1  --kill --daemonize --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log &
[2] 15609
[root@10-31-1-119 ~]# Overwriting PID file /tmp/ptkill.pid because the PID that it contains, 13171, is not running at /usr/bin/pt-kill line 2420.
Overwriting PID file /tmp/ptkill.pid because the PID that it contains, 13171, is not running at /usr/bin/pt-kill line 2420.

[2]+  完成                  pt-kill --user=lepus --password=lepus --host=10.31.1.112 --victims all --busy-time=0 --match-info="select IFNULl.*" --interval 1 --kill --daemonize --pid=/tmp/ptkill.pid --print --log=/tmp/pt-kill.log
mysql> select IFNULl(id,'NULL'),name from t1;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 

查看日志

[root@10-31-1-119 ~]# more /tmp/pt-kill.log 
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 possibly with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-kill line 5761.
# 2020-08-31T10:35:03 KILL 14177 (Sleep 233 sec) NULL
# 2020-08-31T10:35:03 KILL 14180 (Sleep 167 sec) NULL
# 2020-08-31T10:35:48 KILL 14188 (Sleep 9 sec) NULL
# 2020-08-31T10:36:03 KILL 14189 (Sleep 9 sec) NULL
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 possibly with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-kill line 5761.
*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 possibly with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/bin/pt-kill line 5761.

# A software update is available:
# 2020-08-31T10:53:06 KILL 10663 (Query 22 sec) select count(*) from t1 order by name
# 2020-08-31T10:56:29 KILL 10680 (Query 0 sec) select IFNULl(id,'NULL'),name from t1
[root@10-31-1-119 ~]# 

参考:

1.https://www.cnblogs.com/zishengY/p/6852280.html
2.https://www.jianshu.com/p/36ace5c2bc8b
3.https://www.cnblogs.com/zhoujinyi/p/3392800.html

你可能感兴趣的:(MySQL运维)