MySQL 大数据操作注意事项
MySQL 大数据操作注意事项
http://netkiller.github.io/journal/mysql.parallel.html
Mr. Neo Chen (netkiller), 陈景峰(BG7NYT)
目录
1. 关于 delete
2. 关于 update
3. 关于创建索引
4. 关于 OPTIMIZE
5. 关于切换引擎
6. 确保SELECT不被受阻
1. 关于 delete
delete from mytable 必死无疑，你需要分批删除，尽量缩小每个批次删除的记录数，delete 是可以并行执行的，你可以同时运行多个删除操作
mysql> show processlist; +--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 52 | Waiting for next activation | NULL | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | | 117446 | dba | localhost | example | Query | 20 | updating | delete from mytable where OPEN_TIME like '2011.11.28%' | | 117525 | dba | localhost | example | Query | 2 | updating | delete from mytable where OPEN_TIME like '2011.12.02%' | | 117526 | dba | localhost | example | Query | 49 | updating | delete from mytable where OPEN_TIME like '2011.12.12%' | | 117527 | dba | localhost | example | Query | 6 | updating | delete from mytable where OPEN_TIME like '2011.12.21%' | | 117528 | dba | localhost | example | Query | 64 | updating | delete from mytable where OPEN_TIME like '2011.12.30%' | | 117546 | dba | localhost | example | Query | 33 | updating | delete from mytable where OPEN_TIME like '2011.11.10%' | +--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+ 23 rows in set (0.00 sec)
2. 关于 update
在电商领域常常遇到一个问题“调价”，经常需要调整一批商品的价格, 程序猿一条语句搞定有没有？
update goods set price=price+10 where category_id = xxx
在开发，测试环境是可以通过测试的，一旦部署到生产环境，必死无疑
3. 关于创建索引
大表创建索引需要很久的时间，通常要经历 manage keys 与 copy to tmp table 的过程
mysql> show processlist; +--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 47 | Waiting for next activation | NULL | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | | 118814 | dba | 192.168.6.20:50459 | example | Query | 8 | copy to tmp table | ALTER TABLE `mytable` ADD INDEX `modifiy_time` (`MODIFY_TIME`) | +--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+ 17 rows in set (0.00 sec)
删除索引，也需要经理 copy to tmp table 过程，漫长的等待
mysql> show processlist; +--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 11 | Waiting for next activation | NULL | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | | 118814 | dba | 192.168.6.20:50459 | example | Query | 4 | copy to tmp table | ALTER TABLE `mytable` DROP INDEX `modifiy_time` | +--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+ 17 rows in set (0.00 sec)
所以数据设计要深思熟虑，做到提前未雨绸缪，不要亡羊补牢
4. 关于 OPTIMIZE
OPTIMIZE 的操作是将当前表复制到临时表操作后再删除当前表，最后将临时表改名
mysql> show processlist; +--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 14 | Waiting for next activation | NULL | | 115835 | dba | 192.168.6.20:49664 | example | Query | 9 | copy to tmp table | OPTIMIZE TABLE `mytable` | | 115986 | dba | localhost | example | Query | 0 | NULL | show processlist | +--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+ 17 rows in set (0.00 sec)
5. 关于切换引擎
转换ENGINE从MyISAM到InnoDB会经历creating table然后copy to tmp table在修改表名几个阶段，过程非常缓慢
mysql> show processlist; +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 10 | Waiting for next activation | NULL | | 3167 | dba | 192.168.6.20:56723 | example | Query | 2 | creating table | ALTER TABLE `mytable` ENGINE=InnoDB | | 3172 | dba | localhost | example | Query | 0 | NULL | show processlist | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ 18 rows in set (0.00 sec)
copy to tmp table 过程
mysql> show processlist; +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ | 1 | event_scheduler | localhost | NULL | Daemon | 21 | Waiting for next activation | NULL | | 3167 | dba | 192.168.6.20:56723 | example | Query | 13 | copy to tmp table | ALTER TABLE `mytable` ENGINE=InnoDB | | 3172 | dba | localhost | example | Query | 0 | NULL | show processlist | +------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+ 18 rows in set (0.00 sec)
此时我们查看mysql data目录会看到临时表文件
# ll /var/lib/mysql/hx9999_real_history/ -rw-rw---- 1 mysql mysql 9522 May 16 17:17 #sql-c2f_c5f.frm -rw-rw---- 1 mysql mysql 48 May 16 17:17 #sql-c2f_c5f.par -rw-rw---- 1 mysql mysql 637534208 May 16 17:29 #sql-c2f_c5f#P#p0.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p1.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p2.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p3.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p4.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p5.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p6.ibd -rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p7.ibd
6. 确保SELECT不被受阻
使用各种手段保证select操作不被受阻，只要select一直可以查询网站前端就能提供80%的功能，一旦select受阻一切都是浮云。
保证 select 操作优先于其他操作
UPDATE [LOW_PRIORITY] [IGNORE] tbl_name SET col_name1=expr1 [, col_name2=expr2 ...] [WHERE where_definition] [ORDER BY ...] [LIMIT row_count]
update的时候增加 LOW_PRIORITY 参数，可以降低更新语句的优先级。
my.cnf
[mysqld] low_priority_updates=1
或者启动是添加--low-priority-updates参数
全局开启
SET @@global.low_priority_updates = 1;
适用于本次会话连接
SET @@session.low_priority_updates = 1;