一次数据库死锁的问题排查

由于外键导致的死锁问题排查

使用jmeter 进行前端稳定性循环压测的时候分别进行模型的上线、下线、修改、共享、取消共享操作时出现了大量的死锁的异常。

2019-06-18 15:00:47,086 [qtp2128665553-168] o.h.e.j.s.SqlExceptionHelper WARN SQL Error: 1213, SQLState: 40001
2019-06-18 15:00:47,087 [qtp2128665553-168] o.h.e.j.s.SqlExceptionHelper ERROR Deadlock found when trying to get lock; try restarting transaction

问题排查

由于我们的数据库默认是innodb，所以先查询下发生死锁的sql show engine innodb status

显示如下：

LATEST DETECTED DEADLOCK

2019-06-18 21:52:08 7ff17cdf9700
*** (1) TRANSACTION:
TRANSACTION 3620735312, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 6 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 2
MySQL thread id 12, OS thread handle 0x7ff17cdb8700, query id 1908119 192.168.129.214 root updating
update he_model set ai_model=’[“你好”,“再见”,“修改”]’, color=’#000000’, conetext_num=5, create_time=‘2019-06-18 21:52:08’, creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=813, last_modify_date=‘2019-06-18 21:52:08.19’, model_group_id=72, model_type=‘ai’, name=‘AI模型’, process_percent=‘1’, rte_content=‘

AI模型修改

’, status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where id=809
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2208 page no 293 n bits 112 index PRIMARY of table haw21063. he_model trx id 3620735312 lock_mode X locks rec but not gap waiting
Record lock, heap no 38 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000329; asc );;
1: len 6; hex 0000d7d00927; asc ';;
2: len 7; hex 0b00001ead0998; asc ;;
3: len 28; hex 5b22e4bda0e5a5bd222c22e5868de8a781222c22e8b0a2e8b0a2225d; asc [" “,” “,” "];;
4: len 7; hex 23303030303030; asc #000000;;
5: len 4; hex 80000005; asc ;;
6: len 5; hex 99a3655d08; asc e] ;;
7: len 1; hex 00; asc ;;
8: SQL NULL;
9: SQL NULL;
10: len 1; hex 00; asc ;;
11: len 4; hex 8000000a; asc ;;
12: len 5; hex 99a3655d08; asc e] ;;
13: len 2; hex 6169; asc ai;;
14: len 8; hex 4149e6a8a1e59e8b; asc AI ;;
15: len 1; hex 31; asc 1;;
16: len 15; hex 3c703e4149e6a8a1e59e8b3c2f703e; asc

AI

;;
17: len 4; hex 80000000; asc ;;
18: SQL NULL;
19: SQL NULL;
20: SQL NULL;
21: SQL NULL;
22: len 1; hex 00; asc ;;
23: len 4; hex 8000032d; asc -;;
24: SQL NULL;
25: len 4; hex 8000032d; asc -;;
26: len 4; hex 80000048; asc H;;

*** (2) TRANSACTION:
TRANSACTION 3620735311, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 2
MySQL thread id 11, OS thread handle 0x7ff17cdf9700, query id 1908121 192.168.129.214 root updating
update he_model set ai_model=’[“你好”,“再见”,“修改”]’, color=’#000000’, conetext_num=5, create_time=‘2019-06-18 21:52:08’, creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=812, last_modify_date=‘2019-06-18 21:52:08.19’, model_group_id=72, model_type=‘ai’, name=‘AI模型’, process_percent=‘1’, rte_content=‘

AI模型修改

’, status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where id=809
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 2208 page no 293 n bits 112 index PRIMARY of table haw21063. he_model trx id 3620735311 lock mode S locks rec but not gap
Record lock, heap no 38 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000329; asc );;
1: len 6; hex 0000d7d00927; asc ';;
2: len 7; hex 0b00001ead0998; asc ;;
3: len 28; hex 5b22e4bda0e5a5bd222c22e5868de8a781222c22e8b0a2e8b0a2225d; asc [" “,” “,” "];;
4: len 7; hex 23303030303030; asc #000000;;
5: len 4; hex 80000005; asc ;;
6: len 5; hex 99a3655d08; asc e] ;;
7: len 1; hex 00; asc ;;
8: SQL NULL;
9: SQL NULL;
10: len 1; hex 00; asc ;;
11: len 4; hex 8000000a; asc ;;
12: len 5; hex 99a3655d08; asc e] ;;
13: len 2; hex 6169; asc ai;;
14: len 8; hex 4149e6a8a1e59e8b; asc AI ;;
15: len 1; hex 31; asc 1;;
16: len 15; hex 3c703e4149e6a8a1e59e8b3c2f703e; as
17: len 4; hex 80000000; asc ;;
18: SQL NULL;
19: SQL NULL;
20: SQL NULL;
21: SQL NULL;
22: len 1; hex 00; asc ;;
23: len 4; hex 8000032d; asc -;;
24: SQL NULL;
25: len 4; hex 8000032d; asc -;;
26: len 4; hex 80000048; asc H;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2208 page no 293 n bits 112 index PRIMARY of table haw21063.he_model trx id 3620735311 lock_mode X locks rec but not gap waiting
Record lock, heap no 38 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000329; asc );;
1: len 6; hex 0000d7d00927; asc ';;
2: len 7; hex 0b00001ead0998; asc ;;
3: len 28; hex 5b22e4bda0e5a5bd222c22e5868de8a781222c22e8b0a2e8b0a2225d; asc [" “,” “,” "];;
4: len 7; hex 23303030303030; asc #000000;;
5: len 4; hex 80000005; asc ;;
6: len 5; hex 99a3655d08; asc e] ;;
7: len 1; hex 00; asc ;;
8: SQL NULL;
9: SQL NULL;
10: len 1; hex 00; asc ;;
11: len 4; hex 8000000a; asc ;;
12: len 5; hex 99a3655d08; asc e] ;;
13: len 2; hex 6169; asc ai;;
14: len 8; hex 4149e6a8a1e59e8b; asc AI ;;
15: len 1; hex 31; asc 1;;
16: len 15; hex 3c703e4149e6a8a1e59e8b3c2f703e; asc

AI

;;
17: len 4; hex 80000000; asc ;;
18: SQL NULL;
19: SQL NULL;
20: SQL NULL;
21: SQL NULL;
22: len 1; hex 00; asc ;;
23: len 4; hex 8000032d; asc -;;
24: SQL NULL;
25: len 4; hex 8000032d; asc -;;
26: len 4; hex 80000048; asc H;;

在修改模型id为809的模型发生了死锁，这个不可能的呀，如果同时修改一个记录，会设置X锁进行排他，不能发生死锁的问题呀。
之后百度了一下，遇到这种现象

初步分析

查看Innodb的操作二进制文件，首先查看二进制文件是否开启

mysql> show variables like ‘log_bin’;
±--------------±------+
| Variable_name | Value |
±--------------±------+
| log_bin | OFF |
±--------------±------+
二进制文件默认不开启，需要去数据库my.cnf增加二进制文件开关，find / -name “my.cnf” 找到对应的cnf文件

2. 二进制文件开启后，重启mysql，使用jmeter进行稳定性测试，发生死锁日志。

3. 注意一下，默认二进制文件直接使用文本编辑器打不开，所以需要将二进制文件转换为txt文件来查看

mysqlbinlog -u root -p mysql-bin.000001 > 1.txt
转换的txt文件显示需要结合数据库查询show binlog events in ‘mysql-bin.000001’;来确定pos的id对应的sql，根据修改到的内容和时间点定位到

顺序	事务1	事务2	说明
1	begin
2		begin
3	insert into model_update_history (ai_model, create_time, dim_formula, dim_tree, word_interval, model, model_name, model_type, modifier, modify_time, strategy_formular, strategy_tree, text_formula, text_tree) values ('[""你好"",""再见"",""谢谢""]', '2019-06-18 21:52:08.191', null, null, 10, 809, 'AI模型', 'ai', 'modeler', '2019-06-18 21:52:08', null, null, null, null)"
4		insert into model_update_history (ai_model, create_time, dim_formula, dim_tree, word_interval, model, model_name, model_type, modifier, modify_time, strategy_formular, strategy_tree, text_formula, text_tree) values ('[""你好"",""再见"",""谢谢""]', '2019-06-18 21:52:08.191', null, null, 10, 809, 'AI模型', 'ai', 'modeler', '2019-06-18 21:52:08', null, null, null, null)"	事务2 在he_model_history获取到history的锁，然后修改了历史记录，然后向A表申请S锁来查询数据
6	update he_model set ai_model='["你好","再见","修改"]', color='#000000', conetext_num=5, create_time='2019-06-18 21:52:08', creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=813, last_modify_date='2019-06-18 21:52:08.19', model_group_id=72, model_type='ai', name='AI模型', process_percent='1', rte_content=' AI模型修改 ', status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where id=809		事务1 在he_model表上申请到X锁以修改he_model表记录，并且要检查model_update_history是否满足外键约束，需要S锁
		update he_model set ai_model='["你好","再见","修改"]', color='#000000', conetext_num=5, create_time='2019-06-18 21:52:08', creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=813, last_modify_date='2019-06-18 21:52:08.19', model_group_id=72, model_type='ai', name='AI模型', process_percent='1', rte_content=' AI模型修改 ', status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where	事物2打算给sku表id为809记录上 X 排它锁，发现被其他事务上了，而且此事务居然还在等他提交，这时MYSQL立刻回滚事务2…(发现MYSQL返回死锁信息，记录该信息到错误日志…发送回滚指令…mysql已经“帮”他回滚了
7			事务1发现别人的锁释放了，获得X锁，执行成功
	执行成功
8			事务1执行成功，记录binlog日志
	commit

解决办法：
对表B上的外键列建立索引，从而使事务A发生时将range锁放到索引上，从而降低死锁发生的概率
​