一次数据库死锁的问题排查

一次死锁问题的排查

  • 由于外键导致的死锁问题排查
    • 问题排查
    • 显示如下:
    • LATEST DETECTED DEADLOCK
    • 初步分析

由于外键导致的死锁问题排查

使用jmeter 进行前端稳定性循环压测的时候分别进行模型的上线、下线、修改、共享、取消共享操作时出现了大量的死锁的异常。

2019-06-18 15:00:47,086 [qtp2128665553-168] o.h.e.j.s.SqlExceptionHelper WARN SQL Error: 1213, SQLState: 40001
2019-06-18 15:00:47,087 [qtp2128665553-168] o.h.e.j.s.SqlExceptionHelper ERROR Deadlock found when trying to get lock; try restarting transaction

问题排查

由于我们的数据库默认是innodb,所以先查询下发生死锁的sql show engine innodb status

显示如下:

LATEST DETECTED DEADLOCK

2019-06-18 21:52:08 7ff17cdf9700
*** (1) TRANSACTION:
TRANSACTION 3620735312, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 6 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 2
MySQL thread id 12, OS thread handle 0x7ff17cdb8700, query id 1908119 192.168.129.214 root updating
update he_model set ai_model=’[“你好”,“再见”,“修改”]’, color=’#000000’, conetext_num=5, create_time=‘2019-06-18 21:52:08’, creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=813, last_modify_date=‘2019-06-18 21:52:08.19’, model_group_id=72, model_type=‘ai’, name=‘AI模型’, process_percent=‘1’, rte_content=‘

AI模型修改

’, status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where id=809
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2208 page no 293 n bits 112 index PRIMARY of table haw21063. he_model trx id 3620735312 lock_mode X locks rec but not gap waiting
Record lock, heap no 38 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000329; asc );;
1: len 6; hex 0000d7d00927; asc ';;
2: len 7; hex 0b00001ead0998; asc ;;
3: len 28; hex 5b22e4bda0e5a5bd222c22e5868de8a781222c22e8b0a2e8b0a2225d; asc [" “,” “,” "];;
4: len 7; hex 23303030303030; asc #000000;;
5: len 4; hex 80000005; asc ;;
6: len 5; hex 99a3655d08; asc e] ;;
7: len 1; hex 00; asc ;;
8: SQL NULL;
9: SQL NULL;
10: len 1; hex 00; asc ;;
11: len 4; hex 8000000a; asc ;;
12: len 5; hex 99a3655d08; asc e] ;;
13: len 2; hex 6169; asc ai;;
14: len 8; hex 4149e6a8a1e59e8b; asc AI ;;
15: len 1; hex 31; asc 1;;
16: len 15; hex 3c703e4149e6a8a1e59e8b3c2f703e; asc

AI

;;
17: len 4; hex 80000000; asc ;;
18: SQL NULL;
19: SQL NULL;
20: SQL NULL;
21: SQL NULL;
22: len 1; hex 00; asc ;;
23: len 4; hex 8000032d; asc -;;
24: SQL NULL;
25: len 4; hex 8000032d; asc -;;
26: len 4; hex 80000048; asc H;;

*** (2) TRANSACTION:
TRANSACTION 3620735311, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 2
MySQL thread id 11, OS thread handle 0x7ff17cdf9700, query id 1908121 192.168.129.214 root updating
update he_model set ai_model=’[“你好”,“再见”,“修改”]’, color=’#000000’, conetext_num=5, create_time=‘2019-06-18 21:52:08’, creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=812, last_modify_date=‘2019-06-18 21:52:08.19’, model_group_id=72, model_type=‘ai’, name=‘AI模型’, process_percent=‘1’, rte_content=‘

AI模型修改

’, status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where id=809
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 2208 page no 293 n bits 112 index PRIMARY of table haw21063. he_model trx id 3620735311 lock mode S locks rec but not gap
Record lock, heap no 38 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000329; asc );;
1: len 6; hex 0000d7d00927; asc ';;
2: len 7; hex 0b00001ead0998; asc ;;
3: len 28; hex 5b22e4bda0e5a5bd222c22e5868de8a781222c22e8b0a2e8b0a2225d; asc [" “,” “,” "];;
4: len 7; hex 23303030303030; asc #000000;;
5: len 4; hex 80000005; asc ;;
6: len 5; hex 99a3655d08; asc e] ;;
7: len 1; hex 00; asc ;;
8: SQL NULL;
9: SQL NULL;
10: len 1; hex 00; asc ;;
11: len 4; hex 8000000a; asc ;;
12: len 5; hex 99a3655d08; asc e] ;;
13: len 2; hex 6169; asc ai;;
14: len 8; hex 4149e6a8a1e59e8b; asc AI ;;
15: len 1; hex 31; asc 1;;
16: len 15; hex 3c703e4149e6a8a1e59e8b3c2f703e; as
17: len 4; hex 80000000; asc ;;
18: SQL NULL;
19: SQL NULL;
20: SQL NULL;
21: SQL NULL;
22: len 1; hex 00; asc ;;
23: len 4; hex 8000032d; asc -;;
24: SQL NULL;
25: len 4; hex 8000032d; asc -;;
26: len 4; hex 80000048; asc H;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2208 page no 293 n bits 112 index PRIMARY of table haw21063.he_model trx id 3620735311 lock_mode X locks rec but not gap waiting
Record lock, heap no 38 PHYSICAL RECORD: n_fields 27; compact format; info bits 0
0: len 4; hex 80000329; asc );;
1: len 6; hex 0000d7d00927; asc ';;
2: len 7; hex 0b00001ead0998; asc ;;
3: len 28; hex 5b22e4bda0e5a5bd222c22e5868de8a781222c22e8b0a2e8b0a2225d; asc [" “,” “,” "];;
4: len 7; hex 23303030303030; asc #000000;;
5: len 4; hex 80000005; asc ;;
6: len 5; hex 99a3655d08; asc e] ;;
7: len 1; hex 00; asc ;;
8: SQL NULL;
9: SQL NULL;
10: len 1; hex 00; asc ;;
11: len 4; hex 8000000a; asc ;;
12: len 5; hex 99a3655d08; asc e] ;;
13: len 2; hex 6169; asc ai;;
14: len 8; hex 4149e6a8a1e59e8b; asc AI ;;
15: len 1; hex 31; asc 1;;
16: len 15; hex 3c703e4149e6a8a1e59e8b3c2f703e; asc

AI

;;
17: len 4; hex 80000000; asc ;;
18: SQL NULL;
19: SQL NULL;
20: SQL NULL;
21: SQL NULL;
22: len 1; hex 00; asc ;;
23: len 4; hex 8000032d; asc -;;
24: SQL NULL;
25: len 4; hex 8000032d; asc -;;
26: len 4; hex 80000048; asc H;;

在修改模型id为809的模型发生了死锁,这个不可能的呀,如果同时修改一个记录,会设置X锁进行排他,不能发生死锁的问题呀。
之后百度了一下,遇到这种现象

初步分析

查看Innodb的操作二进制文件,首先查看二进制文件是否开启

mysql> show variables like ‘log_bin’;
±--------------±------+
| Variable_name | Value |
±--------------±------+
| log_bin | OFF |
±--------------±------+
二进制文件默认不开启,需要去数据库my.cnf增加二进制文件开关,find / -name “my.cnf” 找到对应的cnf文件

一次数据库死锁的问题排查_第1张图片
2. 二进制文件开启后,重启mysql,使用jmeter进行稳定性测试,发生死锁日志。

一次数据库死锁的问题排查_第2张图片
3. 注意一下,默认二进制文件直接使用文本编辑器打不开,所以需要将二进制文件转换为txt文件来查看

mysqlbinlog -u root -p mysql-bin.000001 > 1.txt
转换的txt文件显示需要结合数据库查询show binlog events in ‘mysql-bin.000001’;来确定pos的id对应的sql,根据修改到的内容和时间点定位到在修改模型的时候会对模型修改历史进行修改

顺序 事务1 事务2 说明
1 begin
2 begin
3 insert into model_update_history (ai_model, create_time, dim_formula, dim_tree, word_interval, model, model_name, model_type, modifier, modify_time, strategy_formular, strategy_tree, text_formula, text_tree) values ('[""你好"",""再见"",""谢谢""]', '2019-06-18 21:52:08.191', null, null, 10, 809, 'AI模型', 'ai', 'modeler', '2019-06-18 21:52:08', null, null, null, null)"
4 insert into model_update_history (ai_model, create_time, dim_formula, dim_tree, word_interval, model, model_name, model_type, modifier, modify_time, strategy_formular, strategy_tree, text_formula, text_tree) values ('[""你好"",""再见"",""谢谢""]', '2019-06-18 21:52:08.191', null, null, 10, 809, 'AI模型', 'ai', 'modeler', '2019-06-18 21:52:08', null, null, null, null)" 事务2 在he_model_history获取到history的锁,然后修改了历史记录,然后向A表申请S锁来查询数据
6 update he_model set ai_model='["你好","再见","修改"]', color='#000000', conetext_num=5, create_time='2019-06-18 21:52:08', creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=813, last_modify_date='2019-06-18 21:52:08.19', model_group_id=72, model_type='ai', name='AI模型', process_percent='1', rte_content='

AI模型修改

', status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where id=809
事务1 在he_model表上申请到X锁以修改he_model表记录,并且要检查model_update_history是否满足外键约束,需要S锁
update he_model set ai_model='["你好","再见","修改"]', color='#000000', conetext_num=5, create_time='2019-06-18 21:52:08', creator=813, deleted=0, dim_formula=null, dim_tree=null, warning_id=null, fly_screen=0, word_interval=10, last_modifier=813, last_modify_date='2019-06-18 21:52:08.19', model_group_id=72, model_type='ai', name='AI模型', process_percent='1', rte_content='

AI模型修改

', status=1, strategy_formular=null, strategy_tree=null, text_formula=null, text_tree=null, warning=0 where
事物2打算给sku表id为809记录上 X 排它锁,发现被其他事务上了,而且此事务居然还在等他提交,这时MYSQL立刻回滚事务2…(发现MYSQL返回死锁信息,记录该信息到错误日志…发送回滚指令…mysql已经“帮”他回滚了
7 事务1发现别人的锁释放了,获得X锁,执行成功
执行成功
8 事务1执行成功,记录binlog日志
commit

解决办法:
对表B上的外键列建立索引,从而使事务A发生时将range锁放到索引上,从而降低死锁发生的概率

你可能感兴趣的:(故障排除,Pachira)