Mysql因为索引造成死锁DEADLOCK
近日生产发生死锁，日志和表索引如下。
分析是
因为索引的原因，在循环对数据进行Update的时候，因为查询的结果集颗粒度很大，造成多个线程因为索引INX_ACCOUNT_ID产生了行级锁。
解决办法是，缩小查询结果集的颗粒度，就是缩小查询的结果范围，在循环Update的时候不会造成资源竞争。
循环如下
//新的循环
List> accountMap=jsperManagerService.selectJsperDataUnProcessMoreKey();
logger.info("STORM SPOUT UNACCOUNTID LIST SIZE IS " + accountMap.size());
// TODO 这里Spout业务
if(CollectionUtils.isNotEmpty(accountMap)){
logger.info("STORM SPOUT BIGLOOP START " + System.currentTimeMillis());
for(Map mapAccountId:accountMap){
if(mapAccountId.get("ACCOUNT_ID")!=null){
String accountId = "";
String number = "";
accountId = String.valueOf(mapAccountId.get("ACCOUNT_ID"));
number = String.valueOf(mapAccountId.get("NUMBER"));
List> list = jsperManagerService.selectJsperInterDataMoreKey(accountId,number);//获取接口信息
logger.info("STORM SPOUT LITTLELOOP START " + System.currentTimeMillis());
for(Map map:list){
if(map != null){
if(updateJsperStatus(map,true)){
outputCollector.emit(new Values(map));
}
}
}
logger.info("STORM SPOUT LITTLELOOP END " + System.currentTimeMillis());
}
}
logger.info("STORM SPOUT BIGLOOP END " + System.currentTimeMillis());
}
//原有循环
//List> unaccountId=jsperManagerService.selectJsperDataUnProcess();
// logger.info("STORM SPOUT UNACCOUNTID LIST SIZE IS " + unaccountId.size());
// // TODO 这里Spout业务
// if(CollectionUtils.isNotEmpty(unaccountId)){
// logger.info("STORM SPOUT BIGLOOP START " + System.currentTimeMillis());
// for(Map mapAccountId:unaccountId){
// if(mapAccountId.get("ACCOUNT_ID")!=null){
// List> list = jsperManagerService.selectJsperInterData(mapAccountId.get("ACCOUNT_ID").toString());//获取接口信息
// logger.info("STORM SPOUT LITTLELOOP START " + System.currentTimeMillis());
// for(Map map:list){
// if(map != null){
// //JpInterCallerDto dto = (JpInterCallerDto) ConversionUtil.map2po(map, JpInterCallerDto.class);
// if(updateJsperStatus(map,true)){
// outputCollector.emit(new Values(map));
// }
// }
// }
// logger.info("STORM SPOUT LITTLELOOP END " + System.currentTimeMillis());
// }
// }
// logger.info("STORM SPOUT BIGLOOP END " + System.currentTimeMillis());
// }
索引
MariaDB [jliot]> show index in JP_INTER_CALLER;
+-----------------+------------+----------------------+--------------+-------------+-----------+-------------+----------+--------+------+----------- -+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------+------------+----------------------+--------------+-------------+-----------+-------------+----------+--------+------+----------- -+---------+---------------+
| jp_inter_caller | 1 | JP_CALLER_INX | 1 | CREATE_TIME | A | 234671 | NULL | NULL | | BTREE | | |
| jp_inter_caller | 1 | JP_CALLER_CUSTID_INX | 1 | CUST_NUMBER | A | 1173359 | NULL | NULL | YES | BTREE | | |
| jp_inter_caller | 1 | INX_ACCOUNT_ID | 1 | USE_TAG | A | 14 | NULL | NULL | YES | BTREE | | |
| jp_inter_caller | 1 | INX_ACCOUNT_ID | 2 | ACCOUNT_ID | A | 96 | NULL | NULL | YES | BTREE | | |
| jp_inter_caller | 1 | JP_CALLER_PARAM3_INX | 1 | PARAM3 | A | 28 | NULL | NULL | YES | BTREE | | |
+-----------------+------------+----------------------+--------------+-------------+-----------+-------------+----------+--------+------+----------- -+---------+---------------+
5 rows in set (0.00 sec)
错误日志
MariaDB [(none)]> show engine innodb status \G;
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:
=====================================
2019-05-24 21:45:41 0x7f9a602ac700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 62 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 801730 srv_active, 0 srv_shutdown, 5432915 srv_idle
srv_master_thread log flush and writes: 6234588
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 4063295
OS WAIT ARRAY INFO: signal count 5446146
RW-shared spins 0, rounds 5526022, OS waits 1696887
RW-excl spins 0, rounds 145027489, OS waits 384698
RW-sx spins 34180, rounds 511779, OS waits 4569
Spin rounds per wait: 5526022.00 RW-shared, 145027489.00 RW-excl, 14.97 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2019-05-24 21:45:17 0x7f98e2d7f700
*** (1) TRANSACTION:
TRANSACTION 202714835, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 1217887, OS thread handle 140294629988096, query id 273875416 wl wjasper1 10.10.2.171 iotuser Update
insert into JP_INTER_CALLER
( NUMBER,
ID,
ICCID,
PARAM4,
USE_TAG,
UPDATE_TIME,
CREATE_TIME,
CUST_NUMBER,
EXEC_TIME,
ACCOUNT_ID )
values ( 1051905240288090,
1,
'89860617060069974202',
'1',
'0',
'2019-05-24 21:44:03',
'2019-05-24 21:44:03',
1031905060000068,
'2019-05-24 21:44:03',
'101629735' )
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 161 page no 4407 n bits 656 index INX_ACCOUNT_ID of table `jliot`.`jp_inter_caller` trx id 202714835 lock_mode X locks gap before rec inse rt intention waiting
Record lock, heap no 365 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 1; hex 31; asc 1;;
1: len 9; hex 313031363239373335; asc 101629735;;
2: len 6; hex 000000029213; asc ;;
*** (2) TRANSACTION:
TRANSACTION 202714838, ACTIVE 1 sec fetching rows
mysql tables in use 2, locked 2
3143 lock struct(s), heap size 368848, 250566 row lock(s)
MySQL thread id 1399235, OS thread handle 140294617560832, query id 273875417 wl wjasper1 10.10.2.171 iotuser Sending data
update JP_INTER_CALLER
INNER JOIN (SELECT IF(COUNT(*) < 5,1,0) AS EXECUTING FROM JP_INTER_CA LLER WHERE USE_TAG = '1' AND ACCOUNT_ID = '101629735' ) j
SET USE_TAG = j.EXECUTING,
UPDATE_TIME = '2019-05-24 21:44:03'
WHERE NUMBER = 1051905230166054
and ACCOUNT_ID = '101629735'
AND USE_TAG = '0' AND j.EXECUTING='1'
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 161 page no 4407 n bits 656 index INX_ACCOUNT_ID of table `jliot`.`jp_inter_caller` trx id 202714838 lock mode S
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
Record lock, heap no 365 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 1; hex 31; asc 1;;
1: len 9; hex 313031363239373335; asc 101629735;;
2: len 6; hex 000000029213; asc ;;
Record lock, heap no 588 PHYSICAL RECORD: n_fields 3; compact format; info bits 0
0: len 1; hex 31; asc 1;;
1: len 9; hex 313031363239373335; asc 101629735;;
2: len 6; hex 000000029252; asc R;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 161 page no 5208 n bits 104 index GEN_CLUST_INDEX of table `jliot`.`jp_inter_caller` trx id 202714838 lock_mode X waiting
Record lock, heap no 33 PHYSICAL RECORD: n_fields 24; compact format; info bits 0
0: len 6; hex 000000040146; asc F;;
1: len 6; hex 00000c152ed3; asc . ;;
2: len 7; hex e200000199114b; asc K;;
3: len 8; hex 80100d010e52815a; asc R Z;;
4: len 8; hex 8000000000000001; asc ;;
5: len 20; hex 3839383630363137303630303639393734323032; asc 898606170600699742 02;;
6: SQL NULL;
7: SQL NULL;
8: SQL NULL;
9: len 1; hex 31; asc 1;;
10: SQL NULL;
11: SQL NULL;
12: SQL NULL;
13: SQL NULL;
14: SQL NULL;
15: SQL NULL;
16: len 1; hex 30; asc 0;;
17: len 4; hex 5ce7f523; asc \ #;;
18: len 4; hex 5ce7f523; asc \ #;;
19: SQL NULL;
20: len 16; hex 31303331393035303630303030303638; asc 1031905060000068;;
21: SQL NULL;
22: len 4; hex 5ce7f523; asc \ #;;
23: len 9; hex 313031363239373335; asc 101629735;;
*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------
Trx id counter 202715616
Purge done for trx's n:o < 202715616 undo n:o < 0 state: running but idle
History list length 26
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421776435299888, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435295672, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435312536, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435304104, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435308320, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435291456, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435333616, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435337832, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435283024, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435278808, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435274592, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435270376, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435257728, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435266160, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435261944, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435249296, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421776435253512, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 202715613, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
178 lock struct(s), heap size 24784, 39411 row lock(s)
MySQL thread id 1217887, OS thread handle 140294629988096, query id 273877258 wl wjasper1 10.10.2.171 iotuser Sending data
update JP_INTER_CALLER
INNER JOIN (SELECT IF(COUNT(*) < 5,1,0) AS EXECUTING FROM JP_INTER_CA LLER WHERE USE_TAG = '1' AND ACCOUNT_ID = '101629735' ) j
SET USE_TAG = j.EXECUTING,
UPDATE_TIME = '2019-05-24 21:44:28'
WHERE NUMBER = 1051905230166078
and ACCOUNT_ID = '101629735'
AND USE_TAG = '0' AND j.EXECUTING='1'
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
5014 OS file reads, 22823425 OS file writes, 12492173 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 19.53 writes/s, 9.53 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 31 merges
merged operations:
insert 44, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 553253, node heap has 276 buffer(s)
Hash table size 553253, node heap has 172 buffer(s)
Hash table size 553253, node heap has 503 buffer(s)
Hash table size 553253, node heap has 2 buffer(s)
Hash table size 553253, node heap has 15 buffer(s)
Hash table size 553253, node heap has 350 buffer(s)
Hash table size 553253, node heap has 8 buffer(s)
Hash table size 553253, node heap has 2 buffer(s)
57096.92 hash searches/s, 610.72 non-hash searches/s
---
LOG
---
Log sequence number 15575607624
Log flushed up to 15575606070
Pages flushed up to 15574801559
Last checkpoint at 15574748418
0 pending log flushes, 0 pending chkp writes
3667135 log i/o's done, 1.39 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 2230321152
Dictionary memory allocated 4226497
Buffer pool size 131072
Free buffers 41907
Database pages 87836
Old database pages 32259
Modified db pages 261
Percent of dirty pages(LRU & free pages): 0.201
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 524, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 4386, created 83450, written 15112302
0.00 reads/s, 0.21 creates/s, 14.95 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 87836, unzip_LRU len: 0
I/O sum[0]:cur[1640], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size 16384
Free buffers 5543
Database pages 10676
Old database pages 3920
Modified db pages 16
Percent of dirty pages(LRU & free pages): 0.099
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 43, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 561, created 10115, written 1216205
0.00 reads/s, 0.02 creates/s, 1.52 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 10676, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size 16384
Free buffers 4928
Database pages 11294
Old database pages 4149
Modified db pages 5
Percent of dirty pages(LRU & free pages): 0.031
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 80, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 561, created 10733, written 2046132
0.00 reads/s, 0.02 creates/s, 1.60 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 11294, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size 16384
Free buffers 5267
Database pages 10941
Old database pages 4018
Modified db pages 11
Percent of dirty pages(LRU & free pages): 0.068
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 47, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 600, created 10341, written 1980801
0.00 reads/s, 0.02 creates/s, 1.15 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 10941, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size 16384
Free buffers 4990
Database pages 11221
Old database pages 4122
Modified db pages 46
Percent of dirty pages(LRU & free pages): 0.284
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 46, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 512, created 10709, written 2310191
0.00 reads/s, 0.00 creates/s, 2.26 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 11221, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size 16384
Free buffers 5370
Database pages 10849
Old database pages 3984
Modified db pages 54
Percent of dirty pages(LRU & free pages): 0.333
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 104, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 467, created 10382, written 2036017
0.00 reads/s, 0.00 creates/s, 1.89 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 10849, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size 16384
Free buffers 5322
Database pages 10903
Old database pages 4004
Modified db pages 55
Percent of dirty pages(LRU & free pages): 0.339
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 30, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 486, created 10417, written 2225958
0.00 reads/s, 0.02 creates/s, 2.47 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 10903, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size 16384
Free buffers 5271
Database pages 10943
Old database pages 4019
Modified db pages 53
Percent of dirty pages(LRU & free pages): 0.327
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 149, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 612, created 10331, written 2535189
0.00 reads/s, 0.02 creates/s, 2.29 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 10943, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size 16384
Free buffers 5216
Database pages 11009
Old database pages 4043
Modified db pages 21
Percent of dirty pages(LRU & free pages): 0.129
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 25, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 587, created 10422, written 761809
0.00 reads/s, 0.13 creates/s, 1.79 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 11009, unzip_LRU len: 0
I/O sum[0]:cur[205], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=10259, Main thread ID=140298356655872, state: sleeping
Number of rows inserted 2760569, updated 2693868, deleted 248879, read 497579099 864
7.65 inserts/s, 4.60 updates/s, 0.05 deletes/s, 61833.94 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 138618
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
1 row in set (0.00 sec)
ERROR: No query specified
参考资料：
https://blog.csdn.net/varyall/article/details/80219459