enq: SQ – contention
下午三点多数据库出现堵塞现象，所有访问序列的会话均hang住，后台出现大量enq: SQ – contention、library cache pin、cursor: pin S wait on X等待事件。
检查故障时间段的等待事件，发现出现大量的enq: SQ – contention、library cache pin、cursor: pin S wait on X，如下图红色标注所示：
SQL> selectblocking_session,event,count(*)
2 from dba_hist_active_sess_history
3 where
4 sample_time >to_date('2013-01-09 14:00','yyyy-mm-dd hh24:mi')
and sample_time <to_date('2013-01-09 16:00','yyyy-mm-dd hh24:mi')
group by blocking_session,event
order by count(*)
; 5 6 7 8
：：：省略显示：：：
BLOCKING_SESSIONEVENT COUNT(*)
------------------------------------------------------------------ ----------
668 enq: TX - row lockcontention 247
2486 enq: TX - row lock contention 269
4386 enq: TX - row lockcontention 273
4956 library cache lock 313
4575 enq: TX - row lockcontention 328
3626 enq: TX - row lockcontention 507
4381 enq: SQ - contention 530
2762 library cache lock 531
KJC: Wait for msg sends tocomplete 586
5335 enq: TX - row lockcontention 621
2961 enq: TX - row lockcontention 1066
BLOCKING_SESSIONEVENT COUNT(*)
------------------------------------------------------------------ ----------
423 enq: SQ - contention 1346
4956 cursor: pin S wait on X 1747
3102
library cache pin 3982
2490 enq: SQ - contention 8890
2289 enq: SQ - contention 85522
182 rowsselected.
从上图可以看出累计的session 2289、2490、4956阻塞其他会话的次数最多，接下来我们看这些阻塞者又被哪些会话阻塞。
SQL> selectSESSION_ID,USER_ID,SQL_ID,BLOCKING_SESSION,event,sample_time
2 from DBA_HIST_ACTIVE_SESS_HISTORY
3 where
session_id = &sid
and
sample_time >to_date('2013-01-09 14:00','yyyy-mm-dd hh24:mi')
and
sample_time <to_date('2013-01-09 16:00','yyyy-mm-dd hh24:mi')
order by sample_time
; 4 5 6 7 8 9 10
Enter value forsid: 2490
old 4: session_id = &sid
new 4: session_id = 2490
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ---------------------------------------------------------------------------------------------------------------------------------------------
2490 100 3p7yc5vjap1uq row cache lock 09-JAN-1302.59.25.314 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1302.59.35.397 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1302.59.45.483 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1302.59.55.600 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.00.05.664 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.00.15.757 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.00.25.851 PM
：：：
2490 100 3p7yc5vjap1uq library cache pin 09-JAN-1303.19.26.631 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.19.36.122 PM
2490 100 3p7yc5vjap1uq library cache pin 09-JAN-1303.19.36.792 PM
2490 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.19.46.212 PM
2490 100 3p7yc5vjap1uq library cache pin 09-JAN-1303.19.46.924 PM
：：：
2490 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.28.15.194 PM
2490 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.28.25.367 PM
2490 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.28.35.539 PM
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ---------------------------------------------------------------------------------------------------------------------------------------------
2490 100 3p7yc5vjap1uq 4381 enq: SQ - contention 09-JAN-1303.28.45.683 PM
221 rowsselected.
会话2490被2289阻塞。等待事件为enq: SQ – contention、library cache pin、row cache lock。
SQL> /
Enter value forsid: 4956
old 4: session_id = &sid
new 4: session_id = 4956
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ------------------------------------------------------------------ ---------------------------------------------------------------------------
4956 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.23.20.331 PM
4956 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.23.30.464 PM
4956 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.23.40.630 PM
4956 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.23.50.832 PM
4956 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.24.00.964 PM
：：：
会话4956也是被2289阻塞。等待事件为enq: SQ – contention。
SQL> /
Enter value forsid: 423
old 4: session_id = &sid
new 4: session_id = 423
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ---------------------------------------------------------------------------------------------------------------------------------------------
423 100 3p7yc5vjap1uq library cache lock 09-JAN-1303.21.27.097 PM
423 100 3p7yc5vjap1uq 4566 enq: SQ - contention 09-JAN-1303.21.37.157 PM
423 100 3p7yc5vjap1uq 5706 enq: SQ - contention 09-JAN-1303.21.47.247 PM
423 100 3p7yc5vjap1uq 5706 enq: SQ - contention 09-JAN-1303.21.57.359 PM
423 100 3p7yc5vjap1uq 5706 enq: SQ - contention 09-JAN-1303.22.07.419 PM
423 100 3p7yc5vjap1uq 5898 enq: SQ - contention 09-JAN-1303.22.17.507 PM
423 100 3p7yc5vjap1uq 5995 enq: SQ - contention 09-JAN-1303.22.27.596 PM
423 100 3p7yc5vjap1uq row cache lock 09-JAN-1303.22.37.654 PM
423 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.22.47.732 PM
423 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.22.57.840 PM
423 100 3p7yc5vjap1uq 2289 row cache lock 09-JAN-1303.23.07.901 PM
：：：
423 100 9mmda9cp0n4vs 1149 cursor: pin S wait on X 09-JAN-1303.32.53.046 PM
423 100 9mmda9cp0n4vs 1149 cursor: pin S wait on X 09-JAN-1303.33.03.150 PM
423 100 9mmda9cp0n4vs 1149 cursor: pin S wait on X 09-JAN-1303.33.13.239 PM
423 100 9mmda9cp0n4vs 1149cursor: pin S wait on X 09-JAN-1303.33.23.308 PM
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
---------- ----------------------- ---------------------------------------------------------------------------------------------------------------------------------------------
423 100 9mmda9cp0n4vs 1149 cursor: pin S wait on X 09-JAN-13 03.33.33.396 PM
423 100 9mmda9cp0n4vs 1149 cursor: pin S wait on X 09-JAN-1303.33.43.484 PM
423 95 2dv9uznd1sh28 db file scattered read 09-JAN-13 03.53.32.015 PM
69 rowsselected.
会话423被2289、1149、5076等阻塞过，但根据时间可以断定故障时刻最终阻塞423的是1149。那我们再看一下谁阻塞了1149。
SQL> /
Enter value forsid: 1149
old 4: session_id = &sid
new 4: session_id = 1149
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ------------------------------------------------------------------ ---------------------------------------------------------------------------
1149 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.00.00.337 PM
1149 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.00.10.435 PM
1149 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.00.20.539 PM
1149 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.00.30.616 PM
1149 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.00.40.723 PM
1149 100 3p7yc5vjap1uq 2289 enq: SQ - contention 09-JAN-1303.00.50.857 PM
：：：
会话1149又被2289阻塞过，这里虽然不能表明循环阻塞，但至少可以看出所有阻塞都来自2289。那我们来看一下2289到底被谁阻塞:
如下图：
SQL> selectSESSION_ID,USER_ID,SQL_ID,BLOCKING_SESSION,event,SAMPLE_TIME
2 from DBA_HIST_ACTIVE_SESS_HISTORY
3 where
session_id = 2289
and
sample_time >to_date('2013-01-09 14:00','yyyy-mm-dd hh24:mi')
and
sample_time <to_date('2013-01-09 16:00','yyyy-mm-dd hh24:mi')
; 4 5 6 7 8 9
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ---------------- -----------------------------------------------------------------------------------------------------------------------------
2289 100 4f1p23y83tdzj 09-JAN-13 03.57.19.032 PM
2289 100 55r3z51xwu3c3 09-JAN-13 03.56.58.843 PM
2289 100 55r3z51xwu3c3 09-JAN-13 03.54.17.476 PM
:::
2289 100 9q1fmpuvyxftd KJC: Wait for msg sends tocomplete 09-JAN-1302.01.00.476 PM
2289 100 9q1fmpuvyxftd KJC: Wait for msg sends tocomplete 09-JAN-1302.00.50.414 PM
2289 100 9q1fmpuvyxftd KJC: Wait for msg sends tocomplete 09-JAN-1302.00.40.289 PM
2289 100 9q1fmpuvyxftd KJC: Wait for msg sends tocomplete 09-JAN-1302.00.30.188 PM
2289 100 9q1fmpuvyxftd KJC: Wait for msg sends tocomplete 09-JAN-1302.00.20.108 PM
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ---------------------------------------------------------------------------------------------------------------------------------------------
2289 100 9q1fmpuvyxftd KJC: Wait for msg sends tocomplete 09-JAN-13 02.00.09.987 PM
617 rowsselected.
可以看出会话2289的阻塞列为空，这往往表示不是由某一个用户进程阻塞。等待事件为KJC: Wait for msg sends to complete。那么接下来我们就针对KJC: Wait for msg sends to complete事件来讨论。对于事件KJC: Wait for msg sends to complete，表示集群内消息传递出现问题。我们也可以从故障前半小时的AWR中看出，全局队列响应时间为76.5ms,太长了，正常情况可以接受的最佳时间为1ms以下，这说明全局队列或消息出现问题。
Global Cache and Enqueue Services - WorkloadCharacteristics
| Avg global enqueue get time (ms): |
76.5 |
| Avg global cache cr block receive time (ms): |
0.7 |
| Avg global cache current block receive time (ms): |
0.6 |
| Avg global cache cr block build time (ms): |
0.0 |
| Avg global cache cr block send time (ms): |
0.0 |
| Global cache log flushes for cr blocks served %: |
3.2 |
| Avg global cache cr block flush time (ms): |
1.0 |
| Avg global cache current block pin time (ms): |
0.0 |
| Avg global cache current block send time (ms): |
0.0 |
| Global cache log flushes for current blocks served %: |
0.0 |
| Avg global cache current block flush time (ms): |
2.0 |
那我们再看一下该会话何时出现KJC: Wait for msg sends to complete等待事件？
SQL> selectSESSION_ID,USER_ID,SQL_ID,BLOCKING_SESSION,event,sample_time
2 from DBA_HIST_ACTIVE_SESS_HISTORY
3 where
4 session_id = &sid
and
sample_time <to_date('2013-01-09 14:00','yyyy-mm-dd hh24:mi')
order by sample_time
; 5 6 7 8
Enter value forsid: 2289
old 4: session_id = &sid
new 4: session_id = 2289
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ------------------------------------------------------------------ ---------------------------------------------------------------------------
2289 100 55r3z51xwu3c3 09-JAN-13 03.35.40.544 AM
2289 100 4f1p23y83tdzj 09-JAN-1303.36.10.759 AM
2289 100 3s6jnux9tyhb6 09-JAN-13 05.20.44.967 AM
2289 100 4f1p23y83tdzj 09-JAN-1307.27.20.877 AM
2289 100 4f1p23y83tdzj 09-JAN-13 07.27.30.947 AM
2289 100 4f1p23y83tdzj 09-JAN-1307.27.41.055 AM
2289 100 4f1p23y83tdzj 09-JAN-13 07.27.51.185 AM
2289 100 4f1p23y83tdzj 09-JAN-1307.30.32.508 AM
2289 100 btbb4djs2us0r 09-JAN-13 08.08.52.572 AM
2289 100 bn3zsa3r8vrcf 09-JAN-1308.09.42.990 AM
2289 100 bmfc989p02cv6 KJC: Wait for msg sends tocomplete 09-JAN-1308.11.14.017 AM
SESSION_ID USER_ID SQL_ID BLOCKING_SESSION EVENT SAMPLE_TIME
-------------------- ------------- ---------------------------------------------------------------------------------------------------------------------------------------------
2289 100 bmfc989p02cv6 KJC: Wait for msg sends tocomplete 09-JAN-1308.11.24.155 AM
2289 100 bmfc989p02cv6 KJC: Wait for msg sends tocomplete 09-JAN-1308.11.34.281 AM
2289 100 bmfc989p02cv6 KJC: Wait for msg sends to complete 09-JAN-13 08.11.44.464 AM
2289 100 bmfc989p02cv6 KJC: Wait for msg sends tocomplete 09-JAN-1308.11.54.633 AM
从上面的时间排序，我们可以看到该事件从上午8:09:42开始产生。那么这个时刻到底数据库出现什么问题呢？
检查alert_emdb1.log，我们发现在1月9日8点8分数据库报ORA-04030错误，如下：
Wed Jan 0902:59:58 2013
CJQ0 startedwith pid=55, OS id=28635
Wed Jan 0903:01:46 2013
XDB installed.
XDB initialized.
Wed Jan 0903:04:14 2013
SERVER COMPONENTid=UTLRP_END: timestamp=2013-01-09 03:04:14
Wed Jan 0908:08:33 2013
Errors in file /opt/oracle/app/oracle/diag/rdbms/emdb/emdb1/trace/emdb1_ora_6109.trc (incident=663836):
ORA-04030:out of process memory when trying to allocate 29600 bytes (kkoutlCreatePh,apppred: kkotbalp)
Incidentdetails in:/opt/oracle/app/oracle/diag/rdbms/emdb/emdb1/incident/incdir_663836/emdb1_ora_6109_i663836.trc
Use ADRCI orSupport Workbench to package the incident.
See Note 411.1at My Oracle Support for error and packaging details.
Wed Jan 0908:08:33 2013
Process startupfailed, error stack:
Wed Jan 0908:08:34 2013
从TRACE中/opt/oracle/app/oracle/diag/rdbms/emdb/emdb1/trace/emdb1_ora_6109.trc中我们发现出现ORA-4030时，该进程正在运行PLSQL“FUNCTION UOP_MALL.F_SYS_GETSEQID”，如下所示
*** 2013-01-0908:10:11.766
*********STARTPLSQL RUNTIME DUMP************
***Got internalerror Exception caught in pl/sql run-time while running PLSQL***
***Got ORA-4030while running PLSQL***
FUNCTIONUOP_MALL.F_SYS_GETSEQID:
libraryunit=138ed00a78 line=1 opcode=0 static link=0 scope=0
FP=ffffffff7ab31208PC=124b425892 Page=0 AP=ffffffff7ab31208 ST=ffffffff7ab322a0
DL0=ffffffff7dfb2368GF=ffffffff7dfb23c0 DL1=ffffffff7dfb2388 DPF=ffffffff7dfb23b0 DS=124b425a98
经检查FUNCTION UOP_MALL.F_SYS_GETSEQID的代码，我们发现代码中含有当前堵塞的序列SEQ_SALTING_ID.NEXTVAL，见下图，从这里可以看出这个session正在获取sequence，此时因ORA-04030错误导致意外终止，终止后系统并没有释放这个资源，因此才导致后面一旦获取这个序列就堵塞的现象。
SQL> selectowner,object_type from dba_objects where object_name = 'F_SYS_GETSEQID';
OWNER OBJECT_TYPE
-------------------------------------------------
UOP_CUSTOM FUNCTION
UMT_CUSTOM SYNONYM
UOP_MALL FUNCTION
UMT_MALL SYNONYM
SQL> setheading off;
SQL> set echooff;
SQL> setpages 999;
SQL> set long90000;
SQL> selectdbms_metadata.get_ddl('FUNCTION','F_SYS_GETSEQID','UOP_MALL') from dual;
CREATE OR REPLACE FUNCTION"UOP_MALL"."F_SYS_GETSEQID" (V_SEQNAME IN VARCHAR2,
V_LENGTH IN NUMBER DEFAULT 16)
RETURN VARCHAR2 IS
IV_SALT VARCHAR2(2);
IV_DATE VARCHAR2(8);
IV_SEQNAME VARCHAR2(50);
IV_SQLSTR VARCHAR2(200);
IV_SEQ VARCHAR2(16);
IV_SEQID VARCHAR2(16);
BEGIN
IV_SEQNAME := LOWER(TRIM(V_SEQNAME));
IV_SQLSTR := 'SELECT '||IV_SEQNAME||'.nextval FROM DUAL';
EXECUTE IMMEDIATE IV_SQLSTR INTO IV_SEQ;
IF LENGTH(IV_SEQ) < 8 THEN
IV_SEQ :=LPAD(SUBSTR(IV_SEQ,-LENGTH(IV_SEQ)),8,'0');
ELSE
IV_SEQ := LPAD(SUBSTR(IV_SEQ,-8),8,'0');
END IF;
SELECT LPAD(SEQ_SALTING_ID.NEXTVAL,2, '0') INTO IV_SALT FROM DUAL;
IF V_LENGTH > 10 THEN
SELECT TO_CHAR(SYSDATE, 'yymmdd') INTOIV_DATE FROM DUAL;
IV_SEQID := IV_SALT || IV_DATE ||
LPAD(SUBSTR(IV_SEQ, 8 -V_LENGTH), V_LENGTH - 8, '0');
ELSE
IV_SEQID := IV_SALT ||
LPAD(TRIM(SUBSTR(IV_SEQ, 2 -V_LENGTH)), V_LENGTH - 2, '0');
END IF;
RETURN IV_SEQID;
EXCEPTION
WHEN OTHERS THEN
RETURN NULL;
END;
SQL>
那接下来我们的焦点就是什么导致了ORA-04030的错误，经咨询现场工程师，他们说这个ORA-04030是在上午8点左右由于应用问题导致的。
综上所示，我们可以肯定上午应用程序问题导致的死亡进程占用的资源还没有释放，因此积累导致下午三点多的堵塞故障发生。另外从现象上看，下午故障结束后不再有KJC: Wait for msg sends to complete等待事件产生，详细见下面查询结果。
SQL> colevent for a40
SQL> selectevent,count(*)
2 from DBA_HIST_ACTIVE_SESS_HISTORY
3 where
4 sample_time >to_date('2013-01-09 15:00','yyyy-mm-dd hh24:mi')
5 and
6 sample_time <to_date('2013-01-09 16:00','yyyy-mm-dd hh24:mi')
7 and
8 event = 'KJC: Wait for msgsends to complete'
9 group by event
10 order by count(*)
; 11
EVENT COUNT(*)
--------------------------------------------------
KJC:Wait for msg sends to complete 230
SQL> colevent for a40
SQL> selectevent,count(*)
2 from DBA_HIST_ACTIVE_SESS_HISTORY
3 where
4 sample_time >to_date('2013-01-09 16:00','yyyy-mm-dd hh24:mi')
5 and
6 event = 'KJC: Wait for msgsends to complete'
7 group by event
8 order by count(*)
9 ;
norows selected
SQL>
另外，我们也可以从故障时的TRACE emdb1_dia0_28035_1.trc中看到堵塞的原因是由'KJC: Wait for msg sends to complete'阻塞导致'enq:SQ - contention'，'enq: SQ - contention'堵塞导致<='enq: TX - row lock contention'。
-------------------------------------------------------------------------------
Chain 1:
-------------------------------------------------------------------------------
Oracle session identified by:
{
instance: 1 (emdb.emdb1)
os id: 6099
process id: 960, oracle@emdb1
session id: 15
session serial #: 3
}
is waiting for 'enq: TX - row lockcontention' with wait info:
{
p1:'name|mode'=0x54580006
p2: 'usn<<16 |slot'=0x5a0020
p3: 'sequence'=0xec53
time in wait: 1 min 48 sec
timeout after: never
wait id: 15598
blocking: 0 sessions
wait history:
* time between current wait andwait #1: 0.001751 sec
1. event: 'SQL*Net message from client'
time waited: 56.675122 sec
wait id: 15597 p1: 'driver id'=0x28444553
p2: '#bytes'=0x1
* time between wait #1 and #2:0.000056 sec
2. event: 'SQL*Net message to client'
time waited: 0.000002 sec
wait id: 15596 p1: 'driver id'=0x28444553
p2:'#bytes'=0x1
* time between wait #2 and #3:0.000184 sec
3. event: 'SQL*Net message from client'
time waited: 1 min 59 sec
wait id: 15595 p1: 'driver id'=0x28444553
p2:'#bytes'=0x1
}
and is blocked by
=> Oracle session identified by:
{
instance: 1 (emdb.emdb1)
os id: 6063
process id: 942, oracle@emdb1
session id: 4386
session serial #: 11
}
which is waiting for 'enq: SQ - contention'with wait info:
{
p1:'name|mode'=0x53510006
p2: 'object #'=0x15be5
p3: '0'=0x0
time in wait: 2 min 6 sec
timeout after: never
wait id: 702
blocking: 2 sessions
wait history:
* time between current wait andwait #1: 0.001458 sec
1. event: 'gc cr block 2-way'
time waited: 0.000614 sec
wait id: 701 p1: ''=0x60
p2: ''=0xb4b3
p3: ''=0x1
* time between wait #1 and #2:0.001911 sec
2. event: 'SQL*Net message from client'
time waited: 0.000565 sec
wait id: 700 p1: 'driver id'=0x28444553
p2: '#bytes'=0x1
* time between wait #2 and #3:0.000009 sec
3. event: 'SQL*Net message to client'
time waited: 0.000001 sec
wait id: 699 p1: 'driver id'=0x28444553
p2:'#bytes'=0x1
}
and is blocked by
=> Oracle session identified by:
{
instance: 1 (emdb.emdb1)
os id: 1908
process id: 536, oracle@emdb1
session id: 2289
session serial #: 3
}
which is waiting for 'KJC: Wait for msgsends to complete' with wait info:
{
p1: 'msg'=0x1344e37228
p2: 'dest|rcvr'=0x10000
p3: 'mtype'=0xc
time in wait: 2 min 34 sec
timeout after: 2 min 26 sec
wait id: 29912
blocking: 264 sessions
wait history:
* time between current wait andwait #1: 0.000536 sec
1. event: 'row cache lock'
time waited: 0.000359 sec
wait id: 29911 p1: 'cache id'=0xd
p2: 'mode'=0x0
p3: 'request'=0x5
* time between wait #1 and #2:0.028126 sec
2. event: 'KJC: Wait for msg sends tocomplete'
time waited: 5 min 1 sec
wait id: 29910 p1: 'msg'=0x1344e37228
p2: 'dest|rcvr'=0x10000
p3: 'mtype'=0xc
* time between wait #2 and #3:0.000471 sec
3. event: 'KJC: Wait for msg sends to complete'
time waited: 5 min 1 sec
wait id: 29909 p1: 'msg'=0x1344e37228
p2: 'dest|rcvr'=0x10000
p3: 'mtype'=0xc
}
Chain1 Signature: 'KJC: Wait for msg sends to complete'<='enq: SQ -contention'<='enq: TX - row lock contention'
最终结论：该故障是由1月9日上午八点应用程序出现问题导致死亡进程占用的资源没有释放，而引起下午三点多的堵塞故障发生。