2016-1-8 ORA-1652: unable to extend temp segment by 128 in tablespace解决方案
一次ORA-1652的诊断过程，系统不能使用，重启后可以使用。
weblogic日志：
####<2016-1-8 上午09时40分27秒 CST> <Error> <WebLogicServer> <zc-sc-app01> <fwms_app01_8003> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1452217227682> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "603" seconds working on the request "Http Request Information: weblogic.servlet.internal.ServletRequestImpl@5d4d331d[GET /web/defect/extend-attr/5b469596984e4c53ae0f4cdafd3fd39b]
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:152)
java.net.SocketInputStream.read(SocketInputStream.java:122)
oracle.net.ns.Packet.receive(Packet.java:300)
oracle.net.ns.DataPacket.receive(DataPacket.java:106)
####<2016-1-8 上午09时56分53秒 CST> <Error> <WebLogicServer> <zc-tzjh-app1> <IPMS_APP01_8101> <[STANDBY] ExecuteThread: '114' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1452218213276> <BEA-000337> <[STUCK] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "607" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@56d8af13[GET /web/menu_icon/zongheguanli.png?tSession=1452217537514 HTTP/1.1
X-Forwarded-For: 10.119.124.109
Accept: */*
Referer: http://10.121.180.109/web/app/MyApp.jsp
Accept-Language: zh-CN
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; GWX:DOWNLOADED)
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Cookie: DWRSESSIONID=UmKxc9wQCOJBrOHXa$Zx1oNVu9l; JSESSIONIDMINI=BtcLWPJprp2xpBhqbsgv7NzKmJ8yLZhvxFLpThyGcz50JzGyh4vH!2102636630; ComtopSessionSID=<SNAID>BtcLWPJprp2xpBhqbsgv7NzKmJ8yLZhvxFLpThyGcz50JzGyh4vH!2102636630!1452217406173</SNAID>; cs_cStorage_=1
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:385)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:342)
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:329)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:417)
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:324)
alert日志：
Fri Jan 08 09:41:39 2016
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP
Fri Jan 08 09:41:39 2016
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP
Fri Jan 08 10:01:36 2016
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP
Fri Jan 08 10:11:35 2016
ORA-1652: unable to extend temp segment by 128 in tablespace TEMP
AWR:
| Snap Id | Snap Time | Sessions | Cursors/Session | Instances | |
|---|---|---|---|---|---|
| Begin Snap: | 5732 | 08-Jan-16 09:00:48 | 548 | 73.3 | 2 |
| End Snap: | 5733 | 08-Jan-16 10:00:59 | 545 | 73.0 | 2 |
| Elapsed: | 60.18 (mins) | ||||
| DB Time: | 126.97 (mins) |
| Event | Waits | Total Wait Time (sec) | Wait Avg(ms) | % DB time | Wait Class |
|---|---|---|---|---|---|
| DB CPU | 6414.3 | 84.2 | |||
| direct path write temp | 323,000 | 479.8 | 1 | 6.3 | User I/O |
| Elapsed Time (s) | Executions | Elapsed Time per Exec (s) | %Total | %CPU | %IO | SQL Id | SQL Module | SQL Text |
|---|---|---|---|---|---|---|---|---|
| 3,579.67 | 1,666 | 2.15 | 46.99 | 99.76 | 0.00 | ga37qxa3p485g | JDBC Thin Client | select tddakentit0_.ID as ID66... |
| 2,917.88 | 6 | 486.31 | 38.30 | 81.96 | 16.41 | dxcdycgfr1w5n | select nvl(round(avg(ctime), 2... | |
| 190.66 | 6 | 31.78 | 2.50 | 14.15 | 85.29 | 8yavyxsts8hu9 | select a.tablespace_name table... | |
| 28.21 | 6 | 4.70 | 0.37 | 99.74 | 0.00 | brb01q8dygvy6 | select SQL_TEXT d_maxtime_sql ... | |
| 16.66 | 6 | 2.78 | 0.22 | 99.74 | 0.00 | 45b35s83zwmah | select round(max(ELAPSED_TIME)... | |
| 11.57 | 2 | 5.79 | 0.15 | 58.51 | 0.00 | 4x91zq0fkd05y | DECLARE job BINARY_INTEGER := ... | |
| 11.57 | 6 | 1.93 | 0.15 | 58.51 | 0.00 | 4snzdp176n9wv | CALL C_DATA_SYNCH_MAI... | |
| 11.34 | 2 | 5.67 | 0.15 | 59.11 | 0.00 | 5k565njgcd272 | CALL G_DATA_SYNCH('MM... | |
| 7.74 | 246 | 0.03 | 0.10 | 97.86 | 0.00 | 2h1xrpzcygsdz | JDBC Thin Client | select distinct p.portlet_id, ... |
| 6.70 | 1,729 | 0.00 | 0.09 | 87.11 | 0.00 | dfb1dmsvm3nx8 | JDBC Thin Client | select m.media_id, m.storage_p... |
检查temp表空间select tablespace_name,file_name,bytes/1024/1024 file_size,autoextensible from dba_temp_files;
发现有32G，且是自动扩展的。
select sql_id, count(*) from gv$session where event = 'direct path write temp'
group by sql_id order by count(*) desc;
--查询无数据
select sql_id, count(*) from dba_hist_active_sess_history
where event = 'direct path write temp'
group by sql_id order by count(*) desc;
SQL_ID COUNT(*)
------------- ----------
dxcdycgfr1w5n 71739
3v1z2vmmzadm4 16
bmndrgc7xkmjb 12
3mnjv5g06mv0x 7
看sql_id为dxcdycgfr1w5n就是AWR中那条执行慢的SQL
| dxcdycgfr1w5n | select nvl(round(avg(ctime), 2), 0) d_lock_wait_time from v$lock where REQUEST > 0 |
先看下这条SQL的执行计划：
select nvl(round(avg(ctime),2),0) d_lock_wait_time from v$lock whereREQUEST > 0
Plan hash value: 2399206389
---------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)|
---------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 1 (100)|
| 1 | SORT AGGREGATE | | 1 | 63 | |
|* 2 | HASH JOIN | | 1 | 63 | 0 (0)|
| 3 | MERGE JOIN CARTESIAN | | 100 | 3800 | 0 (0)|
|* 4 | FIXED TABLE FULL | X$KSUSE | 1 | 19 | 0 (0)|
| 5 | BUFFER SORT | | 100 | 1900 | 0 (0)|
| 6 | FIXED TABLE FULL | X$KSQRS | 100 | 1900 | 0 (0)|
| 7 | VIEW | GV$_LOCK | 10 | 250 | 0 (0)|
| 8 | UNION-ALL | | | | |
|* 9 | FILTER | | | | |
| 10 | VIEW | GV$_LOCK1 | 2 | 50 | 0 (0)|
| 11 | UNION-ALL | | | | |
|* 12 | FIXED TABLE FULL| X$KDNSSF | 1 | 77 | 0 (0)|
|* 13 | FIXED TABLE FULL| X$KSQEQ | 1 | 77 | 0 (0)|
|* 14 | FIXED TABLE FULL | X$KTADM | 1 | 77 | 0 (0)|
|* 15 | FIXED TABLE FULL | X$KTATRFIL | 1 | 77 | 0 (0)|
|* 16 | FIXED TABLE FULL | X$KTATRFSL | 1 | 77 | 0 (0)|
|* 17 | FIXED TABLE FULL | X$KTATL | 1 | 77 | 0 (0)|
|* 18 | FIXED TABLE FULL | X$KTSTUSC | 1 | 77 | 0 (0)|
|* 19 | FIXED TABLE FULL | X$KTSTUSS | 1 | 77 | 0 (0)|
|* 20 | FIXED TABLE FULL | X$KTSTUSG | 1 | 77 | 0 (0)|
|* 21 | FIXED TABLE FULL | X$KTCXB | 1 | 77 | 0 (0)|
---------------------------------------------------------------------------
实际测试：
SQL> select nvl(round(avg(ctime), 2), 0) d_lock_wait_time from v$lock where REQUEST > 0;
select nvl(round(avg(ctime), 2), 0) d_lock_wait_time from v$lock where REQUEST > 0
ERROR at line 1:
ORA-01652: unable to extend tem segment by 128 in tablespace TEMP
Elapsed:00:07:50.70
SQL>exec dbms_stats.gather_fixed_objects_stats();
再次执行只需要0.3s，执行计划不是笛卡尔积了。
对于RAC的情况下，两个节点都要收集，不然，还有问题。