Oracle EBS R12 - adpatch在adutlrcmp.sql卡住，utlrp.sql卡住，shutdown immediate又卡住

Oracle EBS R12 - adpatch在adutlrcmp.sql卡住，utlrp.sql卡住，shutdown immediate又卡住


Version:
AP: Oracle EBS R12.0.6
DB: Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit


adpatch hang在compile invalids阶段，只得ctrl+c中断掉adutlrcmp.sql让adpatch先走下去，最后再执行utlrp.sql手动编译invalids:

...
STRT_TASK: [Compile invalid objects in DB] [] [Fri Aug 16 2013 03:07:08]


Compiling invalid objects...



STRT_TASK: [Run ST parallel compile (APPS)] [] [Fri Aug 16 2013 03:07:08]


sqlplus -s APPS/***** @/dabcdi/applmgr/1200/ad/12.0.0/sql/adutlrcmp.sql APPLSYS ***** APPS ***** ***** 8  0 NONE FALSE

Arguments are:

AOL_schema = APPLSYS, AOL_password = *****,
Schema_to_compile = APPS, Schema_to_compile_pw = *****,
SYSTEM_password = *****, Total_workers = 8, Logical_worker_num = 0
Object_type_to_not_compile = NONE
Use_stored_dependencies = FALSE

Connected.

Running utl_recomp.recomp_parallel(8), if it exists


OWNER                          NUM_INVALID
------------------------------ -----------
BOLINF                                   6
APPSREAD                                 1
OHS_ACCNT                                6
INV                                      1
BIAPPS                                   5
SOA                                      5
APPS                                    13
SYSMAN                                  21
RAC_ACCNT                                6

9 rows selected.

Elapsed: 00:00:02.16

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.01

STATUS_MESSAGE
------------------------------------------------------------------------------
Running UTL_RECOMP.RECOMP_PARALLEL...

declare
*
ERROR at line 1:
ORA-01013: user requested cancel of current operation



AutoPatch error:
Error running adutlrcmp.sql in adprdcAdUtilrcmp()


An error occurred while compiling invalids using the SYS.UTL_RECOMP() package.
Ignoring the failure and continuing...



STOP_TASK: [Run ST parallel compile (APPS)] [] [Fri Aug 16 2013 03:29:24]



STRT_TASK: [Run ST parallel compile (APPS) second pass] [] [Fri Aug 16 2013 03:29:24]


sqlplus -s APPS/***** @/dabcdi/applmgr/1200/ad/12.0.0/sql/adutlrcmp.sql APPLSYS ***** APPS ***** ***** 8  0 NONE FALSE

AutoPatch error:
Error running adutlrcmp.sql in adprdcAdUtilrcmp()


An error occurred while compiling invalids using the SYS.UTL_RECOMP() package.
Ignoring the failure and continuing...



STOP_TASK: [Run ST parallel compile (APPS) second pass] [] [Fri Aug 16 2013 03:29:24]



STOP_TASK: [Compile invalid objects in DB] [] [Fri Aug 16 2013 03:29:24]



STRT_TASK: [Get final list of invalid objects in DB] [] [Fri Aug 16 2013 03:29:24]


Number of invalid objects: 12

Getting list of invalid objects in APPS schema.

Invalid object reports are recorded in
/dabcdi/applmgr/1200/admin/DABCDI/log/151927_postenv.*


STOP_TASK: [Get final list of invalid objects in DB] [] [Fri Aug 16 2013 03:29:28]



STOP_TASK: [Database portion steps] [] [Fri Aug 16 2013 03:29:28]



STRT_TASK: [Generate portion steps] [] [Fri Aug 16 2013 03:29:28]


...




尝试手动在db节点上执行re-complie,还是hang住:
@?/rdbms/admin/utlrp.sql


查看 v$session_wait, 发现编译的session卡在 library cache pin 这个等待事件，一时没想到怎么解决，打算索性重起下db，没想到重起db又在 shutdown immediate 时hang住。


查看alert log，发现是卡在等待一个进程退出，kill掉进程后之前卡住shutdown可以正常结束了：
-bash-3.2$ tail alert_DABCDI.log
Fri Aug 16 03:39:47 CUT 2013
Stopping background process MMON
License high water mark = 221
Fri Aug 16 03:39:49 CUT 2013
Job queue slave processes stopped
Fri Aug 16 03:44:48 CUT 2013
Active call for process 20639 user 'ordabcdi' program 'oracle@vmohsabcd002'
System State dumped to trace file /dabcdi/log/udump/dabcdi_ora_5101.trc
Fri Aug 16 03:45:06 CUT 2013
SHUTDOWN: waiting for active calls to complete.
-bash-3.2$ ps -ef | grep 20639
ordabcdi 11760  9451  0 03:47 pts/9    00:00:00 grep 20639
ordabcdi 20639     1  3 Aug12 ?        02:48:23 oracleDABCDI (LOCAL=NO)
-bash-3.2$ kill 20639
-bash-3.2$ ps -ef | grep 20639
ordabcdi 11797  9451  0 03:47 pts/9    00:00:00 grep 20639
ordabcdi 20639     1  3 Aug12 ?        02:48:24 oracleDABCDI (LOCAL=NO)
-bash-3.2$ kill -9 20639
-bash-3.2$ ps -ef | grep 20639
ordabcdi 11845  9451  0 03:47 pts/9    00:00:00 grep 20639
-bash-3.2$


最后，startup数据库，再执行utlrp.sql就不会卡住了。

REF:

0. Oracle EBS R12.1.3 打patch出现hang住的现象如何解决
http://blog.itpub.net/35489/viewspace-749417/

1. [原创]Oracle关闭不了，SHUTDOWN: waiting for active calls to complete.
http://blog.vsharing.com/songtao1971/A784641.html

2. Library Cache Lock、Library Cache Pin、Library Cache Load Lock等待事件
http://space.itpub.net/23135684/viewspace-731593