Oracle alert 文件分析

$ adrci
ADRCI: Release 11.2.0.3.0 - Production on Mon May 20 09:59:57 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
ADR base = "/u01/app/oracle"
adrci> show alert
Choose the alert log from the following homes to view:
1: diag/rdbms/cfsdb/cfsdb1
2: diag/rdbms/hetdb/hetdb1
3: diag/rdbms/mzldb/mzldb1
4: diag/rdbms/tlodb/tlodb1
5: diag/tnslsnr/htcmdb1/listener
Q: to quit

Please select option: 4
Output the results to file: /tmp/alert_9765742_1_tlodb1_1.ado
2012-11-08 12:39:04.072000 +08:00
Adjusting the default value of parameter parallel_max_servers
from 3200 to 985 due to the value of parameter processes (1000)
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Private Interface 'en5' configured from GPnP for use as a private interconnect.
  [name='en5', type=1, ip=169.254.25.4, mac=e4-1f-13-4f-b8-da, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interc
onnect/62]
Public Interface 'en4' configured from GPnP for use as a public interface.
  [name='en4', type=1, ip=10.48.13.131, mac=e4-1f-13-4f-b8-d8, net=10.48.13.128/27, mask=255.255.255.224, use=public/1]
Public Interface 'en4' configured from GPnP for use as a public interface.
  [name='en4', type=1, ip=10.48.13.130, mac=e4-1f-13-4f-b8-d8, net=10.48.13.128/27, mask=255.255.255.224, use=public/1]
Public Interface 'en4' configured from GPnP for use as a public interface.
  [name='en4', type=1, ip=10.48.13.136, mac=e4-1f-13-4f-b8-d8, net=10.48.13.128/27, mask=255.255.255.224, use=public/1]
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 3      -- http://www.freelists.org/post/oracle-l/latchfree-SCN-scheme-10103
                                    -- Duplicate SCN Identified In Different REDO LOG Threads [ID 260304.1]
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
WARNING: db_recovery_file_dest is same as db_create_file_dest
Autotune of undo retention is turned on.      -- http://www.stechno.net/sap-notes.html?view=sapnote&id=1035137
IMODE=BR
ILAT =176
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2
System name:AIX
Node name:htcmdb1
Release:1
Version:6
Machine:00F65CCD4C00
Using parameter settings in client-side pfile /u01/app/oracle/admin/tlodb/pfile/init.ora on machine htcmdb1
System parameters with non-default values:
  processes                = 1000
 LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:4 errcnt:0
2013-05-20 09:26:57.692000 +08:00
Starting background process SMCO
SMCO started with pid=41, OS id=2884590
2013-05-20 09:30:51.966000 +08:00
Time drift detected. Please check VKTM trace file for more details.      -- VKTM (virtual keeper of time) is responsible for providing a wall-clock time
                                                                         -- (updated every second) and reference-time counter (updated every 20 ms and
                                                                         -- available only when running at elevated priority).
2013-05-20 09:36:21.723000 +08:00
db_recovery_file_dest_size of 5727 MB is 62.98% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
2013-05-20 09:21:51.131000 +08:00
Redo thread 2 internally disabled at seq 541 (CKPT)
[6620006] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:570407839 end:570410008 diff:2169 (21 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
2013-05-20 09:21:52.634000 +08:00
minact-scn: Inst 1 is now the master inc#:2 mmon proc-id:6226738 status:0x7
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000      -- "min active scn" introduced in 11g which is supposed to enhance delayed block cleanout operations. Note : - Disabling min active scn optimization feature will not have any impact on database.
Archived Log entry 1052 added for thread 2 sequence 540 ID 0xffffffff88d0f3f7 dest 1:
2013-05-20 09:21:53.759000 +08:00
ARC0: Archiving disabled thread 2 sequence 541
Archived Log entry 1053 added for thread 2 sequence 541 ID 0xffffffff88d0f3f7 dest 1:
Starting background process GTX0      -- 全局事务进程0，在 RAC 集群中，Oracle 数据库现在提供改进的 XA 事务处理。此进程协调 XA 事务。如果 XA 上的数据库负载增长，将自动创建更多进程，分别命名为 GTX1、GTX2，直至 GTXJ。
GTX0 started with pid=89, OS id=9764872
2013-05-20 09:21:54.814000 +08:00
Starting background process RCBG      -- 高速缓存后台进程，支持新的结果高速缓存特性。
RCBG started with pid=90, OS id=8258348
replication_dependency_tracking turned off (no async multimaster replication found)      -- REPLICATION_DEPENDENCY_TRACKING enables or disables dependency tracking for read/write operations to the database. Dependency tracking is essential for propagating changes in a replicated environment in parallel.
                                                                                         -- Note:Do not specify this value unless you are sure that your application will not perform any read/write operations to the replicated tables.
2013-05-20 09:21:56.014000 +08:00
Starting background process QMNC      -- 队列监视器进程
QMNC started with pid=92, OS id=10289208
2013-05-20 09:21:59.295000 +08:00
Completed: ALTER DATABASE OPEN /* db agent *//* {2:26630:2} */
2013-05-20 09:22:03.686000 +08:00
db_recovery_file_dest_size of 5727 MB is 62.98% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Starting background process CJQ0      -- 作业队列进程
CJQ0 started with pid=100, OS id=11534496
2013-05-20 09:22:19.267000 +08:00
Reconfiguration started (old inc 2, new inc 4)
List of instances:
 1 2 (myinst: 1)
 Global Resource Directory frozen
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:4 errcnt:0
2013-05-20 09:26:57.692000 +08:00
Starting background process SMCO      -- 该进程负责空间管理协调管理工作，负责执行空间的分配和回收。
SMCO started with pid=41, OS id=2884590
2013-05-20 09:30:51.966000 +08:00
Time drift detected. Please check VKTM trace file for more details.
2013-05-20 09:36:21.723000 +08:00
db_recovery_file_dest_size of 5727 MB is 62.98% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.48.13.130)(PORT=1521))))' SCOPE=
  [name='en5', type=1, ip=169.254.25.4, mac=e4-1f-13-4f-b8-da, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interc
Dumping diagnostic data in directory=[cdmp_20130520085448], requested by (instance=1, osid=6619738 (LMHB)), summary=[abnorma
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
2013-05-20 08:52:15.183000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80074):
ORA-32701: Possible hangs up to hang ID=136 detected [email protected]
2013-05-20 08:52:30.247000 +08:00
DIA0 terminating blocker (ospid: 14484116 sid: 644 ser#: 859) of hang with ID = 136
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
   by terminating the process
DIA0 successfully terminated process ospid:14484116.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=136.
2013-05-20 08:52:34.807000 +08:00
Sweep [inc][80074]: completed
Sweep [inc][80073]: completed
2013-05-20 08:52:37.671000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80075):
ORA-32701: Possible hangs up to hang ID=137 detected
2013-05-20 08:52:52.777000 +08:00
DIA0 terminating blocker (ospid: 11862904 sid: 23 ser#: 355) of hang with ID = 137
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
   by terminating the process
DIA0 successfully terminated process ospid:11862904.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=137.
2013-05-20 08:52:55.847000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80076):
ORA-32701: Possible hangs up to hang ID=139 detected
2013-05-20 08:53:10.928000 +08:00
DIA0 terminating blocker (ospid: 18743428 sid: 903 ser#: 7) of hang with ID = 139
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
   by terminating the process
DIA0 successfully terminated process ospid:18743428.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=139.
2013-05-20 08:54:22.281000 +08:00
LMON (ospid: 6357366) waits for event 'Disk file operations I/O' for 81 secs.
2013-05-20 08:54:37.385000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_lmhb_6619738.trc  (incident=72153):
ORA-29770: global enqueue process LMON (OSID 6357366) is hung for more than 70 seconds
Incident details in: /u01/app/oracle/diag/rdbms/tlodb/tlodb1/incident/incdir_72153/tlodb1_lmhb_6619738_i72153.trc
2013-05-20 08:54:48.544000 +08:00
ERROR: Some process(s) is not making progress.
LMHB (ospid: 6619738) is terminating the instance.      -- http://www.askmaclean.com/archives/11gr2-lmhb-lock-manager-heart-beat.html
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
Please check LMHB trace file for more details.
Incident details in: /u01/app/oracle/diag/rdbms/tlodb/tlodb1/incident/incdir_72153/tlodb1_lmhb_6619738_i72153.trc
2013-05-20 08:54:48.544000 +08:00
2013-05-20 08:54:48.544000 +08:00
ERROR: Some process(s) is not making progress.
LMHB (ospid: 6619738) is terminating the instance.
LMHB (ospid: 6619738) is terminating the instance.
Please check LMHB trace file for more details.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
LMHB (ospid: 6619738): terminating the instance due to error 29770
Please also check the CPU load, I/O load and other system properties for anomalous behavior
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
Please also check the CPU load, I/O load and other system properties for anomalous behavior
ERROR: Some process(s) is not making progress.
ERROR: Some process(s) is not making progress.
LMHB (ospid: 6619738): terminating the instance due to error 29770
System state dump requested by (instance=1, osid=6619738 (LMHB)), summary=[abnormal instance termination].
Dumping diagnostic data in directory=[cdmp_20130520085448], requested by (instance=1, osid=6619738 (LMHB)), summary=[abnorma
System state dump requested by (instance=1, osid=6619738 (LMHB)), summary=[abnormal instance termination].
Dumping diagnostic data in directory=[cdmp_20130520085448], requested by (instance=1, osid=6619738 (LMHB)), summary=[abnorma
l instance termination].
ORA-1092 : opitsk aborting process
Dumping diagnostic data in directory=[cdmp_20130520085448], requested by (instance=1, osid=6619738 (LMHB)), summary=[abnorma
l instance termination].
ORA-1092 : opitsk aborting process
2013-05-20 08:55:02.927000 +08:00
Termination issued to instance processes. Waiting for the processes to exit
2013-05-20 08:55:08.929000 +08:00
Instance termination failed to kill one or more processes
Instance terminated by LMHB, pid = 6619738
2013-05-20 09:20:55.106000 +08:00
Adjusting the default value of parameter parallel_max_servers
from 3200 to 985 due to the value of parameter processes (1000)
Starting ORACLE instance (normal)
2013-05-20 09:20:57.134000 +08:00
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
2013-05-20 09:20:57.134000 +08:00
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
2013-05-20 09:20:58.602000 +08:00
Private Interface 'en5' configured from GPnP for use as a private interconnect.
  [name='en5', type=1, ip=169.254.25.4, mac=e4-1f-13-4f-b8-da, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'en4' configured from GPnP for use as a public interface.
  [name='en4', type=1, ip=10.48.13.131, mac=e4-1f-13-4f-b8-d8, net=10.48.13.128/27, mask=255.255.255.224, use=public/1]
Picked latch-free SCN scheme 3
WARNING: db_recovery_file_dest is same as db_create_file_dest
Autotune of undo retention is turned on.
2013-05-20 09:20:59.769000 +08:00
LICENSE_MAX_USERS = 0
SYS auditing is disabled
2013-05-20 09:21:01.270000 +08:00
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2
System name:AIX
Node name:htcmdb1
Release:1
Version:6
Machine:00F65CCD4C00
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2/dbs/inittlodb1.ora
System parameters with non-default values:
  processes                = 1000
  sessions                 = 1600
  spfile                   = "+TLDATA/tlodb/spfiletlodb.ora"
  memory_target            = 18304M
  control_files            = "+TLDATA/tlodb/controlfile/current.256.798813565"
  control_files            = "+TLDATA/tlodb/controlfile/current.257.798813565"
  db_block_size            = 16384
  compatible               = "11.2.0.0.0"
Dumping diagnostic data in directory=[cdmp_20130520085448], requested by (instance=1, osid=6619738 (LMHB)), summary=[abnorma
Sweep [inc][80061]: completed
2013-05-20 07:55:47.600000 +08:00
DIA0 terminating blocker (ospid: 13435258 sid: 65 ser#: 277) of hang with ID = 88
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:13435258.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=88.
2013-05-20 07:56:02.672000 +08:00
DIA0 terminating blocker (ospid: 8323632 sid: 1441 ser#: 1) of hang with ID = 89
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although hangs of this root type are typically
    self-resolving, the previously ignored hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:8323632.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=89.
2013-05-20 07:56:17.731000 +08:00
DIA0 terminating blocker (ospid: 14090364 sid: 703 ser#: 3) of hang with ID = 91
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:14090364.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=91.
2013-05-20 07:56:20.801000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80062):
ORA-32701: Possible hangs up to hang ID=99 detected
2013-05-20 07:56:36.080000 +08:00
DIA0 terminating blocker (ospid: 11731582 sid: 724 ser#: 3) of hang with ID = 94
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although hangs of this root type are typically
    self-resolving, the previously ignored hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:11731582.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=94.
2013-05-20 07:56:38.252000 +08:00
Sweep [inc][80062]: completed
2013-05-20 07:56:51.149000 +08:00
DIA0 terminating blocker (ospid: 13566860 sid: 1504 ser#: 317) of hang with ID = 95
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although hangs of this root type are typically
    self-resolving, the previously ignored hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:13566860.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=95.
2013-05-20 07:56:54.213000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80063):
ORA-32701: Possible hangs up to hang ID=99 detected
2013-05-20 07:57:09.300000 +08:00
DIA0 terminating blocker (ospid: 8127290 sid: 582 ser#: 1) of hang with ID = 97
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:8127290.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=97.
2013-05-20 07:57:24.359000 +08:00
DIA0 terminating blocker (ospid: 12714600 sid: 921 ser#: 445) of hang with ID = 98
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although hangs of this root type are typically
    self-resolving, the previously ignored hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:12714600.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=98.
2013-05-20 07:57:27.450000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80064):
ORA-32701: Possible hangs up to hang ID=99 detected
2013-05-20 07:57:42.537000 +08:00
DIA0 terminating blocker (ospid: 12452844 sid: 1044 ser#: 12845) of hang with ID = 99
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:12452844.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=99.
2013-05-20 08:02:22.532000 +08:00
Sweep [inc][80064]: completed
Sweep [inc][80063]: completed
2013-05-20 08:16:58.068000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80065):
ORA-32701: Possible hangs up to hang ID=122 detected
2013-05-20 08:17:02.744000 +08:00
Sweep [inc][80065]: completed
2013-05-20 08:17:13.145000 +08:00
DIA0 terminating blocker (ospid: 18743428 sid: 903 ser#: 7) of hang with ID = 122
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
    by terminating session sid: 903 ospid: 18743428
2013-05-20 08:17:17.434000 +08:00
DIA0 successfully terminated session sid:903 ospid:18743428 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=122.
2013-05-20 08:18:14.658000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80066):
ORA-32701: Possible hangs up to hang ID=122 detected
2013-05-20 08:18:29.724000 +08:00
DIA0 terminating blocker (ospid: 8323632 sid: 1441 ser#: 1) of hang with ID = 119
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
    by terminating session sid: 1441 ospid: 8323632
2013-05-20 08:18:34.669000 +08:00
DIA0 successfully terminated session sid:1441 ospid:8323632 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=119.
2013-05-20 08:18:44.929000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80067):
ORA-32701: Possible hangs up to hang ID=122 detected
2013-05-20 08:19:00.005000 +08:00
DIA0 terminating blocker (ospid: 9634296 sid: 1022 ser#: 1) of hang with ID = 120
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although hangs of this root type are typically
    self-resolving, the previously ignored hang was automatically resolved.
    by terminating session sid: 1022 ospid: 9634296
2013-05-20 08:19:04.669000 +08:00
DIA0 successfully terminated session sid:1022 ospid:9634296 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=120.
2013-05-20 08:19:07.736000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80068):
ORA-32701: Possible hangs up to hang ID=122 detected
2013-05-20 08:19:22.802000 +08:00
DIA0 terminating blocker (ospid: 18743428 sid: 903 ser#: 7) of hang with ID = 122
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
   by terminating the process
DIA0 successfully terminated process ospid:18743428.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=122.
2013-05-20 08:20:09.176000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80069):
ORA-32701: Possible hangs up to hang ID=122 detected
2013-05-20 08:20:24.239000 +08:00
DIA0 terminating blocker (ospid: 8323632 sid: 1441 ser#: 1) of hang with ID = 119
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although the number of affected sessions did not
    justify automatic hang resolution initially, this previously ignored
    hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:8323632.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=119.
2013-05-20 08:20:39.549000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80070):
ORA-32701: Possible hangs up to hang ID=122 detected
2013-05-20 08:20:54.619000 +08:00
DIA0 terminating blocker (ospid: 9634296 sid: 1022 ser#: 1) of hang with ID = 120
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Although hangs of this root type are typically
    self-resolving, the previously ignored hang was automatically resolved.
   by terminating the process
DIA0 successfully terminated process ospid:9634296.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=120.
2013-05-20 08:22:29.126000 +08:00
Sweep [inc][80070]: completed
Sweep [inc][80069]: completed
Sweep [inc][80068]: completed
Sweep [inc][80067]: completed
Sweep [inc][80066]: completed
2013-05-20 08:28:20.280000 +08:00
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=220
System State dumped to trace file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_j004_10093488.trc
2013-05-20 08:28:27.184000 +08:00
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=219
2013-05-20 08:39:28.521000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80071):
ORA-32701: Possible hangs up to hang ID=128 detected
2013-05-20 08:39:43.596000 +08:00
DIA0 terminating blocker (ospid: 14484116 sid: 644 ser#: 859) of hang with ID = 123
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
    by terminating session sid: 644 ospid: 14484116
2013-05-20 08:39:47.762000 +08:00
DIA0 successfully terminated session sid:644 ospid:14484116 with status 31.
System State dumped to trace file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_j004_10093488.trc
2013-05-20 08:28:27.184000 +08:00
>>> WAITED TOO LONG FOR A ROW CACHE ENQUEUE LOCK! pid=219
2013-05-20 08:39:28.521000 +08:00
Errors in file /u01/app/oracle/diag/rdbms/tlodb/tlodb1/trace/tlodb1_dia0_6423436.trc  (incident=80071):
ORA-32701: Possible hangs up to hang ID=128 detected
2013-05-20 08:39:43.596000 +08:00
DIA0 terminating blocker (ospid: 14484116 sid: 644 ser#: 859) of hang with ID = 123
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
    by terminating session sid: 644 ospid: 14484116
2013-05-20 08:39:47.762000 +08:00
DIA0 successfully terminated session sid:644 ospid:14484116 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=123.
2013-05-20 08:40:02.810000 +08:00
DIA0 terminating blocker (ospid: 11862904 sid: 23 ser#: 355) of hang with ID = 124
    requested by master DIA0 process on instance 1
    Hang Resolution Reason: Automatic hang resolution was performed to free a
    significant number of affected sessions.
    by terminating session sid: 23 ospid: 11862904
2013-05-20 08:40:07.960000 +08:00
DIA0 successfully terminated session sid:23 ospid:11862904 with status 31.
DIA0 successfully resolved a LOCAL, HIGH confidence hang with ID=124.
2013-05-20 08:40:23.034000 +08:00