首次遭遇ORA-15064
整理一下前段时间遇到的ORA-15064错误。
有一套测试环境是11gR2的rac使用ASM，应用在测试时发现一个节点宕机了，检查了一下alert日志如下错误：
Mon Jun 04 23:27:32 2012
NOTE: ASMB terminatingErrors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
Errors in file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
ASMB (ospid: 13828564): terminating the instance due to error 15064
Termination issued to instance processes. Waiting for the processes to exit
Mon Jun 04 23:27:43 2012
Instance termination failed to kill one or more processes
Instance terminated by ASMB, pid = 13828564
通过错误可知，改错无与ASM有关，应该是ASM意外关闭才导致的Instance关闭。
错误的trace文件内容如下：
[pt1rac2@root]# cat /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trc
Trace file /soft/oracle/diag/rdbms/pt1db/pt1db2/trace/pt1db2_asmb_13828564.trcOracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /soft/oracle/product/11.2.0.2/dbh
System name: AIX
Node name: pt1rac2
Release: 1
Version: 6
Machine: 00F770E14C00
Instance name: pt1db2
Redo thread mounted by this instance: 0 <none>
Oracle process number: 27
Unix process pid: 13828564, image: oracle@pt1rac2 (ASMB)
*** 2012-06-04 16:19:14.483
*** SESSION ID:(2998.1) 2012-06-04 16:19:14.483
*** CLIENT ID:() 2012-06-04 16:19:14.483
*** SERVICE NAME:() 2012-06-04 16:19:14.483
*** MODULE NAME:() 2012-06-04 16:19:14.483
*** ACTION NAME:() 2012-06-04 16:19:14.483
NOTE: initiating MARK startup
*** 2012-06-04 16:19:18.290
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7
*** 2012-06-04 23:07:12.286
Mode of disk 0 (DATA_0000) changing from 0x7 to 0x7
*** 2012-06-04 23:27:32.492
NOTE: ASMB terminating
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
error 15064 detected in background process
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID:
Session ID: 23 Serial number: 1161
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+644<-kjzdssdmp()+444<-kjzduptcctx()+272<-kjzdicrshnfy()+96<-ksuitm()+1284<-ksbrdp()+4508<-opirip()+1216<-opidrv()+1156<-sou2o()+192<-opimai_real()+308<-ssthrdmain()+340<-main()+216<-__start()+112
----- End of Abridged Call Stack Trace -----
*** 2012-06-04 23:27:32.854
ASMB (ospid: 13828564): terminating the instance due to error 15064
*** 2012-06-04 23:27:43.387
Instance termination failed to kill one or more processes
可知的确是由于ASM意外关闭才导致的instance关闭，ASM意外关闭的原因继续查。
准备生产一下awr报告看一下，结果发现由于数据关闭了，没有awr可看。
一下数据说明数据库被重启过几次，且经过与应用人员确认，这些重启的时间点他们都在做压力测试，因此初步怀疑是和性能有关，继续查。
Snap
Instance DB Name Snap Id Snap Started Level
------------ ------------ --------- ------------------ -----
pt1db2 PT1DB 503 02 Jun 2012 16:00 1
504 02 Jun 2012 17:00 1
505 04 Jun 2012 09:43 1
506 04 Jun 2012 11:00 1
507 04 Jun 2012 12:00 1
508 04 Jun 2012 13:00 1
509 04 Jun 2012 14:00 1
510 04 Jun 2012 15:00 1
511 04 Jun 2012 16:00 1
512 04 Jun 2012 17:00 1
513 04 Jun 2012 18:00 1
514 04 Jun 2012 19:00 1
515 04 Jun 2012 20:00 1
516 04 Jun 2012 21:00 1
517 04 Jun 2012 22:00 1
518 04 Jun 2012 23:00 1
522 05 Jun 2012 01:20 1
523 05 Jun 2012 01:30 1
524 05 Jun 2012 01:45 1
先查看ocssd.log日志
2012-06-04 23:26:03.126: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.126: [ CSSD][1029]clssgmAllocProc: (11328bcd0) allocated
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11328bcd0 - 0,1,2,3,4
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea97f) proc(11328bcd0) pid(15335558) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.127: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.274: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:03.274: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:03.309: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:03.309: [ CSSD][1029]clssgmAllocProc: (1132d30d0) allocated
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132d30d0 - 0,1,2,3,4
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2ea9fa) proc(1132d30d0) pid(9306434/9306434) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:03.310: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:03.315: [ CSSD][1029]clssscSelect: cookie accept request 1132d30d0
2012-06-04 23:26:03.315: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132d30d0
2012-06-04 23:26:03.316: [ CSSD][1029]clssgmRegisterClient: proc(57/1132d30d0), client(1/11313ef70)
2012-06-04 23:26:03.316: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(112f32d30) con(2ea049) skgpid ospid 35520704 with 0 clients, refcount 0
2012-06-04 23:26:29.658: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea049
2012-06-04 23:26:29.675: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:29.675: [ CSSD][1029]clssgmAllocProc: (113242130) allocated
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113242130 - 0,1,2,3,4
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eadef) proc(113242130) pid(10879472) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:29.676: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.226: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.226: [ CSSD][1029]clssgmAllocProc: (1132a3390) allocated
2012-06-04 23:26:30.227: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132a3390 - 0,1,2,3,4
2012-06-04 23:26:30.227: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eae64) proc(1132a3390) pid(35520706) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.228: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.542: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.542: [ CSSD][1029]clssgmAllocProc: (1132ad030) allocated
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132ad030 - 0,1,2,3,4
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaede) proc(1132ad030) pid(8585490/8585490) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.544: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.553: [ CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.553: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.553: [ CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(1/1132ad370)
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmQueueShare: (1132ad510) target global grock DG_DATA member 1 type 1 queued from client (1132ad370), global grock DG_DATA, refcount 15
2012-06-04 23:26:30.554: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 15
2012-06-04 23:26:30.557: [ CSSD][1029]clssscSelect: cookie accept request 1132ad030
2012-06-04 23:26:30.557: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132ad030
2012-06-04 23:26:30.557: [ CSSD][1029]clssgmRegisterClient: proc(63/1132ad030), client(2/1132a3d90)
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmQueueShare: (1132ad910) target global grock DB+ASM member 1 type 1 queued from client (1132a3d90), global grock DB+ASM, refcount 17
2012-06-04 23:26:30.558: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 17
2012-06-04 23:26:30.636: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:26:30.637: [ CSSD][1029]clssgmAllocProc: (11333ba70) allocated
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11333ba70 - 0,1,2,3,4
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eaf6f) proc(11333ba70) pid(8913188/8913188) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:26:30.638: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:26:30.650: [ CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.650: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.650: [ CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(1/112e8e550)
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmQueueShare: (11328ed90) target global grock DG_DATA member 1 type 1 queued from client (112e8e550), global grock DG_DATA, refcount 16
2012-06-04 23:26:30.652: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 16
2012-06-04 23:26:30.766: [ CSSD][1029]clssscSelect: cookie accept request 11333ba70
2012-06-04 23:26:30.766: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11333ba70
2012-06-04 23:26:30.766: [ CSSD][1029]clssgmRegisterClient: proc(64/11333ba70), client(2/11328eff0)
2012-06-04 23:26:30.772: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:26:30.772: [ CSSD][1029]clssgmQueueShare: (11328f190) target global grock DB+ASM member 1 type 1 queued from client (11328eff0), global grock DB+ASM, refcount 18
2012-06-04 23:26:30.773: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 18
2012-06-04 23:26:31.382: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:31.382: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:34.394: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 145718 msecs
2012-06-04 23:26:35.392: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:35.392: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:36.396: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 47678 msecs
2012-06-04 23:26:38.571: [ CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 75% maximum time, 200000 ms, will be considered unusable in 49104 ms
2012-06-04 23:26:39.408: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:39.409: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:42.413: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 153738 msecs
2012-06-04 23:26:43.413: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:43.413: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:44.414: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 55696 msecs
2012-06-04 23:26:47.421: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:47.421: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:50.433: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 161758 msecs
2012-06-04 23:26:51.431: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:51.431: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:52.439: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 63721 msecs
2012-06-04 23:26:55.442: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:55.442: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:26:58.445: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 169771 msecs
2012-06-04 23:26:59.455: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:26:59.455: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:00.493: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 71776 msecs
2012-06-04 23:27:03.470: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:03.470: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:06.473: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 177799 msecs
2012-06-04 23:27:07.475: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:07.476: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDeadProc: proc 112f35f50
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(112f35f50) con(2ea340) skgpid ospid 10551654 with 0 clients, refcount 0
2012-06-04 23:27:08.192: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea340
2012-06-04 23:27:08.192: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 79475 msecs
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDeadProc: proc 1131b4b10
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1131b4b10) con(2ea3c4) skgpid ospid 10355074 with 0 clients, refcount 0
2012-06-04 23:27:08.498: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea3c4
2012-06-04 23:27:08.569: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.569: [ CSSD][1029]clssgmAllocProc: (1131fe670) allocated
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1131fe670 - 0,1,2,3,4
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb1f8) proc(1131fe670) pid(10551656) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.571: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.637: [ CSSD][2834]clssnmvDiskCheck: (/dev/rhdisk1) No I/O completed after 90% maximum time, 200000 ms, will be considered unusable in 19038 ms
2012-06-04 23:27:08.809: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.810: [ CSSD][1029]clssgmAllocProc: (1132593f0) allocated
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132593f0 - 0,1,2,3,4
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb278) proc(1132593f0) pid(10355076) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.811: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.855: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:08.855: [ CSSD][1029]clssgmAllocProc: (113259730) allocated
2012-06-04 23:27:08.857: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 113259730 - 0,1,2,3,4
2012-06-04 23:27:08.858: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb2ec) proc(113259730) pid(8192508/8192508) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:08.858: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:08.867: [ CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.867: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.867: [ CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(1/112f35f50)
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmQueueShare: (11334b430) target global grock DG_DATA member 1 type 1 queued from client (112f35f50), global grock DG_DATA, refcount 17
2012-06-04 23:27:08.868: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 17
2012-06-04 23:27:08.872: [ CSSD][1029]clssscSelect: cookie accept request 113259730
2012-06-04 23:27:08.872: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113259730
2012-06-04 23:27:08.872: [ CSSD][1029]clssgmRegisterClient: proc(65/113259730), client(2/113215ed0)
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmQueueShare: (11334b690) target global grock DB+ASM member 1 type 1 queued from client (113215ed0), global grock DB+ASM, refcount 19
2012-06-04 23:27:08.874: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 19
2012-06-04 23:27:09.000: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:09.000: [ CSSD][1029]clssgmAllocProc: (11334b8f0) allocated
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 11334b8f0 - 0,1,2,3,4
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb37d) proc(11334b8f0) pid(8061184/8061184) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:09.001: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:09.010: [ CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.010: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.010: [ CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(1/1131b5310)
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmQueueShare: (11334bc30) target global grock DG_DATA member 1 type 1 queued from client (1131b5310), global grock DG_DATA, refcount 18
2012-06-04 23:27:09.011: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 18
2012-06-04 23:27:09.014: [ CSSD][1029]clssscSelect: cookie accept request 11334b8f0
2012-06-04 23:27:09.014: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 11334b8f0
2012-06-04 23:27:09.014: [ CSSD][1029]clssgmRegisterClient: proc(66/11334b8f0), client(2/11334be90)
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmQueueShare: (113350230) target global grock DB+ASM member 1 type 1 queued from client (11334be90), global grock DB+ASM, refcount 20
2012-06-04 23:27:09.015: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 20
2012-06-04 23:27:11.480: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:11.480: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:14.494: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 185820 msecs
2012-06-04 23:27:15.500: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:15.500: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:16.504: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 87787 msecs
2012-06-04 23:27:19.519: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:19.519: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:20.285: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.285: [ CSSD][1029]clssgmAllocProc: (1132b9290) allocated
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b9290 - 0,1,2,3,4
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb49d) proc(1132b9290) pid(8257826) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:20.294: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:20.299: [ CSSD][1029]clssnmGetNodeNumber: pt1rac2
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDeadProc: proc 1132b9290
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b9290) con(2eb49d) skgpid ospid 8257826 with 0 clients, refcount 0
2012-06-04 23:27:20.457: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb49d
2012-06-04 23:27:20.999: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:20.999: [ CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 0,1,2,3,4
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb51a) proc(1132b2650) pid(7733742) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:21.000: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:21.005: [ CSSD][1029]clssnmGetNodeNumber: pt1rac1
2012-06-04 23:27:21.289: [ CSSD][1029]clssgmDeadProc: proc 1132b2650
2012-06-04 23:27:21.290: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1132b2650) con(2eb51a) skgpid ospid 7733742 with 0 clients, refcount 0
2012-06-04 23:27:21.290: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2eb51a
2012-06-04 23:27:22.001: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 193327 msecs
2012-06-04 23:27:23.533: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:23.534: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDeadProc: proc 113248030
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(113248030) con(2ea626) skgpid ospid 9699686 with 0 clients, refcount 0
2012-06-04 23:27:24.945: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2ea626
2012-06-04 23:27:25.142: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.143: [ CSSD][1029]clssgmAllocProc: (1132b2650) allocated
2012-06-04 23:27:25.143: [ CSSD][1029]clssscMonitorThreads clssnmvWorkerThread not scheduled for 96425 msecs
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b2650 - 1,2,3,4
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb5c3) proc(1132b2650) pid(9699688) version 11:2:1:4, properties: 1,2,3,4
2012-06-04 23:27:25.144: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.323: [ CSSD][1029]clssscSelect: cookie accept request 110abb440
2012-06-04 23:27:25.324: [ CSSD][1029]clssgmAllocProc: (1132b7e50) allocated
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1132b7e50 - 0,1,2,3,4
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(2eb63a) proc(1132b7e50) pid(7733746/7733746) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:27:25.326: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:27:25.485: [ CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.485: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.485: [ CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(1/113153d90)
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmRegisterShared: grp DG_DATA, mbr 1, type 1
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmQueueShare: (113338450) target global grock DG_DATA member 1 type 1 queued from client (113153d90), global grock DG_DATA, refcount 19
2012-06-04 23:27:25.487: [ CSSD][1029]clssgmRegisterShared: global grock DG_DATA member 1 share type 1, refcount 19
2012-06-04 23:27:25.495: [ CSSD][1029]clssscSelect: cookie accept request 1132b7e50
2012-06-04 23:27:25.495: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 1132b7e50
2012-06-04 23:27:25.495: [ CSSD][1029]clssgmRegisterClient: proc(67/1132b7e50), client(2/1132b42b0)
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmRegisterShared: grp DB+ASM, mbr 1, type 1
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmQueueShare: (11334c0f0) target global grock DB+ASM member 1 type 1 queued from client (1132b42b0), global grock DB+ASM, refcount 21
2012-06-04 23:27:25.497: [ CSSD][1029]clssgmRegisterShared: global grock DB+ASM member 1 share type 1, refcount 21
2012-06-04 23:27:27.542: [ CSSD][3862]clssnmSendingThread: sending status msg to all nodes
2012-06-04 23:27:27.542: [ CSSD][3862]clssnmSendingThread: sent 4 status msgs to all nodes
2012-06-04 23:27:27.680: [ CSSD][2834](:CSSNM00058:)clssnmvDiskCheck: No I/O completions for 200006 ms for voting file /dev/rhdisk1)
2012-06-04 23:27:27.693: [ CSSD][2834]clssnmvDiskAvailabilityChange: voting file /dev/rhdisk1 now offline
2012-06-04 23:27:27.693: [ CSSD][2834](:CSSNM00018:)clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
2012-06-04 23:27:27.694: [ CSSD][2834]###################################
2012-06-04 23:27:27.694: [ CSSD][2834] clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.694: [ CSSD][2834]###################################
2012-06-04 23:27:27.694: [ CSSD][2834](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2012-06-04 23:27:27.695: [ CSSD][2834]
----- Call Stack Trace -----
2012-06-04 23:27:27.723: [ CSSD][2834]calling call entry argument values in hex
2012-06-04 23:27:27.723: [ CSSD][2834]location type point (? means dubious value)
2012-06-04 23:27:27.723: [ CSSD][2834]-------------------- -------- -------------------- ----------------------------
2012-06-04 23:27:27.724: [ CSSD][2834]clssscExit()+692 bl 100067114 67800000678 ? 1000EB40C ?
2012-06-04 23:27:27.724: [ CSSD][2834] 110091190 ? 900000000BA4D14 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 000000000 ? 013960031 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [ CSSD][2834]clssnmvDiskCheck()+ call clssscExit() 111E38B90 ? 200000002 ?
2012-06-04 23:27:27.724: [ CSSD][2834]2576 000000001 ? 1000F52A0 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 110091190 ? 013960031 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.724: [ CSSD][2834]clssnmvDiskPingMoni call clssnmvDiskCheck() 111E38B90 ? 110BF7FB0 ?
2012-06-04 23:27:27.724: [ CSSD][2834]torThread()+392 112468680 ? 000000000 ?
2012-06-04 23:27:27.724: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 013960031 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]clssscthrdmain()+20 call clssnmvDiskPingMoni 111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]8 torThread() 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]_pthread_body()+240 call clssscthrdmain() 111E38B90 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834]fffffffffffffffc call _pthread_body() 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.725: [ CSSD][2834] 000000000 ? 000000000 ?
2012-06-04 23:27:27.726: [ CSSD][2834]
2012-06-04 23:27:27.726: [ CSSD][2834]--------------------- Binary Stack Dump ---------------------
2012-06-04 23:27:27.726: [ CSSD][2834]
2012-06-04 23:27:27.726: [ CSSD][2834]========== FRAME [1] (clssscExit()+692 -> 100067114) ==========
2012-06-04 23:27:27.726: [ CSSD][2834]defined by frame pointers 0x1124683f0 and 0x112468260
2012-06-04 23:27:27.726: [ CSSD][2834]CALL TYPE: bl ERROR SIGNALED: no CALLER: clssscExit
2012-06-04 23:27:27.726: [ CSSD][2834]
Dump of memory from 0x112468260 to 0x1124683f0
2012-06-04 23:27:27.726: [ CSSD][2834]112468260 00000001 124683F0 44000224 000003D7 [.....F..D..$....]
2012-06-04 23:27:27.726: [ CSSD][2834]112468270 00000001 000031D8 00000000 00000083 [......1.........]
2012-06-04 23:27:27.726: [ CSSD][2834]112468280 00000000 00000029 00000001 100182B0 [.......)........]
2012-06-04 23:27:27.726: [ CSSD][2834]112468290 00000678 00000678 00000001 000EB40C [...x...x........]
2012-06-04 23:27:27.726: [ CSSD][2834]1124682A0 00000001 10091190 09000000 00BA4D14 [..............M.]
2012-06-04 23:27:27.726: [ CSSD][2834]1124682B0 00000000 00000000 00000000 13960031 [...............1]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682C0 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682D0 00000001 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682E0 00000000 00000001 43525300 12468368 [........CRS..F.h]
2012-06-04 23:27:27.727: [ CSSD][2834]1124682F0 00000001 00000000 00000001 100181F0 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468300 00000001 10018208 00000001 100181F0 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468310 00000001 10018208 00000001 11E38B90 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468320 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.727: [ CSSD][2834]112468330 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.728: [ CSSD][2834] Repeat 3 times
2012-06-04 23:27:27.728: [ CSSD][2834]112468370 00000000 00000000 00000000 13960031 [...............1]
2012-06-04 23:27:27.728: [ CSSD][2834]112468380 09000000 00C27160 00000000 00000000 [......q`........]
2012-06-04 23:27:27.728: [ CSSD][2834]112468390 00000001 1002BD70 00000001 117B6590 [.......p.....{e.]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683A0 00000001 00000001 00000000 00000000 [................]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683B0 00000000 00000000 00000646 00000646 [...........F...F]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683C0 00000001 12468428 00000001 117C6C84 [.....F.(.....|l.]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683D0 00000001 117C6C84 00000001 1008FF50 [.....|l........P]
2012-06-04 23:27:27.728: [ CSSD][2834]1124683E0 00000000 00000000 00000001 000F24E8 [..............$.]
2012-06-04 23:27:27.728: [ CSSD][2834]========== FRAME [2] (clssnmvDiskCheck()+2576 -> clssscExit()) ==========
2012-06-04 23:27:27.728: [ CSSD][2834]defined by frame pointers 0x112468600 and 0x1124683f0
2012-06-04 23:27:27.729: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssnmvDiskCheck
2012-06-04 23:27:27.729: [ CSSD][2834]
Dump of memory from 0x1124683f0 to 0x112468600
2012-06-04 23:27:27.729: [ CSSD][2834]1124683F0 00000001 12468600 09001000 A0688640 [.....F.......h.@]
2012-06-04 23:27:27.729: [ CSSD][2834]112468400 00000001 0002CD94 4400B22B 00000000 [........D..+....]
2012-06-04 23:27:27.729: [ CSSD][2834]112468410 00000001 001058B8 00000001 100182B0 [......X.........]
2012-06-04 23:27:27.729: [ CSSD][2834]112468420 00000001 11E38B90 00000002 00000002 [................]
2012-06-04 23:27:27.729: [ CSSD][2834]112468430 00000000 00000001 00000001 000F52A0 [..............R.]
2012-06-04 23:27:27.729: [ CSSD][2834]112468440 00000001 10091190 00000000 13960031 [...............1]
2012-06-04 23:27:27.729: [ CSSD][2834]112468450 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.730: [ CSSD][2834]112468460 00000001 1008FF50 00000001 118508F0 [.......P........]
2012-06-04 23:27:27.730: [ CSSD][2834]112468470 00000001 117B6590 8C91440C 00030D46 [.....{e...D....F]
2012-06-04 23:27:27.730: [ CSSD][2834]112468480 00000001 12C508B0 00000000 00030D40 [...............@]
2012-06-04 23:27:27.730: [ CSSD][2834]112468490 00010001 9DD81012 D0E54F97 BFEE0F2C [..........O....,]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684A0 AF8AA88C 00000000 2200088B 00000000 [........".......]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684B0 09000000 014F0814 09001000 A0688640 [.....O.......h.@]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684C0 00030D40 1045F530 00000004 00000004 [[email protected]........]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684D0 00000001 00105DE8 00000001 000F2BD4 [......].......+.]
2012-06-04 23:27:27.730: [ CSSD][2834]1124684E0 00000001 000F2BD4 00030D40 000F24E8 [......+....@..$.]
2012-06-04 23:27:27.731: [ CSSD][2834]1124684F0 00000001 00000001 2200022B 00000000 [........"..+....]
2012-06-04 23:27:27.731: [ CSSD][2834]112468500 00000000 00000000 FFFFFFF5 000F24E8 [..............$.]
2012-06-04 23:27:27.731: [ CSSD][2834]112468510 00000001 12468600 09001000 A0688640 [.....F.......h.@]
2012-06-04 23:27:27.731: [ CSSD][2834]112468520 00000001 00070DFC 00000001 117C6C84 [.............|l.]
2012-06-04 23:27:27.731: [ CSSD][2834]112468530 00000001 117C6C84 00000001 100182B0 [.....|l.........]
2012-06-04 23:27:27.731: [ CSSD][2834]112468540 00000001 1045F530 00000004 00000004 [.....E.0........]
2012-06-04 23:27:27.731: [ CSSD][2834]112468550 00000001 00105DE8 00000000 00000009 [......].........]
2012-06-04 23:27:27.731: [ CSSD][2834]112468560 00000000 000003E8 00000000 000003E8 [................]
2012-06-04 23:27:27.731: [ CSSD][2834]112468570 00000000 13960031 00000000 00000040 [.......1.......@]
2012-06-04 23:27:27.732: [ CSSD][2834]112468580 00000001 1008FF50 00000001 117B6590 [.......P.....{e.]
2012-06-04 23:27:27.732: [ CSSD][2834]112468590 00000001 1002BD70 00000001 117B65A0 [.......p.....{e.]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685A0 00000001 117B66A0 FFFFFFB2 000003D4 [.....{f.........]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685B0 00000001 117C6C84 00000001 117C6C84 [.....|l......|l.]
2012-06-04 23:27:27.732: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.732: [ CSSD][2834]1124685D0 00000001 000F2BD4 00000001 000F2BD4 [......+.......+.]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685E0 00000000 00000000 00000001 12C508B0 [................]
2012-06-04 23:27:27.732: [ CSSD][2834]1124685F0 00000000 00000000 00000001 000F24E8 [..............$.]
2012-06-04 23:27:27.732: [ CSSD][2834]========== FRAME [3] (clssnmvDiskPingMonitorThread()+392 -> clssnmvDiskCheck()) ==========
2012-06-04 23:27:27.732: [ CSSD][2834]defined by frame pointers 0x1124686f0 and 0x112468600
2012-06-04 23:27:27.732: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssnmvDiskPingMonitorThread
2012-06-04 23:27:27.732: [ CSSD][2834]
Dump of memory from 0x112468600 to 0x1124686f0
2012-06-04 23:27:27.733: [ CSSD][2834]112468600 00000001 124686F0 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468610 00000001 0002CFAC 09001000 A0688640 [.............h.@]
2012-06-04 23:27:27.733: [ CSSD][2834]112468620 00000000 00000000 00000001 100182B0 [................]
2012-06-04 23:27:27.733: [ CSSD][2834]112468630 00000001 11E38B90 00000001 10BF7FB0 [................]
2012-06-04 23:27:27.733: [ CSSD][2834]112468640 00000001 12468680 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468650 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.733: [ CSSD][2834]112468660 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590 [.............{e.]
2012-06-04 23:27:27.733: [ CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0 [...@..@$........]
2012-06-04 23:27:27.733: [ CSSD][2834]112468690 09000000 014F0814 00000000 00000000 [.....O..........]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686A0 00000000 00000000 00000000 00000001 [................]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8 [................]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.734: [ CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78 [.............F.x]
2012-06-04 23:27:27.734: [ CSSD][2834]========== FRAME [4] (clssscthrdmain()+208 -> clssnmvDiskPingMonitorThread()) ==========
2012-06-04 23:27:27.734: [ CSSD][2834]defined by frame pointers 0x112468790 and 0x1124686f0
2012-06-04 23:27:27.734: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: clssscthrdmain
2012-06-04 23:27:27.734: [ CSSD][2834]
Dump of memory from 0x1124686f0 to 0x112468790
2012-06-04 23:27:27.734: [ CSSD][2834]1124686F0 00000001 12468790 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.734: [ CSSD][2834]112468700 00000001 0000A294 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468710 00000000 00000000 00000001 100182B0 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468720 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468730 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.735: [ CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.735: [ CSSD][2834]112468770 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.735: [ CSSD][2834]112468780 00000000 00000000 00000001 1008FF50 [...............P]
2012-06-04 23:27:27.735: [ CSSD][2834]========== FRAME [5] (_pthread_body()+240 -> clssscthrdmain()) ==========
2012-06-04 23:27:27.735: [ CSSD][2834]defined by frame pointers 0x112468810 and 0x112468790
2012-06-04 23:27:27.735: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: _pthread_body
2012-06-04 23:27:27.735: [ CSSD][2834]
Dump of memory from 0x112468790 to 0x112468890
2012-06-04 23:27:27.736: [ CSSD][2834]112468790 00000001 12468810 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687A0 09000000 00B7BD54 00000000 00000000 [.......T........]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687B0 00000000 00000000 09001000 A0288350 [.............(.P]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687C0 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.736: [ CSSD][2834]1124687D0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.737: [ CSSD][2834] Repeat 11 times
2012-06-04 23:27:27.737: [ CSSD][2834]========== FRAME [6] (fffffffffffffffc -> _pthread_body()) ==========
2012-06-04 23:27:27.737: [ CSSD][2834]defined by frame pointers 0x0 and 0x112468810
2012-06-04 23:27:27.737: [ CSSD][2834]CALL TYPE: call ERROR SIGNALED: no CALLER: fffffffffffffffc
2012-06-04 23:27:27.737: [ CSSD][2834]
----- Argument/Register Address Dump -----
2012-06-04 23:27:27.737: [ CSSD][2834]Argument/Register addr=0x1000eb40c.
Dump of memory from 0x1000eb3cc to 0x1000eb50c
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3C0 6C206572 [l er]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3D0 726F7220 6F636375 72726564 20616E64 [ror occurred and]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3E0 20746865 20435353 20646165 6D6F6E20 [ the CSS daemon ]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB3F0 69732074 65726D69 6E617469 6E672061 [is terminating a]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB400 626E6F72 6D616C6C 79004942 283A4353 [bnormally.IB(:CS]
2012-06-04 23:27:27.737: [ CSSD][2834]1000EB410 53534330 30303132 3A290049 636C7373 [SSC00012:).Iclss]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB420 73634578 69743A20 53746172 74696E67 [scExit: Starting]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB430 20435253 4420636C 65616E75 70004942 [ CRSD cleanup.IB]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB440 636C7373 73634578 69743A20 43525344 [clssscExit: CRSD]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB450 20636C65 616E7570 20737461 74757320 [ cleanup status ]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB460 25640049 636C7373 73634578 69743A20 [%d.IclssscExit: ]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB470 43525344 20636C65 616E7570 20666169 [CRSD cleanup fai]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB480 6C656420 77697468 20256400 636C7373 [led with %d.clss]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB490 73634578 69743A20 43525344 20636C65 [scExit: CRSD cle]
2012-06-04 23:27:27.738: [ CSSD][2834]1000EB4A0 616E7570 20737563 63657373 66756C6C [anup successfull]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4B0 7920636F 6D706C65 74656400 636C7373 [y completed.clss]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4C0 73634578 69743A20 61626F72 7420616C [scExit: abort al]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4D0 72656164 79207365 74202564 0049424D [ready set %d.IBM]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4E0 41535345 52542025 73202564 0049424D [ASSERT %s %d.IBM]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB4F0 636C7373 73632E63 0049424D 283A4353 [clsssc.c.IBM(:CS]
2012-06-04 23:27:27.739: [ CSSD][2834]1000EB500 53534330 30303133 3A29636C [SSC00013:)cl]
2012-06-04 23:27:27.739: [ CSSD][2834]Argument/Register addr=0x110091190.
Dump of memory from 0x110091150 to 0x110091290
2012-06-04 23:27:27.739: [ CSSD][2834]110091150 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.739: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.739: [ CSSD][2834]110091180 00000000 00000000 00000001 00000001 [................]
2012-06-04 23:27:27.740: [ CSSD][2834]110091190 2F736F66 742F6772 69642F70 726F6475 [/soft/grid/produ]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911A0 63742F31 312E322E 302E322F 6769682F [ct/11.2.0.2/gih/]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911B0 6C6F672F 70743172 6163322F 63737364 [log/pt1rac2/cssd]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911C0 2F6F6373 73642E6C 6F670000 00000000 [/ocssd.log......]
2012-06-04 23:27:27.740: [ CSSD][2834]1100911D0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ CSSD][2834] Repeat 11 times
2012-06-04 23:27:27.741: [ CSSD][2834]Argument/Register addr=0x900000000ba4d14.
Dump of memory from 0x900000000ba4cd4 to 0x900000000ba4e14
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CD0 38600000 4BFD5608 00000000 [8`..K.V.....]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CE0 000C0000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4CF0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D00 348D8278 41810008 38800000 E8630000 [4..xA...8....c..]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D10 38A00008 81840230 7C0004AC 398CFFFF [8......0|...9...]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D20 E9040110 38E00000 65081000 7D251828 [....8...e...}%.(]
2012-06-04 23:27:27.741: [ CSSD][2834]900000000BA4D30 7C894000 40860014 7CE5192D 40A2FFF0 [|.@.@...|..-@...]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D40 91840230 4E800020 7C632A14 4BFD4694 [...0N.. |c*.K.F.]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D50 00000000 000C0000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D60 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D80 7D251828 64871000 2C890000 40860014 [}%.(d...,...@...]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4D90 7CE5192D 4C00012C 4D820020 4BFFFFE4 [|..-L..,M.. K...]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DA0 7C862378 7C851A14 7CC53378 4BFD5534 [|.#x|...|.3xK.U4]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DB0 00000000 000C0000 00000000 00000000 [................]
2012-06-04 23:27:27.742: [ CSSD][2834]900000000BA4DC0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.743: [ CSSD][2834] Repeat 3 times
2012-06-04 23:27:27.743: [ CSSD][2834]900000000BA4E00 7D251828 64871000 2C890000 40860014 [}%.(d...,...@...]
2012-06-04 23:27:27.743: [ CSSD][2834]900000000BA4E10 7CE5192D [|..-]
2012-06-04 23:27:27.743: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.743: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.744: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.744: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.744: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.745: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.745: [ CSSD][2834]Argument/Register addr=0x111e38b90.
Dump of memory from 0x111e38b50 to 0x111e38c90
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B50 00000001 00000001 8C8FC179 000003E8 [...........y....]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B60 8C914038 8C913A22 0000000B 00000000 [..@8..:"........]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B70 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B80 5B5B5B5B 00000000 00000000 000001E0 [[[[[............]
2012-06-04 23:27:27.745: [ CSSD][2834]111E38B90 00000001 1008FF50 00000001 10BF7FB0 [.......P........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BA0 00000001 11E3A320 00000001 11E389A0 [....... ........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BB0 00000B12 00000B12 636C7373 6E6D7644 [........clssnmvD]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BC0 69736B50 696E674D 6F6E6974 6F725468 [iskPingMonitorTh]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BD0 72656164 00000000 00000000 00000000 [read............]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BE0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38BF0 00000000 00000000 00020000 00000000 [................]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C00 00000001 1045F530 00000001 11E38B90 [.....E.0........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C10 00000001 100179B0 00000000 00000000 [......y.........]
2012-06-04 23:27:27.746: [ CSSD][2834]111E38C20 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.747: [ CSSD][2834] Repeat 4 times
2012-06-04 23:27:27.747: [ CSSD][2834]111E38C70 00000000 00000001 00000000 00000005 [................]
2012-06-04 23:27:27.747: [ CSSD][2834]111E38C80 00000000 00001677 00000000 00000000 [.......w........]
2012-06-04 23:27:27.747: [ CSSD][2834]Argument/Register addr=0x1.
Dump of memory from 0x0 to 0x101
2012-06-04 23:27:27.747: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.748: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.748: [ CSSD][2834]000000100 7C7243A6 [|rC.]
2012-06-04 23:27:27.748: [ CSSD][2834]Argument/Register addr=0x1000f52a0.
Dump of memory from 0x1000f5260 to 0x1000f53a0
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5260 6B436865 636B3A20 41626F72 74696E67 [kCheck: Aborting]
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5270 2C202564 206F6620 25642025 7320766F [, %d of %d %s vo]
2012-06-04 23:27:27.748: [ CSSD][2834]1000F5280 74696E67 20646973 6B732061 7661696C [ting disks avail]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5290 61626C65 2C206E65 65642025 75004942 [able, need %u.IB]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52A0 283A4353 534E4D30 30303138 3A290049 [(:CSSNM00018:).I]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52B0 636C7373 6E6D7644 69736B50 696E674D [clssnmvDiskPingM]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52C0 6F6E6974 6F725468 72656164 3A20206E [onitorThread: n]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52D0 6F20766F 74696E67 20646973 6B2C0049 [o voting disk,.I]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52E0 636C7373 6E6D7644 69736B50 4D543A20 [clssnmvDiskPMT: ]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F52F0 736C6565 70696E67 20666F72 20256420 [sleeping for %d ]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5300 6D730049 636C7373 6E6D7653 63616E43 [ms.IclssnmvScanC]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5310 6F6D706C 6574696F 6E733A20 4661696C [ompletions: Fail]
2012-06-04 23:27:27.749: [ CSSD][2834]1000F5320 75726520 2564206E 6F746966 79696E67 [ure %d notifying]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5330 20636C69 656E7473 206F6620 776F726B [ clients of work]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5340 20636F6D 706C6574 696F6E2E 0049424D [ completion..IBM]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5350 636C7373 6E6D7653 63616E43 6F6D706C [clssnmvScanCompl]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5360 6574696F 6E733A20 636F6D70 6C657465 [etions: complete]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5370 64202564 20697465 6D730049 636C7373 [d %d items.Iclss]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5380 6E6D7652 6561644C 65617365 3A20656E [nmvReadLease: en]
2012-06-04 23:27:27.750: [ CSSD][2834]1000F5390 74657200 636C7373 6E6D7652 6561644C [ter.clssnmvReadL]
2012-06-04 23:27:27.750: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.750: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.751: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.751: [ CSSD][2834]Argument/Register addr=0x110bf7fb0.
Dump of memory from 0x110bf7f70 to 0x110bf80b0
2012-06-04 23:27:27.751: [ CSSD][2834]110BF7F70 00000001 10BD5F90 00000001 10BF7F70 [......_........p]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7F80 00000000 00000000 00000000 00022000 [.............. .]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7F90 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7FA0 5B5B5B5B 00000000 00000000 00002938 [[[[[..........)8]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7FB0 4E4D6378 00000000 00000001 1008FF50 [NMcx...........P]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7FC0 00000001 00000100 00000000 00000000 [................]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7FD0 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.752: [ CSSD][2834] Repeat 1 times
2012-06-04 23:27:27.752: [ CSSD][2834]110BF7FF0 00000000 00000000 00000000 00000001 [................]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF8000 00000000 00000005 00000000 00000A14 [................]
2012-06-04 23:27:27.752: [ CSSD][2834]110BF8010 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.753: [ CSSD][2834] Repeat 9 times
2012-06-04 23:27:27.753: [ CSSD][2834]Argument/Register addr=0x112468680.
Dump of memory from 0x112468640 to 0x112468780
2012-06-04 23:27:27.753: [ CSSD][2834]112468640 00000001 12468680 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.753: [ CSSD][2834]112468650 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.753: [ CSSD][2834]112468660 00000000 13960031 00000000 00000000 [.......1........]
2012-06-04 23:27:27.754: [ CSSD][2834]112468670 00000001 10BF7FB0 00000001 117B6590 [.............{e.]
2012-06-04 23:27:27.754: [ CSSD][2834]112468680 00030D40 8C914024 000003E8 100182B0 [...@..@$........]
2012-06-04 23:27:27.754: [ CSSD][2834]112468690 09000000 014F0814 00000000 00000000 [.....O..........]
2012-06-04 23:27:27.754: [ CSSD][2834]1124686A0 00000000 00000000 00000000 00000001 [................]
2012-06-04 23:27:27.754: [ CSSD][2834]1124686B0 000003E8 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.754: [ CSSD][2834]1124686C0 00000000 00000000 000003E8 000003E8 [................]
2012-06-04 23:27:27.754: [ CSSD][2834]1124686D0 000003E8 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.754: [ CSSD][2834]1124686E0 00000001 10BF97D0 00000001 12468A78 [.............F.x]
2012-06-04 23:27:27.754: [ CSSD][2834]1124686F0 00000001 12468790 00000000 00000000 [.....F..........]
2012-06-04 23:27:27.754: [ CSSD][2834]112468700 00000001 0000A294 00000000 00000000 [................]
2012-06-04 23:27:27.755: [ CSSD][2834]112468710 00000000 00000000 00000001 100182B0 [................]
2012-06-04 23:27:27.755: [ CSSD][2834]112468720 00000001 11E38B90 00000000 00000000 [................]
2012-06-04 23:27:27.755: [ CSSD][2834]112468730 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.755: [ CSSD][2834] Repeat 2 times
2012-06-04 23:27:27.755: [ CSSD][2834]112468760 00000001 1008FF50 00000001 1002BD70 [.......P.......p]
2012-06-04 23:27:27.755: [ CSSD][2834]112468770 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.755: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.755: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.756: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.756: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.756: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.757: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.757: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.757: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.758: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.758: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.759: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.759: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.759: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.760: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.760: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.761: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.761: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.762: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.762: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.762: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.763: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.763: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.763: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.764: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.764: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.764: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.765: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.765: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.765: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.766: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.766: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.766: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.767: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.767: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.767: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.768: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.768: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.768: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.769: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.769: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.769: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.770: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.770: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.770: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.771: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.771: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.771: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.772: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.772: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.772: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.773: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.773: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.773: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.774: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.774: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.775: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.776: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.776: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.776: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.777: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.777: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.777: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.778: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.778: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.778: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.779: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.779: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.779: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.780: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.780: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.780: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.781: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.781: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.781: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.782: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.782: [ CSSD][2834]Argument/Register addr=0x0.
Dump of memory from 0x0 to 0x100
2012-06-04 23:27:27.782: [ CSSD][2834]000000000 00000000 00000000 00000000 00000000 [................]
2012-06-04 23:27:27.783: [ CSSD][2834] Repeat 15 times
2012-06-04 23:27:27.783: [ CSSD][2834]
call stack performance statistics:
2012-06-04 23:27:27.783: [ CSSD][2834]total : 0.088000 sec
2012-06-04 23:27:27.783: [ CSSD][2834]setup : 0.028000 sec
2012-06-04 23:27:27.783: [ CSSD][2834]stack unwind : 0.000000 sec
2012-06-04 23:27:27.783: [ CSSD][2834]symbol translation : 0.000000 sec
2012-06-04 23:27:27.783: [ CSSD][2834]printing the call stack: 0.002000 sec
2012-06-04 23:27:27.784: [ CSSD][2834]printing frame data : 0.012000 sec
2012-06-04 23:27:27.784: [ CSSD][2834]printing argument data : 0.046000 sec
2012-06-04 23:27:27.784: [ CSSD][2834]
----- End of Call Stack Trace -----
2012-06-04 23:27:27.784: [ CSSD][2834]clssnmSendMeltdownStatus: node pt1rac2, number 2, has experienced a failure in thread number 9 and is shutting down
2012-06-04 23:27:27.784: [ CSSD][2834]clssscExit: Starting CRSD cleanup
2012-06-04 23:27:28.284: [ SKGFD][2320]Lib :UFS:: closing handle 112115390 for disk :/dev/rhdisk1:
2012-06-04 23:27:30.363: [ CSSD][1029]clssscMonitorThreads clssnmvDiskPingThread not scheduled for 201689 msecs
2012-06-04 23:27:31.194: [ CSSD][2834]clssscExit: CRSD cleanup status 0
2012-06-04 23:27:31.194: [ CSSD][2834]clssscExit: CRSD cleanup successfully completed
2012-06-04 23:27:31.198: [ default][2834]kgzf_gen_node_reid2: generated reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2012-06-04 23:27:31.198: [ default][2834]kgzf_send_fence4: queued fence for reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100, handle 2
2012-06-04 23:27:31.198: [ CSSD][2834]clssnmrFenceSage: Fenced node pt1rac2, number 2, with EXADATA, handle 2
2012-06-04 23:27:31.198: [ CSSD][2834]clssgmUpdateEventValue: CmInfo State val 0, changes 60
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: Aborting client (11188d7d0) proc (112c52170), iocapables 1.
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c52170), pid (56951010), iocapables 1, client (11188d7d0)
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e154f0) proc (112fd5c10), iocapables 2.
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fd5c10), pid (52691020), iocapables 2, client (111e154f0)
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: Aborting client (11189b310) proc (112c524b0), iocapables 3.
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c524b0), pid (50200582), iocapables 3, client (11189b310)
2012-06-04 23:27:31.200: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e16130) proc (112f90870), iocapables 4.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 4, client (111e16130)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (111863110) proc (112f90870), iocapables 5.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 5, client (111863110)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (111862ad0) proc (112f90870), iocapables 6.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 6, client (111862ad0)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (111869530) proc (112f90870), iocapables 7.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f90870), pid (52363338), iocapables 7, client (111869530)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e1ddf0) proc (112c65490), iocapables 8.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 8, client (111e1ddf0)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (11186a570) proc (112c65490), iocapables 9.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 9, client (11186a570)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (1118644b0) proc (112fa6d10), iocapables 10.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 10, client (1118644b0)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (111ef2570) proc (112fa6d10), iocapables 11.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 11, client (111ef2570)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (11186d210) proc (112fa6d10), iocapables 12.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 12, client (11186d210)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 13.
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 13, client (112f97d30)
2012-06-04 23:27:31.201: [ CSSD][1029]clssgmSendShutdown: Aborting client (1118a1c90) proc (113039a10), iocapables 14.
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 14, client (1118a1c90)
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e694d0) proc (11305be70), iocapables 15.
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (11305be70), pid (46268552), iocapables 15, client (111e694d0)
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: Aborting client (112c4f470) proc (112ea2b90), iocapables 16.
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 16, client (112c4f470)
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: Aborting client (112c4cef0) proc (112ea2b90), iocapables 17.
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 17, client (112c4cef0)
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: Aborting client (111867490) proc (112c77ff0), iocapables 18.
2012-06-04 23:27:31.204: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c77ff0), pid (15925342), iocapables 18, client (111867490)
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e204f0) proc (112ea1150), iocapables 19.
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea1150), pid (60031284), iocapables 19, client (111e204f0)
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e4eb70) proc (112ebec30), iocapables 20.
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ebec30), pid (34472120), iocapables 20, client (111e4eb70)
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e62d30) proc (112edbcb0), iocapables 21.
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112edbcb0), pid (41812046), iocapables 21, client (111e62d30)
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e56010) proc (112efd370), iocapables 22.
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 22, client (111e56010)
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 23.
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 23, client (111e5fd90)
2012-06-04 23:27:31.205: [ CSSD][1029]clssgmSendShutdown: Aborting client (112ed4510) proc (112f05c70), iocapables 24.
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 24, client (112ed4510)
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: Aborting client (112c485b0) proc (112f05c70), iocapables 25.
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 25, client (112c485b0)
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: Aborting client (112eb7310) proc (112f05c70), iocapables 26.
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 26, client (112eb7310)
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: Aborting client (112e95a70) proc (112f05c70), iocapables 27.
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 27, client (112e95a70)
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: Aborting client (112c68030) proc (112f05c70), iocapables 28.
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112f05c70), pid (33554436), iocapables 28, client (112c68030)
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: Aborting client (112c71a90) proc (112ef8f10), iocapables 29.
2012-06-04 23:27:31.206: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ef8f10), pid (40370574), iocapables 29, client (112c71a90)
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmDeadProc: proc 112c524b02012-06-04 23:27:31.273: [ CSSD][1029]clssgmFenceClient: fencing client (11189b310), member 0 in group GR+GCR1, no share, death fence 1, SAGE fence 0
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmUnreferenceMember: global grock GR+GCR1 member 0 refcount is 1
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmFenceProcessDeath: client (11189b310) pid 50200582 undead
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmQueueFenceForCheck: (1117e0630) Death check for object type 3, pid 50200582
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 16f1
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(112c524b0) con(16c2) skgpid 50200582 ospid 50200582 with 0 clients, refcount 0
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 16c2
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c52170), pid (56951010), iocapables 1, client (11188d7d0)
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 2, client (111e1ddf0)
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c65490), pid (47841464), iocapables 3, client (11186a570)
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: Aborting client (1118644b0) proc (112fa6d10), iocapables 4.
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 4, client (1118644b0)
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: Aborting client (111ef2570) proc (112fa6d10), iocapables 5.
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 5, client (111ef2570)
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: Aborting client (11186d210) proc (112fa6d10), iocapables 6.
2012-06-04 23:27:31.273: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112fa6d10), pid (49479912), iocapables 6, client (11186d210)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 7.
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 7, client (112f97d30)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 8, client (1118a1c90)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 9, client (112c4f470)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea2b90), pid (59179420), iocapables 10, client (112c4cef0)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112c77ff0), pid (15925342), iocapables 11, client (111867490)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ea1150), pid (60031284), iocapables 12, client (111e204f0)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ebec30), pid (34472120), iocapables 13, client (111e4eb70)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e62d30) proc (112edbcb0), iocapables 14.
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112edbcb0), pid (41812046), iocapables 14, client (111e62d30)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 15, client (111e56010)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 16.
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 16, client (111e5fd90)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: Aborting client (112c71a90) proc (112ef8f10), iocapables 17.
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112ef8f10), pid (40370574), iocapables 17, client (112c71a90)
2012-06-04 23:27:31.274: [ CSSD][1029]clssgmClientShutdown: total iocapables 17
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:27:31.508: [ CSSD][3091]clssgmAllocateRPCIndex: allocated rpc 772 (111259170)2012-06-04 23:27:31.508: [ CSSD][3091]clssgmRPC: RPC(#2) to node(772) not sent due to impending local GM shutdown
2012-06-04 23:27:31.508: [ CSSD][3091]clssgmRPC: failed to send RPC#772 to node(2), rpcret(10), master(2), DBInfo(1), masterRPC(1),unsentRPC(1), queuing RPC to unsent queue
2012-06-04 23:27:31.508: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.610: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.611: [ CSSD][3091]clssnmrCheckEDFenceStatus: Node pt1rac2, number 2, EXADATA fence handle 2 has status 1, status request return code ORA-0
2012-06-04 23:27:31.694: [ default][4890]kgzf_recv_main4: got response for request 111efc710 (handle 2)
2012-06-04 23:27:31.694: [ default][4890]kgzf_decode_msg2: ENTITY_FENCED for req handle 2, error = 0
2012-06-04 23:27:35.036: [ CSSD][1029]clssgmClientShutdown: total iocapables 4
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 1.
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 1, client (112f97d30)
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 2, client (1118a1c90)
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 3, client (111e56010)
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 4.
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 4, client (111e5fd90)
2012-06-04 23:27:35.037: [ CSSD][1029]clssgmClientShutdown: total iocapables 4
2012-06-04 23:27:35.038: [ CSSD][1029]clssnmCompleteGMReq: Completed request type 18 with status 1
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmDoneQEle: re-queueing req 1115ded30 status 1
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmCheckReqNMCompletion: Completing request type 18 for proc (1132b2650), operation status 1, client status 0
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 1.
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 1, client (112f97d30)
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 2, client (1118a1c90)
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 3, client (111e56010)
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 4.
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 4, client (111e5fd90)
2012-06-04 23:27:35.038: [ CSSD][1029]clssgmClientShutdown: total iocapables 4
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmFenceClient: fencing client (113057150), member 0 in group DG_LOCAL_DATA, same group share, death fence 1, SAGE fence 0
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmFenceProcessDeath: client (113057150) pid 62390694 undead
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmQueueFenceForCheck: (1115ded30) Death check for object type 2, pid 62390694
2012-06-04 23:27:35.039: [ CSSD][3091]clssgmFenceCompletion: (1115ded30) process death fence completed for process 62390694, object type 2
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 2b3c7e
2012-06-04 23:27:35.039: [ CSSD][3091]clssgmTermShare: (113247350) local grock DG_LOCAL_DATA member 0 type 1
2012-06-04 23:27:35.039: [ CSSD][3091]clssgmUnreferenceMember: local grock DG_LOCAL_DATA member 0 refcount is 8
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmSendShutdown: Aborting client (112f97d30) proc (113039a10), iocapables 1.
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 1, client (112f97d30)
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (113039a10), pid (48169154), iocapables 2, client (1118a1c90)
2012-06-04 23:27:35.039: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 3, client (111e56010)
2012-06-04 23:27:35.040: [ CSSD][1029]clssgmSendShutdown: Aborting client (111e5fd90) proc (112efd370), iocapables 4.
2012-06-04 23:27:35.040: [ CSSD][1029]clssgmSendShutdown: I/O capable proc (112efd370), pid (59834676), iocapables 4, client (111e5fd90)
。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:28:01.619: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1118a2610) client(0)2012-06-04 23:28:01.622: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1115db470) client(0)
2012-06-04 23:28:01.634: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1115db7b0) client(0)
2012-06-04 23:28:01.634: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1115dbaf0) client(0)
2012-06-04 23:28:01.651: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1118a2610) client(0)
2012-06-04 23:28:01.669: [ CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.669: [ CSSD][1029]clssgmAllocProc: (1118a6cd0) allocated
2012-06-04 23:28:01.669: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 1118a6cd0 - 0,1,2,3,4
2012-06-04 23:28:01.669: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(5ee) proc(1118a6cd0) pid(56951112/56951112) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:28:01.669: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.670: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(1118a6cd0) client(0)
2012-06-04 23:28:01.671: [ CSSD][1029]clssgmDeadProc: proc 1118a6cd0
2012-06-04 23:28:01.671: [ CSSD][1029]clssgmDestroyProc: cleaning up proc(1118a6cd0) con(5ee) skgpid 56951112 ospid 56951112 with 0 clients, refcount 0
2012-06-04 23:28:01.671: [ CSSD][1029]clssgmDiscEndpcl: gipcDestroy 5ee
2012-06-04 23:28:01.709: [ CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.709: [ CSSD][1029]clssgmAllocProc: (111e1a130) allocated
2012-06-04 23:28:01.709: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 111e1a130 - 0,1,2,3,4
2012-06-04 23:28:01.709: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(64f) proc(111e1a130) pid(50724892) version 11:2:1:4, properties: 0,1,2,3,4
2012-06-04 23:28:01.709: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.712: [ CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.713: [ CSSD][1029]clssgmAllocProc: (111e51c90) allocated
2012-06-04 23:28:01.713: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 111e51c90 - 1,2,3,4,5
2012-06-04 23:28:01.713: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(6a1) proc(111e51c90) pid(262412) version 11:2:1:4, properties: 1,2,3,4,5
2012-06-04 23:28:01.713: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.717: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e1a130) client(0)
2012-06-04 23:28:01.726: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e51c90) client(0)
2012-06-04 23:28:01.742: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e1a130) client(0)
2012-06-04 23:28:01.746: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(111e51c90) client(0)
2012-06-04 23:28:01.753: [ CSSD][1029]clssscSelect: cookie accept request 110cfec60
2012-06-04 23:28:01.753: [ CSSD][1029]clssgmAllocProc: (112044a30) allocated
2012-06-04 23:28:01.754: [ CSSD][1029]clssgmClientConnectMsg: properties of cmProc 112044a30 - 1,2,3,4,5
2012-06-04 23:28:01.754: [ CSSD][1029]clssgmClientConnectMsg: Connect from con(718) proc(112044a30) pid(196930) version 11:2:1:4, properties: 1,2,3,4,5
2012-06-04 23:28:01.754: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:28:01.762: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(112044a30) client(0)
2012-06-04 23:28:01.772: [ CSSD][1029]clssgmExecuteClientRequest(): type(5) size(96) only connect and exit messages are allowed before lease acquisition proc(112044a30) client(0)
2012-06-04 23:28:02.150: [ SKGFD][2061]NOTE: No asm libraries found in the system
2012-06-04 23:28:02.150: [ CLSF][2061]Allocated CLSF context
2012-06-04 23:28:02.150: [ SKGFD][2061]Handle 112062170 from lib :UFS:: for disk :/dev/rhdisk1:
2012-06-04 23:28:02.151: [ SKGFD][2061]Lib :UFS:: closing handle 112062170 for disk :/dev/rhdisk1:
2012-06-04 23:28:03.151: [ CSSD][1]clssnmlfmtlease: uniqueness 1338823677, gipc addr gipcha://pt1rac2:nm2_pt1rac-cluster
2012-06-04 23:28:03.177: [ CSSD][1]clssnmvStatusBlkInit: myinfo nodename pt1rac2, uniqueness 1338823677
2012-06-04 23:28:03.206: [ CSSD][1]clssnmlgetslot:lease acquisition for node pt1rac2/slot 2 completed in 5041 msecs
2012-06-04 23:28:03.339: [ CSSD][1]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 232297358, wrtcnt, 2006767, LATS 2358366040, lastSeqNo 0, uniqueness 1338794747, timestamp 1338822439/2180206706
2012-06-04 23:28:03.339: [ CSSD][1]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 232297359, wrtcnt, 1995087, LATS 2358366040, lastSeqNo 0, uniqueness 1338823677, timestamp 1338823448/2358131374
2012-06-04 23:28:03.354: [ SKGFD][1]Lib :UFS:: closing handle 111e2d670 for disk :/dev/rhdisk1:
2012-06-04 23:28:03.354: [ CSSD][1]clssnmInitNodeDB: Initializing with OCR id 0
2012-06-04 23:28:04.356: [ CSSD][2062]clssnm_skgxnmon: Compatible vendor clusterware not in use
2012-06-04 23:28:04.357: [ CSSD][1]clssnmNotifyReq: type (20)
2012-06-04 23:28:04.357: [ CSSD][1]clssnmReadNodeInfo: dynamic endp HA gipcha://pt1rac1:nm2_pt1rac-cluster len 35 ver 11.2.0.2.0
2012-06-04 23:28:04.357: [ CSSD][1]clssnmReadNodeInfo: NM endp for node 1 (pt1rac1) - gipcha://pt1rac1:nm2_pt1rac-cluster
2012-06-04 23:28:04.357: [ CSSD][1]clssnmReadNodeInfo: added node 1 (pt1rac1) to cluster, lease uniqueness 1338794747
2012-06-04 23:28:04.357: [ CSSD][1]clssnmReadNodeInfo: dynamic endp HA gipcha://pt1rac2:nm2_pt1rac-cluster len 35 ver 11.2.0.2.0
2012-06-04 23:28:04.360: [ GPNP][1] clsgpnp_Init: [at clsgpnp0.c:583 clsgpnp_Init] '/soft/grid/product/11.2.0.2/gih' in effect as GPnP home base.
2012-06-04 23:28:04.360: [ GPNP][1] clsgpnp_Init: [at clsgpnp0.c:629 clsgpnp_Init] GPnP pid=10879068, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
2012-06-04 23:28:04.389: [ GPNP][1] clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399 clsgpnpkwf_initwfloc] Using FS Wallet Location : /soft/grid/product/11.2.0.2/gih/gpnp/pt1rac2/wallets/peer/
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2012-06-04 23:28:04.473: [ GPNP][1] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2108 clsgpnp_profileCallUrlInt] get-profile call to url "ipc://GPNPD_pt1rac2" disco "" [f=3 claimed- host: cname: seq: auth:]
2012-06-04 23:28:04.490: [ GPNP][1] clsgpnp_profileCallUrlInt: [at clsgpnp.c:2236 clsgpnp_profileCallUrlInt] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_pt1rac2" disco ""
2012-06-04 23:28:04.694: [ CSSD][1]clssnmReadNodeInfo: NM endp for node 2 (pt1rac2) - gipcha://pt1rac2:nm2_pt1rac-cluster
2012-06-04 23:28:04.694: [ CSSD][1]clssnmReadNodeInfo: added node 2 (pt1rac2) to cluster, lease uniqueness 1338823677
2012-06-04 23:28:04.694: [ CSSD][1][INFO]clssnmInitNodeDB: no vendor clusterware, IPMI allowed
[ CLWAL][1]clsw_Initialize: OLR initlevel [70000]
2012-06-04 23:28:04.755: [ CSSD][1]clssnmReadWallet: OpenWallet: CLSW-9: The cluster wallet to be operated on does not exist., returned 9
2012-06-04 23:28:04.756: [ CSSD][1]clssnmSetupCookie: clssnmReadWallet fails - Node not configured for NodeKill
2012-06-04 23:28:04.756: [ CSSD][1]clssnmInitNodeDB: failure 1 reading node kill
2012-06-04 23:28:04.756: [ SKGFD][1]NOTE: No asm libraries found in the system
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。
2012-06-04 23:29:59.973: [ CSSD][1029]clssgmClientConnectMsg: msg flags 0x0000
2012-06-04 23:29:59.985: [ CSSD][1029]clssscSelect: cookie accept request 113034870
2012-06-04 23:29:59.985: [ CSSD][1029]clssscevtypSHRCON: getting client with cmproc 113034870
2012-06-04 23:29:59.986: [ CSSD][1029]clssgmRegisterClient: proc(23/113034870), client(1/1117fd310)
2012-06-04 23:29:59.986: [ CSSD][1029]clssgmJoinGrock: global grock DGPT1DB0 new client 1117fd310 with con 2e67, requested num 1, flags 0x0
2012-06-04 23:29:59.986: [ CSSD][1029]clssgmAddGrockMember: adding member to grock DGPT1DB0
2012-06-04 23:29:59.986: [ CSSD][1029]clssgmAllocateRPCIndex: allocated rpc 160 (11121cc30)
2012-06-04 23:29:59.986: [ CSSD][1029]clssgmRPC: rpc 11121cc30 (RPC#160) tag(a0002a) sent to node 2
2012-06-04 23:29:59.986: [ CSSD][3348]clssgmHandleMasterMemberAdd: [s(2) d(2)]
2012-06-04 23:29:59.986: [ CSSD][3348]clssgmAddMember: new master 1 for group(DGPT1DB0)
2012-06-04 23:29:59.986: [ CSSD][3348]clssgmAddMember: Adding fencing for member 1, group DGPT1DB0, death 1, SAGE 0
2012-06-04 23:29:59.986: [ CSSD][3348]clssgmAddMember: member (1/113044f50) added. pbsz(11) prsz(124) flags 0x0 to grock (113044b30/DGPT1DB0)
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmQueueGrockEvent: groupName(DGPT1DB0) count(1) master(1) event(1), incarn 1, mbrc 1, to member 1, events 0x0, state 0x0
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmCommonAddMember: global group grock DGPT1DB0 member(1/Local) node(2) flags 0x0 0x13044b30
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmRPCDone: rpc 11121cc30 (RPC#160) state 6, flags 0x100
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmAddGrockMemCmpl: rpc 11121cc30, ret 0, client 1117fd310 member 113044f50
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmAddGrockMemCmpl: sending type 6, size 540 to 1117fd310
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmFreeRPCIndex: freeing rpc 160
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmAllocateRPCIndex: allocated rpc 161 (11121cdc0)
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmRPCBroadcast: rpc(0xa1002a), status(0), sendcount(0), filtered by specific properties:
2012-06-04 23:29:59.987: [ CSSD][3348]clssgmFreeRPCIndex: freeing rpc 161
有以上日志可知，最开始出现了ASM的DiskPing检测超时的情况，然后crsd服务进程出现coredump，最终导致实例Shutdown。
继续检查crsd.log日志
2012-06-04 23:26:13.451: [UiServer][11566] {2:53227:2141} Sending message to PE. ctx= 116bce710
2012-06-04 23:26:13.451: [ CRSPE][11309] {2:53227:2141} Processing PE command id=2140. Description: [Stat Resource : 1174374b0]
2012-06-04 23:26:13.456: [UiServer][11566] {2:53227:2141} Done for ctx=116bce710
2012-06-04 23:27:13.615: [UiServer][11823] CS(116c26750)set Properties ( grid,116c230b0)
2012-06-04 23:27:13.640: [UiServer][11566] {2:53227:2142} Sending message to PE. ctx= 116be97d0
2012-06-04 23:27:13.642: [ CRSPE][11309] {2:53227:2142} Processing PE command id=2141. Description: [Stat Resource : 116c7ca10]
2012-06-04 23:27:13.648: [UiServer][11566] {2:53227:2142} Done for ctx=116be97d0
2012-06-04 23:27:18.354: [UiServer][11823] CS(1160f2910)set Properties ( root,1160f2150)
2012-06-04 23:27:18.372: [UiServer][11566] {2:53227:2143} Sending message to PE. ctx= 116bce330
2012-06-04 23:27:18.373: [ CRSPE][11309] {2:53227:2143} Processing PE command id=2142. Description: [Stat Resource : 116c23170]
2012-06-04 23:27:18.407: [UiServer][11566] {2:53227:2143} Done for ctx=116bce330
2012-06-04 23:27:27.841: [UiServer][11823] CS(116c28710)set Properties ( grid,116c23490)
2012-06-04 23:27:27.862: [UiServer][11566] {2:53227:2144} Container [ Name: FENCESERVER
CLIENT:
TextMessage[]
CLIENT_PRIMARY_GROUP:
TextMessage[oinstall]
LOCALE:
TextMessage[AMERICAN_AMERICA.WE8ISO8859P1]
]
2012-06-04 23:27:27.863: [UiServer][11566] {2:53227:2144} Sending message to AGFW. ctx= 116b07a30
2012-06-04 23:27:27.863: [UiServer][11566] {2:53227:2144} Force-disconnecting [0] existing PE clients...
2012-06-04 23:27:27.863: [UiServer][11566] {2:53227:2144} Sending message: 9978 to AGFW proxy server.
2012-06-04 23:27:27.910: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server received the message: UnKnown[Proxy] ID 20489:9978
2012-06-04 23:27:27.978: [ CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.DATA.dg
2012-06-04 23:27:28.035: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER.lsnr pt1rac2 1] ID 4100:9979 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.043: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:9980 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.044: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER_SCAN2.lsnr 1 1] ID 4100:9981 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.045: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.LISTENER_SCAN3.lsnr 1 1] ID 4100:9982 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.045: [ CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.asm
2012-06-04 23:27:28.048: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.cvu 1 1] ID 4100:9983 to the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.055: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.net1.network pt1rac2 1] ID 4100:9984 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.055: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:9985 to the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.057: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.ons pt1rac2 1] ID 4100:9986 to the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.076: [ CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.pt1db.db
2012-06-04 23:27:28.077: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.pt1rac1.vip 1 1] ID 4100:9987 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.077: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.pt1rac2.vip 1 1] ID 4100:9988 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.086: [ CRSPE][10024] {2:53227:2144} Skipping Fence of : ora.registry.acfs
2012-06-04 23:27:28.094: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.scan1.vip 1 1] ID 4100:9989 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.105: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.scan2.vip 1 1] ID 4100:9990 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.106: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server sending message: RESOURCE_CLEAN[ora.scan3.vip 1 1] ID 4100:9991 to the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.319: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:9985 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.320: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.oc4j 1 1] ID 4100:9985 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.343: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.oc4j]
2012-06-04 23:27:28.344: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.cvu 1 1] ID 4100:9983 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.344: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.cvu 1 1] ID 4100:9983 from the agent /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.360: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.cvu]
2012-06-04 23:27:28.369: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server received the message: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:142
2012-06-04 23:27:28.369: [ AGFW][10024] {2:53227:2144} Shutdown request received from /soft/grid/product/11.2.0.2/gih/bin/scriptagent_grid
2012-06-04 23:27:28.369: [ AGFW][10024] {2:53227:2144} Agfw Proxy Server replying to the message: AGENT_SHUTDOWN_REQUEST[Proxy] ID 20486:142
2012-06-04 23:27:28.479: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.net1.network pt1rac2 1] ID 4100:9984 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.645: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER.lsnr pt1rac2 1] ID 4100:9979 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.730: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN2.lsnr 1 1] ID 4100:9981 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.871: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.net1.network pt1rac2 1] ID 4100:9984 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.871: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.net1.network]
2012-06-04 23:27:28.879: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan2.vip 1 1] ID 4100:9990 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.880: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan2.vip 1 1] ID 4100:9990 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.890: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.scan2.vip]
2012-06-04 23:27:28.891: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac2.vip 1 1] ID 4100:9988 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.894: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac2.vip 1 1] ID 4100:9988 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.895: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.pt1rac2.vip]
2012-06-04 23:27:28.899: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac1.vip 1 1] ID 4100:9987 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.907: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.pt1rac1.vip 1 1] ID 4100:9987 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.925: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.pt1rac1.vip]
2012-06-04 23:27:28.925: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan3.vip 1 1] ID 4100:9991 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.926: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan3.vip 1 1] ID 4100:9991 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.938: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.scan3.vip]
2012-06-04 23:27:28.944: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:9980 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.946: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN3.lsnr 1 1] ID 4100:9982 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:28.973: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan1.vip 1 1] ID 4100:9989 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.975: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.scan1.vip 1 1] ID 4100:9989 from the agent /soft/grid/product/11.2.0.2/gih/bin/orarootagent_root
2012-06-04 23:27:28.976: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.scan1.vip]
2012-06-04 23:27:29.734: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.ons pt1rac2 1] ID 4100:9986 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:29.933: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.ons pt1rac2 1] ID 4100:9986 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:29.933: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.ons]
2012-06-04 23:27:30.618: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER.lsnr pt1rac2 1] ID 4100:9979 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:30.633: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER.lsnr]
2012-06-04 23:27:30.637: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN2.lsnr 1 1] ID 4100:9981 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:30.654: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER_SCAN2.lsnr]
2012-06-04 23:27:30.718: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN3.lsnr 1 1] ID 4100:9982 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:30.719: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER_SCAN3.lsnr]
2012-06-04 23:27:31.191: [ AGFW][10024] {2:53227:2144} Received the reply to the message: RESOURCE_CLEAN[ora.LISTENER_SCAN1.lsnr 1 1] ID 4100:9980 from the agent /soft/grid/product/11.2.0.2/gih/bin/oraagent_grid
2012-06-04 23:27:31.191: [ AGFW][10024] {2:53227:2144} Fenced off the resource [ora.LISTENER_SCAN1.lsnr]
2012-06-04 23:27:31.192: [ CRSPE][10024] {2:53227:2144} Fence command status: Container [ Name: UI_DATA
RESULT:
TextMessage[0]
]
2012-06-04 23:27:31.192: [ AGFW][10024] {2:53227:2144} Fence command completed!
2012-06-04 23:27:31.192: [UiServer][11566] {2:53227:2144} UI server recvd reply from Agfw Proxy Server: 2042
2012-06-04 23:27:31.192: [UiServer][11566] {2:53227:2144} Container [ Name: UI_DATA
RESULT:
TextMessage[0]
]
2012-06-04 23:27:42.055: [ default][1] CSS is not ready. Received status 3
2012-06-04 23:27:42.055: [ default][1] First attempt: init CSS context failed. Error = 3
[ clsdmt][515]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=pt1rac2DBG_CRSD))
2012-06-04 23:27:42.067: [ clsdmt][515]PID for the Process [56951100], connkey 1
2012-06-04 23:27:42.068: [ clsdmt][515]Creating PID [56951100] file for home /soft/grid/product/11.2.0.2/gih host pt1rac2 bin crs to /soft/grid/product/11.2.0.2/gih/crs/init/
2012-06-04 23:27:42.068: [ clsdmt][515]Writing PID [56951100] to the file [/soft/grid/product/11.2.0.2/gih/crs/init/pt1rac2.pid]
2012-06-04 23:27:44.350: [ default][1] CRS Daemon Starting
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: AGENT 1
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: AGFW 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CLSFRAME 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CLSVER 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CLUCLS 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSAPP 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSCCL 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSCEVT 0
2012-06-04 23:27:44.361: [ default][1] ENV Logging level for Module: CRSCOMM 0
2012-06-04 23:27:44.361: [ CRSD][1] ENV Logging level for Module: CRSD 0
2012-06-04 23:27:44.361: [ CRSD][1] ENV Logging level for Module: CRSEVT 0
2012-06-04 23:27:44.361: [ CRSD][1] ENV Logging level for Module: CRSMAIN 0
2012-06-04 23:27:44.361: [ CRSD][1] ENV Logging level for Module: CRSOCR 0
2012-06-04 23:27:44.361: [ CRSD][1] ENV Logging level for Module: CRSPE 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSPLACE 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSRES 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSRPT 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSRTI 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSSE 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSSEC 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSTIMER 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: CRSUI 0
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCRAPI 1
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCRASM 1
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCRCAC 1
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCRCLI 1
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCRMAS 1
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCRMSG 1
2012-06-04 23:27:44.362: [ CRSD][1] ENV Logging level for Module: OCROSD 1
2012-06-04 23:27:44.363: [ CRSD][1] ENV Logging level for Module: OCRRAW 1
2012-06-04 23:27:44.363: [ CRSD][1] ENV Logging level for Module: OCRSRV 1
2012-06-04 23:27:44.363: [ CRSD][1] ENV Logging level for Module: OCRUTL 1
2012-06-04 23:27:44.363: [ CRSD][1] ENV Logging level for Module: SuiteTes 1
2012-06-04 23:27:44.363: [ CRSD][1] ENV Logging level for Module: UiServer 0
2012-06-04 23:27:44.363: [ CRSMAIN][1] Checking the OCR device
2012-06-04 23:27:44.363: [ CRSMAIN][1] Sync-up with OCR
2012-06-04 23:27:44.363: [ CRSMAIN][1] Connecting to the CSS Daemon
2012-06-04 23:27:44.376: [ CRSRTI][1] CSS is not ready. Received status 3
2012-06-04 23:27:44.400: [ CRSMAIN][1] Created alert : (:CRSD00109:) : Could not init the CSS context, error: 3
2012-06-04 23:27:44.400: [ CRSD][1][PANIC] CRSD exiting: Could not init the CSS context, error: 3
2012-06-04 23:27:44.400: [ CRSD][1] Done.
相同的时间点crsd也出现了错误
下面是diskmon.log的日志
2012-06-04 23:27:03.235: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:06.236: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:09.268: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:12.239: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:15.268: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:18.272: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:21.267: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:24.292: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:27.298: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:30.307: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:31.544: [ DISKMON][8781838:1286] dskm_process_msg5: received msg type KGZM_FENCE_ENTITY (0x0007)
2012-06-04 23:27:31.662: [ DISKMON][8781838:1286] dskm_proc_fence_entity13: fenced reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100 (size 93, msgno 2, event 2)
2012-06-04 23:27:31.694: [ DISKMON][8781838:1286] dskm_send_entity_fenced1:
2012-06-04 23:27:31.694: [ DISKMON][8781838:1286] dskm_send_entity_fenced4: done
2012-06-04 23:27:31.731: [ DISKMON][8781838:1286] dskm_process_msg7: processed msg 2 type KGZM_FENCE_ENTITY (0x0007), retcode 0
2012-06-04 23:27:34.483: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:35.180: [ DISKMON][8781838:1286] dskm_slave_thrd_main3: peer disconnected
2012-06-04 23:27:35.180: [ DISKMON][8781838:1286] dskm_slave_thrd_main5: client kgzf/8126478 disconnected, reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297340,nmn=2,lnid=232297353,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2012-06-04 23:27:35.531: [ DISKMON][8781838:1] dskm_cleanup_thrds: cleaning up slave 1 thread tid 1286
2012-06-04 23:27:36.945: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:39.681: [ DISKMON][8781838:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:40.442: [ CSSCLNT]clsssRecvMsg: got a disconnect from the server while waiting for message type 1
2012-06-04 23:27:40.464: [ CSSCLNT]clssgsGroupGetStatus: communications failed (0/3/-1)
2012-06-04 23:27:40.464: [ CSSCLNT]clssgsGroupGetStatus: returning 8
2012-06-04 23:27:40.464: [ DISKMON][8781838:515] dskm_rac_thrd_main6: cssd ABORT event
2012-06-04 23:27:40.465: [ DISKMON][8781838:515] SHUTDOWN FORCE due to CSSD ABORT
2012-06-04 23:27:40.466: [ DISKMON][8781838:515] dskm_rac_thrd_main: exiting
2012-06-04 23:27:40.681: [ DISKMON][8781838:1029] dskm_slave_thrd_main5: client orarootagent/7798854 disconnected, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
2012-06-04 23:27:41.955: [ DISKMON][8781838:1] dskm_cleanup_thrds: cleaning up the rac event handling thread tid 515
2012-06-04 23:27:41.962: [ DISKMON][8781838:1] dskm_cleanup_thrds: cleaning up slave 0 thread tid 1029
2012-06-04 23:27:41.978: [ DISKMON][8781838:772] dskm_oss_thrd_main2: posted
2012-06-04 23:27:42.000: [ DISKMON][8781838:772] dskm_oss_thrd_main: exiting
2012-06-04 23:27:42.016: [ DISKMON][8781838:1] Diskmon Aborting
[ DISKMON][8781838]
Process 8781838 exiting on 2012-06-04 at 23:27:42.035
[ DISKMON][8781842]
I/O Fencing and SKGXP HA monitoring daemon -- Version 1.1.0.0
Process 8781842 started on 2012-06-04 at 23:27:44.409
2012-06-04 23:27:44.425: [ DISKMON][8781842] dskm main: starting up
2012-06-04 23:27:44.425: [ DISKMON][8781842:515] dskm_rac_thrd_main: running
2012-06-04 23:27:44.426: [ DISKMON][8781842:515] dskm_clss_ini1: calling clssscbinit
2012-06-04 23:27:44.426: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:44.426: [ DISKMON][8781842:1] dskm_rac_thrd_creat2: got the post from the css event handling thread
2012-06-04 23:27:44.427: [ DISKMON][8781842:772] dskm_oss_thrd_main: running
2012-06-04 23:27:44.427: [ DISKMON][8781842:1] dskm_oss_thrd_creat2: got the post from the oss check status thread
2012-06-04 23:27:44.437: [ DISKMON][8781842:1] dskm main: startup complete
2012-06-04 23:27:44.437: [ DISKMON][8781842:1] listening on -> /tmp/.oracle/master_diskmon
2012-06-04 23:27:44.934: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:45.314: [ DISKMON][8781842:1] dskm_slave_thrd_creat: thread created
2012-06-04 23:27:45.314: [ DISKMON][8781842:1029] dskm_slave_thrd_main1: slave 0 running
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_proc_identify8: client orarootagent/7798854, diskmon kgzm 2.1, slave 0, reid cid=DUMMY,icin=-1,nmn=-1,lnid=-1,gid=-1,gin=-1,gmn=-1,umemid=-1,opid=-1,opsn=-1,lvl=process hdr=0xfece0100
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_send_version1:
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_send_version4: done
2012-06-04 23:27:45.315: [ DISKMON][8781842:1029] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2012-06-04 23:27:45.325: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:45.437: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:45.940: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:46.443: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:46.946: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:47.449: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:47.952: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:48.375: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:48.457: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:48.959: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:49.462: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:49.965: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:50.469: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:51.113: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:51.371: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:51.629: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:52.152: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:52.661: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:53.589: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:54.096: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:54.392: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:54.609: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:55.115: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:55.618: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:56.125: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:56.633: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:57.136: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:57.381: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:27:57.640: [ DISKMON][8781842:515] dskm_clss_ini2: calling clsssinit
2012-06-04 23:27:57.648: [ DISKMON][8781842:515] dskm_clss_ini5: successful clsssinit(), clssvers 2.1
2012-06-04 23:27:57.648: [ DISKMON][8781842:515] dskm_clss_ini6: calling clssnsqlnum
2012-06-04 23:28:00.379: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:03.386: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:04.848: [ DISKMON][8781842:515] dskm_clss_ini8: calling clsssattrib
2012-06-04 23:28:04.848: [ DISKMON][8781842:515] dskm_clss_ini11: calling clssnsqname
2012-06-04 23:28:04.850: [ DISKMON][8781842:515] dskm_clss_ini13: calling clsssattrib
2012-06-04 23:28:04.850: [ DISKMON][8781842:515] dskm_clss_ini15: calling clssgsregnodegrp
2012-06-04 23:28:04.854: [ DISKMON][8781842:515] dskm_clss_ini18: calling clssgsgrpstat
2012-06-04 23:28:04.854: [ DISKMON][8781842:515] dskm_clss_ini21: calling clssgsgrpstat
2012-06-04 23:28:06.414: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:09.390: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_slave_thrd_main1: slave 1 running
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_process_msg5: received msg type KGZM_IDENTIFY (0x0001)
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_proc_identify8: client kgzf/10879068, diskmon kgzm 2.1, slave 1, reid cid=5f2a0577ab16cf3ebf835284821dedeb,icin=232297360,nmn=2,lnid=232297360,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node hdr=0xfece0100
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_send_version1:
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_send_version4: done
2012-06-04 23:28:12.362: [ DISKMON][8781842:NULL-thread] dskm_process_msg7: processed msg 0 type KGZM_IDENTIFY (0x0001), retcode 0
2012-06-04 23:28:12.362: [ DISKMON][8781842:1] dskm_slave_thrd_creat: thread created
2012-06-04 23:28:12.363: [ DISKMON][8781842:1286] dskm_process_msg5: received msg type KGZM_KGZF_HANDSHAKE (0x0010)
2012-06-04 23:28:12.363: [ DISKMON][8781842:1286] dskm_proc_kgzf_handshake3: client kgzf/10879068, kgzf version 2.0, slave 1
2012-06-04 23:28:12.365: [ DISKMON][8781842:515] dskm_clss_ini20: node pt1rac2 (2) registered in cluster
2012-06-04 23:28:12.365: [ DISKMON][8781842:515] dskm_clss_ini23: successful dskm_clss_ini
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_check_cell_config: CELL storage configuration file /etc/oracle/cell/network-config/cellinit.ora not found
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini3: calling clsssattrib
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini7: calling clsssattrib
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini9: calling clsssattrib
2012-06-04 23:28:12.366: [ DISKMON][8781842:515] dskm_rac_ini13: calling clssgsqgrp
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_rac_ini80: called clssgsqgrp:
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_dump_group_priv: vers: 0 flags: 0x0 confInc: 0 My confInc: 0
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_dump_group_priv: CSS Msg Hdr: vers: 0 type: UNKNOWN (0) chunks: NO MORE CHUNKS (0) transport: UNKNOWN (0) mSize: 0
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_dump_group_priv: Group Private Data is not of type DSKM_MSG_SS_REQ. Not proceeding with msg dump
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_nfy_kgzf1: notified thread kgzf enabled
2012-06-04 23:28:12.367: [ DISKMON][8781842:515] dskm_rac_ini30: calling clsslsmutexlock(MASTER#DISKMON#GROUP#MX)
2012-06-04 23:28:12.369: [ DISKMON][8781842:1286] dskm_proc_kgzf_handshake5: got the post from the hb thread
2012-06-04 23:28:12.369: [ CSSCLNT]clsslsNativeLock: CSS daemon has not entered ACTIVE state.Retrying.
2012-06-04 23:28:12.370: [ DISKMON][8781842:1286] dskm_proc_kgzf_handshake9: done, kgzf enabled
2012-06-04 23:28:12.370: [ DISKMON][8781842:1286] dskm_process_msg7: processed msg 0 type KGZM_KGZF_HANDSHAKE (0x0010), retcode 0
2012-06-04 23:28:12.429: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:13.373: [ DISKMON][8781842:515] dskm_rac_ini32: calling clssgsreg
2012-06-04 23:28:13.375: [ DISKMON][8781842:515] dskm_rac_ini34: CELL storage not configured in the cluster; registered in group MASTER#DISKMON#GROUP as memno 2 (GSDGRPSZ 512)
2012-06-04 23:28:13.375: [ DISKMON][8781842:515] dskm_rac_ini63: calling clsslsunlock (proc retcode 0)
2012-06-04 23:28:15.443: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:18.440: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:21.431: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
2012-06-04 23:28:24.435: [ DISKMON][8781842:1029] dskm_process_msg5: received msg type KGZM_PING (0x0011)
由于cssd服务出现了coredump，所以diskmon服务会把asm实例强制shutdown。
综合以上分析，可知故障是由于开发人员做压力测试导致磁盘IO过载，因此Grid在做VOTE磁盘检测时出现了超时，导致cssd服务coredump退出，进而引起diskmon服务关闭asm实例，最终数据库实例也被关闭。