恢复votedisk时遇到CRS-4602，CRS-1638
在做CRS的恢复测试，恢复votedisk时遇到CRS-4602，CRS-1638的错误,具体的操作详见：
http://blog.csdn.net/tanqingru/article/details/9493413
[root@oeltan1 ~]# crsctl replace votedisk +CRSVOTE
CRS-4602: Failed 27 to add voting file a91b94a7cd0d4f5ebfea355111e6145c.
CRS-4602: Failed 27 to add voting file 80b174e04bf24fc4bfea9e4f3c6bae4c.
CRS-4602: Failed 27 to add voting file 49c174b18d7c4ff6bfdf9dc7ece9d1e8.
Failed to replace voting disk group with +CRSVOTE.
CRS-4000: Command Replace failed, or completed with errors.
[root@oeltan1 ~]# crsctl query css votedisk
Located 0 voting disk(s).
alertoeltan1.log
上述问题是由于asm_diskstring参数引起：
File created.
SQL> shutdown immediate
ORA-15100: invalid or missing diskgroup name
ASM instance shutdown
SQL> startup
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> alter system set asm_diskstring='/dev/asm*' scope=both;
System altered.
SQL> startup force mount;
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /apps/grid/gridhome/11.2.0/gri
d/dbs/spfile+ASM1.ora
[root@oeltan1 ~]# crsctl replace votedisk +CRSVOTE
Successful addition of voting disk 54e90342890c4f58bf319f3c57c4acd7.
Successful addition of voting disk 5103d7e756f14f69bf3935ace34e5de7.
Successful addition of voting disk 370645ed11324fe5bf99eac9bf5efb21.
Successfully replaced voting disk group with +CRSVOTE.
CRS-4266: Voting file(s) successfully replaced
http://blog.csdn.net/tanqingru/article/details/9493413
1.在重新添加votedisk时报错如下：
[root@oeltan1 ~]# crsctl replace votedisk +CRSVOTE
CRS-4602: Failed 27 to add voting file a91b94a7cd0d4f5ebfea355111e6145c.
CRS-4602: Failed 27 to add voting file 80b174e04bf24fc4bfea9e4f3c6bae4c.
CRS-4602: Failed 27 to add voting file 49c174b18d7c4ff6bfdf9dc7ece9d1e8.
Failed to replace voting disk group with +CRSVOTE.
CRS-4000: Command Replace failed, or completed with errors.
[root@oeltan1 ~]# crsctl query css votedisk
Located 0 voting disk(s).
2.添加失败，检查alert
alertoeltan1.log
[cssd(20659)]CRS-1638:Unable to locate voting file with ID 49c174b1-8d7c4ff6-bfdf9dc7-ece9d1e8 that is being added to the list of configured voting files; details at (:CSSNM00022:) in /apps/grid/gridhome/11.2.0/grid/log/oeltan1/cssd/ocssd.log 2013-07-26 09:51:22.787 [cssd(20659)]CRS-1638:Unable to locate voting file with ID a91b94a7-cd0d4f5e-bfea3551-11e6145c that is being added to the list of configured voting files; details at (:CSSNM00027:) in /apps/grid/gridhome/11.2.0/grid/log/oeltan1/cssd/ocssd.log 2013-07-26 09:51:22.787
3.再检查OCSSD.LOG
2013-07-26 10:13:10.773: [ CSSD][3978794752]clssnmSendingThread: sending status msg to all nodes 2013-07-26 10:13:10.774: [ CSSD][3978794752]clssnmSendingThread: sent 4 status msgs to all nodes 2013-07-26 10:13:12.978: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7b46d0 2013-07-26 10:13:12.979: [ CSSD][3989223168]clssgmAllocProc: (0x7ff2e41e9e80) allocated 2013-07-26 10:13:12.981: [ CSSD][3989223168]clssgmClientConnectMsg: properties of cmProc 0x7ff2e41e9e80 - 1,2,3,4,5 2013-07-26 10:13:12.981: [ CSSD][3989223168]clssgmClientConnectMsg: Connect from con(0xaf9b) proc(0x7ff2e41e9e80) pid(23626) version 11:2:1:4, properties: 1,2,3,4,5 2013-07-26 10:13:12.981: [ CSSD][3989223168]clssgmClientConnectMsg: msg flags 0x0000 2013-07-26 10:13:13.042: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7b46d0 2013-07-26 10:13:13.043: [ CSSD][3989223168]clssgmAllocProc: (0x7ff2e4241cd0) allocated 2013-07-26 10:13:13.044: [ CSSD][3989223168]clssgmClientConnectMsg: properties of cmProc 0x7ff2e4241cd0 - 1,2,3,4,5 2013-07-26 10:13:13.044: [ CSSD][3989223168]clssgmClientConnectMsg: Connect from con(0xb015) proc(0x7ff2e4241cd0) pid(23631/23631) version 11:2:1:4, properties: 1,2,3,4,5 2013-07-26 10:13:13.044: [ CSSD][3989223168]clssgmClientConnectMsg: msg flags 0x0000 2013-07-26 10:13:13.049: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e4241cd0 2013-07-26 10:13:13.049: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e4241cd0 2013-07-26 10:13:13.049: [ CSSD][3989223168]clssgmRegisterClient: proc(19/0x7ff2e4241cd0), client(1/0x7ff2e41a2e30) 2013-07-26 10:13:13.050: [ CSSD][3989223168]clssgmRegisterShared: grp DG_CRSVOTE, mbr 1, type 1 2013-07-26 10:13:13.050: [ CSSD][3989223168]clssgmQueueShare: (0x7ff2e41d2e60) target global grock DG_CRSVOTE member 1 type 1 queued from client (0x7ff2e41a2e30), global grock DG_CRSVOTE, refcount 7 2013-07-26 10:13:13.050: [ CSSD][3989223168]clssgmRegisterShared: global grock DG_CRSVOTE member 1 share type 1, refcount 7 2013-07-26 10:13:13.052: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e4241cd0 2013-07-26 10:13:13.052: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e4241cd0 2013-07-26 10:13:13.052: [ CSSD][3989223168]clssgmRegisterClient: proc(19/0x7ff2e4241cd0), client(2/0x7ff2e41af6e0) 2013-07-26 10:13:13.054: [ CSSD][3989223168]clssgmRegisterShared: grp DB+ASM, mbr 0, type 1 2013-07-26 10:13:13.054: [ CSSD][3989223168]clssgmQueueShare: (0x7ff2e41cb5a0) target global grock DB+ASM member 0 type 1 queued from client (0x7ff2e41af6e0), global grock DB+ASM, refcount 10 2013-07-26 10:13:13.054: [ CSSD][3989223168]clssgmRegisterShared: global grock DB+ASM member 0 share type 1, refcount 10 2013-07-26 10:13:13.112: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e41e9e80 2013-07-26 10:13:13.113: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e41e9e80 2013-07-26 10:13:13.113: [ CSSD][3989223168]clssgmRegisterClient: proc(18/0x7ff2e41e9e80), client(1/0x7ff2e41e1ab0) 2013-07-26 10:13:13.113: [ CSSD][3989223168]clssgmJoinGrock: global grock #CLSS_CONFIG_LOCK new client 0x7ff2e41e1ab0 with con 0x7ff20000b079, requested num -1, flags 0x142000 2013-07-26 10:13:13.113: [ CSSD][3989223168]clssgmAddGrockMember: adding member to grock #CLSS_CONFIG_LOCK 2013-07-26 10:13:13.113: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 290 (0x7ff2eddece60) 2013-07-26 10:13:13.113: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddece60 (RPC#290) tag(122002a) sent to node 1 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmHandleMasterMemberAdd: [s(1) d(1)] 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmAddMember: new master 0 for group(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmAddMember: Adding fencing for member 0, group #CLSS_CONFIG_LOCK, death 1, SAGE 0 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmAddMember: member (0/0x7ff2e404e140) added. pbsz(0) prsz(0) flags 0x0 to grock (0x7ff2e415dbd0/#CLSS_CONFIG_LOCK) 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmCommonAddMember: global group grock #CLSS_CONFIG_LOCK member(0/Local) node(1) flags 0x0 0xed579070 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddece60 (RPC#290) state 6, flags 0x100 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmAddGrockMemCmpl: rpc 0x7ff2eddece60, ret 0, client 0x7ff2e41e1ab0 member 0x7ff2e404e140 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmAddGrockMemCmpl: sending type 6, size 540 to 0x7ff2e41e1ab0 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 290 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 291 (0x7ff2eddecf08) 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x123002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.114: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 291 2013-07-26 10:13:13.117: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 2 2013-07-26 10:13:13.117: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 292 (0x7ff2eddecfb0) 2013-07-26 10:13:13.117: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddecfb0 (RPC#292) tag(124002a) sent to node 1 2013-07-26 10:13:13.117: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:13.117: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddecfb0 (RPC#292) state 6, flags 0x100 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddecfb0, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 292 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 293 (0x7ff2edded058) 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x125002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 293 2013-07-26 10:13:13.118: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(292) from node(1). Discarding. 2013-07-26 10:13:13.120: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 3 2013-07-26 10:13:13.120: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 294 (0x7ff2edded100) 2013-07-26 10:13:13.120: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded100 (RPC#294) tag(126002a) sent to node 1 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(24) 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded100 (RPC#294) state 6, flags 0x100 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2edded100, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 294 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 295 (0x7ff2edded1a8) 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x127002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.121: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 295 2013-07-26 10:13:13.122: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(294) from node(1). Discarding. 2013-07-26 10:13:13.123: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 2 2013-07-26 10:13:13.123: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 296 (0x7ff2edded250) 2013-07-26 10:13:13.123: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded250 (RPC#296) tag(128002a) sent to node 1 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded250 (RPC#296) state 6, flags 0x100 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2edded250, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 296 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 297 (0x7ff2edded2f8) 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x129002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.124: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 297 2013-07-26 10:13:13.125: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(296) from node(1). Discarding. 2013-07-26 10:13:13.127: [ CSSD][3989223168]clssgmExecuteClientRequest: VOTEDISKQUERY recvd from proc 18 (0x7ff2e41e9e80) 2013-07-26 10:13:13.173: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7b46d0 2013-07-26 10:13:13.173: [ CSSD][3989223168]clssgmAllocProc: (0x7ff2e420a690) allocated 2013-07-26 10:13:13.173: [ CSSD][3989223168]clssgmClientConnectMsg: properties of cmProc 0x7ff2e420a690 - 1,2,3,4,5 2013-07-26 10:13:13.173: [ CSSD][3989223168]clssgmClientConnectMsg: Connect from con(0xb10a) proc(0x7ff2e420a690) pid(23633/23633) version 11:2:1:4, properties: 1,2,3,4,5 2013-07-26 10:13:13.173: [ CSSD][3989223168]clssgmClientConnectMsg: msg flags 0x0000 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e420a690 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e420a690 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssgmRegisterClient: proc(20/0x7ff2e420a690), client(1/0x7ff2e4170900) 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssgmJoinGrock: global grock gpnp-cluster-lock new client 0x7ff2e4170900 with con 0x7ff20000b139, requested num -1, flags 0x12 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssgmAddGrockMember: adding member to grock gpnp-cluster-lock 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 298 (0x7ff2edded3a0) 2013-07-26 10:13:13.222: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded3a0 (RPC#298) tag(12a002a) sent to node 1 2013-07-26 10:13:13.222: [ CSSD][3981948672]clssgmHandleMasterMemberAdd: [s(1) d(1)] 2013-07-26 10:13:13.222: [ CSSD][3981948672]clssgmAddMember: granted member(0) flags(0x12) node(1) grock (0x7ff2e423cf90/gpnp-cluster-lock) 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmQueueGrockEvent: lockName(gpnp-cluster-lock) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0) 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmCommonAddMember: global lock grock gpnp-cluster-lock member(0/Local) node(1) flags 0x12 0xed579070 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded3a0 (RPC#298) state 6, flags 0x100 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmAddGrockMemCmpl: rpc 0x7ff2edded3a0, ret 0, client 0x7ff2e4170900 member 0x7ff2e4245f50 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmAddGrockMemCmpl: sending type 6, size 540 to 0x7ff2e4170900 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 298 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 299 (0x7ff2edded448) 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x12b002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.223: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 299 2013-07-26 10:13:13.389: [ CSSD][3989223168]clssgmExitGrock: client 1 (0x7ff2e4170900), grock gpnp-cluster-lock, member 0 2013-07-26 10:13:13.389: [ CSSD][3989223168]clssgmUnregisterPrimary: Unregistering member 0 (0x7ff2e4245f50) in global grock gpnp-cluster-lock 2013-07-26 10:13:13.389: [ CSSD][3989223168]clssgmUnreferenceMember: global grock gpnp-cluster-lock member 0 refcount is 0 2013-07-26 10:13:13.390: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 300 (0x7ff2edded4f0) 2013-07-26 10:13:13.390: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded4f0 (RPC#300) tag(12c002a) sent to node 1 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmHandleMasterMemberExit: [s(1) d(1)] 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmRemoveMember: grock gpnp-cluster-lock, member number 0 (0x7ff2e4245f50) node number 1 state 0x4 grock type 3 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmResetGrock: grock(gpnp-cluster-lock) reset=1 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded4f0 (RPC#300) state 6, flags 0x100 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmDelMemCmpl: rpc 0x7ff2edded4f0, ret 0, client 0x7ff2e4170900 member 0x7ff2e4245f50 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 300 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 301 (0x7ff2edded598) 2013-07-26 10:13:13.390: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x12d002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.391: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 301 2013-07-26 10:13:13.391: [ CSSD][3981948672]clssgmDeleteGrock: (0x7ff2e423cf90) grock(gpnp-cluster-lock) deleted 2013-07-26 10:13:13.391: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb139 2013-07-26 10:13:13.397: [ CSSD][3989223168]clssgmDeadProc: proc 0x7ff2e420a690 2013-07-26 10:13:13.397: [ CSSD][3989223168]clssgmDestroyProc: cleaning up proc(0x7ff2e420a690) con(0xb10a) skgpid 23633 ospid 23633 with 0 clients, refcount 0 2013-07-26 10:13:13.397: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb10a 2013-07-26 10:13:13.401: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e41e9e80 2013-07-26 10:13:13.401: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e41e9e80 2013-07-26 10:13:13.401: [ CSSD][3989223168]clssgmRegisterClient: proc(18/0x7ff2e41e9e80), client(2/0x7ff2e4170900) 2013-07-26 10:13:13.402: [ CSSD][3989223168]clssgmJoinGrock: global grock gpnp-cluster-lock new client 0x7ff2e4170900 with con 0x7ff20000b171, requested num -1, flags 0x12 2013-07-26 10:13:13.402: [ CSSD][3989223168]clssgmAddGrockMember: adding member to grock gpnp-cluster-lock 2013-07-26 10:13:13.402: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 302 (0x7ff2edded640) 2013-07-26 10:13:13.402: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded640 (RPC#302) tag(12e002a) sent to node 1 2013-07-26 10:13:13.402: [ CSSD][3981948672]clssgmHandleMasterMemberAdd: [s(1) d(1)] 2013-07-26 10:13:13.402: [ CSSD][3981948672]clssgmAddMember: granted member(0) flags(0x12) node(1) grock (0x7ff2e4209300/gpnp-cluster-lock) 2013-07-26 10:13:13.402: [ CSSD][3981948672]clssgmQueueGrockEvent: lockName(gpnp-cluster-lock) type(3) count (1/1) xwaiters(1) event(1) to memberNo(0) 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmCommonAddMember: global lock grock gpnp-cluster-lock member(0/Local) node(1) flags 0x12 0xed579070 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded640 (RPC#302) state 6, flags 0x100 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmAddGrockMemCmpl: rpc 0x7ff2edded640, ret 0, client 0x7ff2e4170900 member 0x7ff2e4209700 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmAddGrockMemCmpl: sending type 6, size 540 to 0x7ff2e4170900 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 302 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 303 (0x7ff2edded6e8) 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x12f002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:13.403: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 303 2013-07-26 10:13:13.516: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7b46d0 2013-07-26 10:13:13.516: [ CSSD][3989223168]clssgmAllocProc: (0x7ff2e420a690) allocated 2013-07-26 10:13:13.517: [ CSSD][3989223168]clssgmClientConnectMsg: properties of cmProc 0x7ff2e420a690 - 1,2,3,4,5 2013-07-26 10:13:13.517: [ CSSD][3989223168]clssgmClientConnectMsg: Connect from con(0xb1da) proc(0x7ff2e420a690) pid(23635/23635) version 11:2:1:4, properties: 1,2,3,4,5 2013-07-26 10:13:13.517: [ CSSD][3989223168]clssgmClientConnectMsg: msg flags 0x0000 2013-07-26 10:13:13.519: [ CSSD][3989223168]clssgmExecuteClientRequest: VOTEDISKQUERY recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:13.521: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:13.521: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:13.527: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:13.527: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:13.534: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e420a690 2013-07-26 10:13:13.534: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e420a690 2013-07-26 10:13:13.534: [ CSSD][3989223168]clssgmRegisterClient: proc(20/0x7ff2e420a690), client(1/0x7ff2e4209e30) 2013-07-26 10:13:13.535: [ CSSD][3989223168]clssgmRegisterShared: grp DG_CRSVOTE, mbr 1, type 1 2013-07-26 10:13:13.535: [ CSSD][3989223168]clssgmQueueShare: (0x7ff2e41e9ad0) target global grock DG_CRSVOTE member 1 type 1 queued from client (0x7ff2e4209e30), global grock DG_CRSVOTE, refcount 8 2013-07-26 10:13:13.535: [ CSSD][3989223168]clssgmRegisterShared: global grock DG_CRSVOTE member 1 share type 1, refcount 8 2013-07-26 10:13:13.536: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e420a690 2013-07-26 10:13:13.536: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e420a690 2013-07-26 10:13:13.537: [ CSSD][3989223168]clssgmRegisterClient: proc(20/0x7ff2e420a690), client(2/0x7ff2e41e4370) 2013-07-26 10:13:13.537: [ CSSD][3989223168]clssgmRegisterShared: grp DB+ASM, mbr 0, type 1 2013-07-26 10:13:13.537: [ CSSD][3989223168]clssgmQueueShare: (0x7ff2e422a760) target global grock DB+ASM member 0 type 1 queued from client (0x7ff2e41e4370), global grock DB+ASM, refcount 11 2013-07-26 10:13:13.538: [ CSSD][3989223168]clssgmRegisterShared: global grock DB+ASM member 0 share type 1, refcount 11 2013-07-26 10:13:13.820: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:13.820: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:13.823: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:13.824: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:14.057: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:14.057: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:14.060: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 20 (0x7ff2e420a690) 2013-07-26 10:13:14.061: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:14.476: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 18 (0x7ff2e41e9e80) 2013-07-26 10:13:14.476: [ CSSD][3989223168]clssgmConfig: type(1) 2013-07-26 10:13:14.480: [ CSSD][3989223168]clssgmUnregisterShared: Same group share client 2 (0x7ff2e41e4370), grp DB+ASM, member 0 2013-07-26 10:13:14.480: [ CSSD][3989223168]clssgmTermShare: (0x7ff2e422a760) global grock DB+ASM member 0 type 1 2013-07-26 10:13:14.480: [ CSSD][3989223168]clssgmUnreferenceMember: global grock DB+ASM member 0 refcount is 10 2013-07-26 10:13:14.480: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb255 2013-07-26 10:13:14.481: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 2 2013-07-26 10:13:14.481: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 304 (0x7ff2edded790) 2013-07-26 10:13:14.481: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded790 (RPC#304) tag(130002a) sent to node 1 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded790 (RPC#304) state 6, flags 0x100 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2edded790, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 304 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 305 (0x7ff2edded838) 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x131002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 305 2013-07-26 10:13:14.482: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(304) from node(1). Discarding. 2013-07-26 10:13:14.484: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 3 2013-07-26 10:13:14.484: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 306 (0x7ff2edded8e0) 2013-07-26 10:13:14.485: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2edded8e0 (RPC#306) tag(132002a) sent to node 1 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(25) 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2edded8e0 (RPC#306) state 6, flags 0x100 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2edded8e0, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 306 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 307 (0x7ff2edded988) 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x133002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.485: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 307 2013-07-26 10:13:14.486: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(306) from node(1). Discarding. 2013-07-26 10:13:14.487: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 18 (0x7ff2e41e9e80) 2013-07-26 10:13:14.488: [ CSSD][3989223168]clssgmConfig: type(2) 2013-07-26 10:13:14.488: [ CSSD][3989223168]clssnmStartConfigChange: profile sequence 11 2013-07-26 10:13:14.488: [ CSSD][3989223168]clssnmStartPendingConfigChange: New configuration request for CIN 0:1374804794:0 2013-07-26 10:13:14.488: [ CSSD][3989223168] misscount 30 reboot latency 3 2013-07-26 10:13:14.488: [ CSSD][3989223168] long I/O timeout 200 short I/O timeout 27 2013-07-26 10:13:14.488: [ CSSD][3989223168] diagnostic wait 0 active version 11.2.0.3.0 2013-07-26 10:13:14.488: [ CSSD][3989223168] Listing unique IDs for 3 voting files: 2013-07-26 10:13:14.488: [ CSSD][3989223168] voting file 1: 050ef75c-57354f3d-bfd981c3-3465a922 2013-07-26 10:13:14.488: [ CSSD][3989223168] voting file 2: a4842269-fe624fba-bfa3afc5-99ea2c3e 2013-07-26 10:13:14.488: [ CSSD][3989223168] voting file 3: 6290701d-23924f35-bf8e93b1-a9934d06 2013-07-26 10:13:14.488: [ CSSD][3989223168]clssnmSendVFDiscover: Sending discover voting files request 2013-07-26 10:13:14.488: [ CSSD][3975640832]clssnmHandleVFDiscover: Processing voting file discovery requested by node oeltan1, number 1 2013-07-26 10:13:14.489: [ GPNP][3986679552]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url "ipc://GPNPD_oeltan1" disco "" [f=0 claimed- host: cname: seq: auth:] 2013-07-26 10:13:14.492: [ CSSD][3989223168]clssgmUnregisterShared: Same group share client 1 (0x7ff2e4209e30), grp DG_CRSVOTE, member 1 2013-07-26 10:13:14.493: [ CSSD][3989223168]clssgmTermShare: (0x7ff2e41e9ad0) global grock DG_CRSVOTE member 1 type 1 2013-07-26 10:13:14.493: [ CSSD][3989223168]clssgmUnreferenceMember: global grock DG_CRSVOTE member 1 refcount is 7 2013-07-26 10:13:14.493: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb23f 2013-07-26 10:13:14.494: [ CSSD][3989223168]clssgmDeadProc: proc 0x7ff2e420a690 2013-07-26 10:13:14.494: [ CSSD][3989223168]clssgmDestroyProc: cleaning up proc(0x7ff2e420a690) con(0xb1da) skgpid 23635 ospid 23635 with 0 clients, refcount 0 2013-07-26 10:13:14.495: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb1da 2013-07-26 10:13:14.506: [ GPNP][3986679552]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_oeltan1" disco "" 2013-07-26 10:13:14.506: [ CSSD][3986679552]clssnmReadDiscoveryProfile: voting file discovery string() 2013-07-26 10:13:14.506: [ CSSD][3986679552]clssnmvDDiscThread: using discovery string for voting file add 2013-07-26 10:13:14.506: [ SKGFD][3986679552]Discovery with str:: 2013-07-26 10:13:14.506: [ SKGFD][3986679552]UFS discovery with :: 2013-07-26 10:13:14.506: [ SKGFD][3986679552]Fetching UFS disk :/dev/raw/rawctl: 2013-07-26 10:13:14.506: [ SKGFD][3986679552]OSS discovery with :: 2013-07-26 10:13:14.510: [ CSSD][3986679552]clssnmvDiskVerify: Successful discovery of 0 disks 2013-07-26 10:13:14.510: [ CSSD][3986679552]clssnmCompleteRmtDiscoveryReq: Completing voting file discovery requested by node oeltan1, number 1 2013-07-26 10:13:14.511: [ CSSD][3986679552]clssnmSendDiscoverAck: Discovery complete, notifying requestor node oeltan1 2013-07-26 10:13:14.511: [ CSSD][3975640832]clssnmvVerifyPendingConfigVFs: Insufficient voting files found, found 0 of 3 configured, needed 2 voting files 2013-07-26 10:13:14.511: [ CSSD][3975640832](:CSSNM00022:)clssnmvVerifyPendingConfigVFs: voting file 0, id 050ef75c-57354f3d-bfd981c3-3465a922 not found 2013-07-26 10:13:14.512: [ CSSD][3975640832](:CSSNM00022:)clssnmvVerifyPendingConfigVFs: voting file 1, id a4842269-fe624fba-bfa3afc5-99ea2c3e not found 2013-07-26 10:13:14.512: [ CSSD][3975640832](:CSSNM00022:)clssnmvVerifyPendingConfigVFs: voting file 2, id 6290701d-23924f35-bf8e93b1-a9934d06 not found 2013-07-26 10:13:14.512: [ CSSD][3975640832]clssnmHandleVFDiscoverAck: Found 0 voting files in a pending configuration change, but 2 are required. Terminating the configuration change due to insufficient pending voting files 2013-07-26 10:13:14.512: [ CSSD][3975640832](:CSSNM00027:)clssnmvVerifyNewVFs: voting file 0, id 050ef75c-57354f3d-bfd981c3-3465a922 not found 2013-07-26 10:13:14.512: [ CSSD][3975640832](:CSSNM00012:)clssnmHandleVFDiscoverAck: New voting files not verified. Configuration change rejected 2013-07-26 10:13:14.512: [ CSSD][3975640832]clssnmCompleteConfigChange: Completed configuration change reconfig for CIN 0:1374804794:0 with status 6 2013-07-26 10:13:14.512: [ CSSD][3975640832]clssnmCompleteConfigChange: Committed configuration change for CIN 0:1374802262:0 2013-07-26 10:13:14.512: [ CSSD][3975640832] misscount 30 reboot latency 3 2013-07-26 10:13:14.512: [ CSSD][3975640832] long I/O timeout 200 short I/O timeout 27 2013-07-26 10:13:14.512: [ CSSD][3975640832] diagnostic wait 0 active version 11.2.0.3.0 2013-07-26 10:13:14.512: [ CSSD][3975640832] Listing unique IDs for 0 voting files: 2013-07-26 10:13:14.513: [ CSSD][3975640832]clssnmSetParamsFromConfig: remote SIOT 27000, local SIOT 27000, LIOT 200000, misstime 30000, reboottime 3000, impending misstime 15000, voting file reopen delay 4000 2013-07-26 10:13:14.513: [ CSSD][3975640832]clssnmCompleteGMReq: Completed request type 1 with status 6 2013-07-26 10:13:14.513: [ CSSD][3975640832]clssgmDoneQEle: re-queueing req 0x7ff2e41f7080 status 6 2013-07-26 10:13:14.513: [ CSSD][3989223168]clssgmCheckReqNMCompletion: Completing request type 1 for proc (0x7ff2e41e9e80), operation status 6, client status -21 2013-07-26 10:13:14.515: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 2 2013-07-26 10:13:14.515: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 308 (0x7ff2eddeda30) 2013-07-26 10:13:14.515: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddeda30 (RPC#308) tag(134002a) sent to node 1 2013-07-26 10:13:14.515: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddeda30 (RPC#308) state 6, flags 0x100 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddeda30, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 308 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 309 (0x7ff2eddedad8) 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x135002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 309 2013-07-26 10:13:14.516: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(308) from node(1). Discarding. 2013-07-26 10:13:14.518: [ CSSD][3989223168]clssgmExitGrock: client 2 (0x7ff2e4170900), grock gpnp-cluster-lock, member 0 2013-07-26 10:13:14.518: [ CSSD][3989223168]clssgmUnregisterPrimary: Unregistering member 0 (0x7ff2e4209700) in global grock gpnp-cluster-lock 2013-07-26 10:13:14.518: [ CSSD][3989223168]clssgmUnreferenceMember: global grock gpnp-cluster-lock member 0 refcount is 0 2013-07-26 10:13:14.518: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 310 (0x7ff2eddedb80) 2013-07-26 10:13:14.518: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddedb80 (RPC#310) tag(136002a) sent to node 1 2013-07-26 10:13:14.518: [ CSSD][3981948672]clssgmHandleMasterMemberExit: [s(1) d(1)] 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmRemoveMember: grock gpnp-cluster-lock, member number 0 (0x7ff2e4209700) node number 1 state 0x4 grock type 3 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmResetGrock: grock(gpnp-cluster-lock) reset=1 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddedb80 (RPC#310) state 6, flags 0x100 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmDelMemCmpl: rpc 0x7ff2eddedb80, ret 0, client 0x7ff2e4170900 member 0x7ff2e4209700 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 310 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 311 (0x7ff2eddedc28) 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x137002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 311 2013-07-26 10:13:14.519: [ CSSD][3981948672]clssgmDeleteGrock: (0x7ff2e4209300) grock(gpnp-cluster-lock) deleted 2013-07-26 10:13:14.519: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb171 2013-07-26 10:13:14.520: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 2 2013-07-26 10:13:14.520: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 312 (0x7ff2eddedcd0) 2013-07-26 10:13:14.520: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddedcd0 (RPC#312) tag(138002a) sent to node 1 2013-07-26 10:13:14.520: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddedcd0 (RPC#312) state 6, flags 0x100 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddedcd0, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 312 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 313 (0x7ff2eddedd78) 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x139002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 313 2013-07-26 10:13:14.521: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(312) from node(1). Discarding. 2013-07-26 10:13:14.524: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 3 2013-07-26 10:13:14.524: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 314 (0x7ff2eddede20) 2013-07-26 10:13:14.524: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddede20 (RPC#314) tag(13a002a) sent to node 1 2013-07-26 10:13:14.525: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.525: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(26) 2013-07-26 10:13:14.525: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddede20 (RPC#314) state 6, flags 0x100 2013-07-26 10:13:14.525: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddede20, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:14.525: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 314 2013-07-26 10:13:14.526: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 315 (0x7ff2eddedec8) 2013-07-26 10:13:14.526: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x13b002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.526: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 315 2013-07-26 10:13:14.526: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(314) from node(1). Discarding. 2013-07-26 10:13:14.526: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41e1ab0), type 2 2013-07-26 10:13:14.526: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 316 (0x7ff2eddedf70) 2013-07-26 10:13:14.527: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddedf70 (RPC#316) tag(13c002a) sent to node 1 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddedf70 (RPC#316) state 6, flags 0x100 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddedf70, ret 0, client 0x7ff2e41e1ab0 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 316 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 317 (0x7ff2eddee018) 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x13d002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.527: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 317 2013-07-26 10:13:14.528: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(316) from node(1). Discarding. 2013-07-26 10:13:14.530: [ CSSD][3989223168]clssgmExitGrock: client 1 (0x7ff2e41e1ab0), grock #CLSS_CONFIG_LOCK, member 0 2013-07-26 10:13:14.530: [ CSSD][3989223168]clssgmUnregisterPrimary: Unregistering member 0 (0x7ff2e404e140) in global grock #CLSS_CONFIG_LOCK 2013-07-26 10:13:14.530: [ CSSD][3989223168]clssgmUnreferenceMember: global grock #CLSS_CONFIG_LOCK member 0 refcount is 0 2013-07-26 10:13:14.530: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 318 (0x7ff2eddee0c0) 2013-07-26 10:13:14.530: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee0c0 (RPC#318) tag(13e002a) sent to node 1 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmHandleMasterMemberExit: [s(1) d(1)] 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmRemoveMember: grock #CLSS_CONFIG_LOCK, member number 0 (0x7ff2e404e140) node number 1 state 0x4 grock type 2 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmResetGrock: grock(#CLSS_CONFIG_LOCK) reset=1 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee0c0 (RPC#318) state 6, flags 0x100 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmDelMemCmpl: rpc 0x7ff2eddee0c0, ret 0, client 0x7ff2e41e1ab0 member 0x7ff2e404e140 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 318 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 319 (0x7ff2eddee168) 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x13f002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 319 2013-07-26 10:13:14.531: [ CSSD][3981948672]clssgmDeleteGrock: (0x7ff2e415dbd0) grock(#CLSS_CONFIG_LOCK) not deleted, persistent 4, non-empty 0, awaiting-acks 0, pending-msgs 0 2013-07-26 10:13:14.532: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb079 2013-07-26 10:13:14.534: [ CSSD][3989223168]clssgmDeadProc: proc 0x7ff2e41e9e80 2013-07-26 10:13:14.534: [ CSSD][3989223168]clssgmDestroyProc: cleaning up proc(0x7ff2e41e9e80) con(0xaf9b) skgpid ospid 23626 with 0 clients, refcount 0 2013-07-26 10:13:14.534: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xaf9b 2013-07-26 10:13:14.542: [ CSSD][3989223168]clssgmUnregisterShared: Same group share client 2 (0x7ff2e41af6e0), grp DB+ASM, member 0 2013-07-26 10:13:14.543: [ CSSD][3989223168]clssgmTermShare: (0x7ff2e41cb5a0) global grock DB+ASM member 0 type 1 2013-07-26 10:13:14.543: [ CSSD][3989223168]clssgmUnreferenceMember: global grock DB+ASM member 0 refcount is 9 2013-07-26 10:13:14.543: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb063 2013-07-26 10:13:14.546: [ CSSD][3989223168]clssgmUnregisterShared: Same group share client 1 (0x7ff2e41a2e30), grp DG_CRSVOTE, member 1 2013-07-26 10:13:14.546: [ CSSD][3989223168]clssgmTermShare: (0x7ff2e41d2e60) global grock DG_CRSVOTE member 1 type 1 2013-07-26 10:13:14.546: [ CSSD][3989223168]clssgmUnreferenceMember: global grock DG_CRSVOTE member 1 refcount is 6 2013-07-26 10:13:14.546: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb04d 2013-07-26 10:13:14.547: [ CSSD][3989223168]clssgmDeadProc: proc 0x7ff2e4241cd0 2013-07-26 10:13:14.548: [ CSSD][3989223168]clssgmDestroyProc: cleaning up proc(0x7ff2e4241cd0) con(0xb015) skgpid 23631 ospid 23631 with 0 clients, refcount 0 2013-07-26 10:13:14.548: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb015 2013-07-26 10:13:14.914: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e4057290 2013-07-26 10:13:14.914: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e4057290 2013-07-26 10:13:14.914: [ CSSD][3989223168]clssgmRegisterClient: proc(19/0x7ff2e4057290), client(9/0x7ff2e41af6e0) 2013-07-26 10:13:14.915: [ CSSD][3989223168]clssgmJoinGrock: global grock #CLSS_CONFIG_LOCK new client 0x7ff2e41af6e0 with con 0x7ff20000b38c, requested num -1, flags 0x142000 2013-07-26 10:13:14.915: [ CSSD][3989223168]clssgmAddGrockMember: adding member to grock #CLSS_CONFIG_LOCK 2013-07-26 10:13:14.915: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 320 (0x7ff2eddee210) 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmHandleMasterMemberAdd: [s(1) d(1)] 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmAddMember: new master 0 for group(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmAddMember: Adding fencing for member 0, group #CLSS_CONFIG_LOCK, death 1, SAGE 0 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmAddMember: member (0/0x7ff2e40419a0) added. pbsz(0) prsz(0) flags 0x0 to grock (0x7ff2e415dbd0/#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmCommonAddMember: global group grock #CLSS_CONFIG_LOCK member(0/Local) node(1) flags 0x0 0xed579070 2013-07-26 10:13:14.915: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee210 (RPC#320) tag(140002a) sent to node 1 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee210 (RPC#320) state 6, flags 0x100 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmAddGrockMemCmpl: rpc 0x7ff2eddee210, ret 0, client 0x7ff2e41af6e0 member 0x7ff2e40419a0 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmAddGrockMemCmpl: sending type 6, size 540 to 0x7ff2e41af6e0 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 320 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 321 (0x7ff2eddee2b8) 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x141002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.916: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 321 2013-07-26 10:13:14.918: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41af6e0), type 2 2013-07-26 10:13:14.918: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 322 (0x7ff2eddee360) 2013-07-26 10:13:14.918: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee360 (RPC#322) tag(142002a) sent to node 1 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee360 (RPC#322) state 6, flags 0x100 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddee360, ret 0, client 0x7ff2e41af6e0 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 322 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 323 (0x7ff2eddee408) 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x143002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.919: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 323 2013-07-26 10:13:14.920: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(322) from node(1). Discarding. 2013-07-26 10:13:14.921: [ CSSD][3989223168]clssgmExecuteClientRequest: CONFIG recvd from proc 19 (0x7ff2e4057290) 2013-07-26 10:13:14.921: [ CSSD][3989223168]clssgmConfig: type(2) 2013-07-26 10:13:14.921: [ CSSD][3989223168]clssnmStartConfigChange: profile sequence 11 2013-07-26 10:13:14.921: [ CSSD][3989223168]clssnmStartPendingConfigChange: New configuration request for CIN 0:1374804794:0 2013-07-26 10:13:14.921: [ CSSD][3989223168] misscount 30 reboot latency 3 2013-07-26 10:13:14.921: [ CSSD][3989223168] long I/O timeout 200 short I/O timeout 27 2013-07-26 10:13:14.922: [ CSSD][3989223168] diagnostic wait 0 active version 11.2.0.3.0 2013-07-26 10:13:14.922: [ CSSD][3989223168] Listing unique IDs for 3 voting files: 2013-07-26 10:13:14.922: [ CSSD][3989223168] voting file 1: 050ef75c-57354f3d-bfd981c3-3465a922 2013-07-26 10:13:14.922: [ CSSD][3989223168] voting file 2: a4842269-fe624fba-bfa3afc5-99ea2c3e 2013-07-26 10:13:14.922: [ CSSD][3989223168] voting file 3: 6290701d-23924f35-bf8e93b1-a9934d06 2013-07-26 10:13:14.922: [ CSSD][3989223168]clssnmSendVFDiscover: Sending discover voting files request 2013-07-26 10:13:14.922: [ CSSD][3975640832]clssnmHandleVFDiscover: Processing voting file discovery requested by node oeltan1, number 1 2013-07-26 10:13:14.923: [ GPNP][3986679552]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2104] get-profile call to url "ipc://GPNPD_oeltan1" disco "" [f=0 claimed- host: cname: seq: auth:] 2013-07-26 10:13:14.935: [ GPNP][3986679552]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2234] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_oeltan1" disco "" 2013-07-26 10:13:14.935: [ CSSD][3986679552]clssnmReadDiscoveryProfile: voting file discovery string() 2013-07-26 10:13:14.935: [ CSSD][3986679552]clssnmvDDiscThread: using discovery string for voting file add 2013-07-26 10:13:14.935: [ SKGFD][3986679552]Discovery with str:: 2013-07-26 10:13:14.936: [ SKGFD][3986679552]UFS discovery with :: 2013-07-26 10:13:14.936: [ SKGFD][3986679552]Fetching UFS disk :/dev/raw/rawctl: 2013-07-26 10:13:14.936: [ SKGFD][3986679552]OSS discovery with :: 2013-07-26 10:13:14.936: [ CSSD][3986679552]clssnmvDiskVerify: Successful discovery of 0 disks 2013-07-26 10:13:14.937: [ CSSD][3986679552]clssnmCompleteRmtDiscoveryReq: Completing voting file discovery requested by node oeltan1, number 1 2013-07-26 10:13:14.937: [ CSSD][3986679552]clssnmSendDiscoverAck: Discovery complete, notifying requestor node oeltan1 2013-07-26 10:13:14.937: [ CSSD][3975640832]clssnmvVerifyPendingConfigVFs: Insufficient voting files found, found 0 of 3 configured, needed 2 voting files 2013-07-26 10:13:14.937: [ CSSD][3975640832](:CSSNM00022:)clssnmvVerifyPendingConfigVFs: voting file 0, id 050ef75c-57354f3d-bfd981c3-3465a922 not found 2013-07-26 10:13:14.937: [ CSSD][3975640832](:CSSNM00022:)clssnmvVerifyPendingConfigVFs: voting file 1, id a4842269-fe624fba-bfa3afc5-99ea2c3e not found 2013-07-26 10:13:14.938: [ CSSD][3975640832](:CSSNM00022:)clssnmvVerifyPendingConfigVFs: voting file 2, id 6290701d-23924f35-bf8e93b1-a9934d06 not found 2013-07-26 10:13:14.938: [ CSSD][3975640832]clssnmHandleVFDiscoverAck: Found 0 voting files in a pending configuration change, but 2 are required. Terminating the configuration change due to insufficient pending voting files 2013-07-26 10:13:14.938: [ CSSD][3975640832](:CSSNM00027:)clssnmvVerifyNewVFs: voting file 0, id 050ef75c-57354f3d-bfd981c3-3465a922 not found 2013-07-26 10:13:14.938: [ CSSD][3975640832](:CSSNM00012:)clssnmHandleVFDiscoverAck: New voting files not verified. Configuration change rejected 2013-07-26 10:13:14.938: [ CSSD][3975640832]clssnmCompleteConfigChange: Completed configuration change reconfig for CIN 0:1374804794:0 with status 6 2013-07-26 10:13:14.938: [ CSSD][3975640832]clssnmCompleteConfigChange: Committed configuration change for CIN 0:1374802262:0 2013-07-26 10:13:14.939: [ CSSD][3975640832] misscount 30 reboot latency 3 2013-07-26 10:13:14.939: [ CSSD][3975640832] long I/O timeout 200 short I/O timeout 27 2013-07-26 10:13:14.939: [ CSSD][3975640832] diagnostic wait 0 active version 11.2.0.3.0 2013-07-26 10:13:14.939: [ CSSD][3975640832] Listing unique IDs for 0 voting files: 2013-07-26 10:13:14.939: [ CSSD][3975640832]clssnmSetParamsFromConfig: remote SIOT 27000, local SIOT 27000, LIOT 200000, misstime 30000, reboottime 3000, impending misstime 15000, voting file reopen delay 4000 2013-07-26 10:13:14.939: [ CSSD][3975640832]clssnmCompleteGMReq: Completed request type 1 with status 6 2013-07-26 10:13:14.939: [ CSSD][3975640832]clssgmDoneQEle: re-queueing req 0x7ff2e40bf020 status 6 2013-07-26 10:13:14.939: [ CSSD][3989223168]clssgmCheckReqNMCompletion: Completing request type 1 for proc (0x7ff2e4057290), operation status 6, client status -21 2013-07-26 10:13:14.941: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41af6e0), type 3 2013-07-26 10:13:14.941: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 324 (0x7ff2eddee4b0) 2013-07-26 10:13:14.941: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee4b0 (RPC#324) tag(144002a) sent to node 1 2013-07-26 10:13:14.941: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.941: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(27) 2013-07-26 10:13:14.941: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee4b0 (RPC#324) state 6, flags 0x100 2013-07-26 10:13:14.942: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddee4b0, ret 0, client 0x7ff2e41af6e0 2013-07-26 10:13:14.942: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 324 2013-07-26 10:13:14.942: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 325 (0x7ff2eddee558) 2013-07-26 10:13:14.942: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x145002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.942: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 325 2013-07-26 10:13:14.942: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(324) from node(1). Discarding. 2013-07-26 10:13:14.942: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41af6e0), type 2 2013-07-26 10:13:14.943: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 326 (0x7ff2eddee600) 2013-07-26 10:13:14.943: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee600 (RPC#326) tag(146002a) sent to node 1 2013-07-26 10:13:14.943: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.943: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.943: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:14.943: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee600 (RPC#326) state 6, flags 0x100 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddee600, ret 0, client 0x7ff2e41af6e0 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 326 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 327 (0x7ff2eddee6a8) 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x147002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 327 2013-07-26 10:13:14.944: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(326) from node(1). Discarding. 2013-07-26 10:13:14.972: [ CSSD][3989223168]clssgmExecuteClientRequest: VOTEDISKQUERY recvd from proc 19 (0x7ff2e4057290) 2013-07-26 10:13:14.973: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41af6e0), type 2 2013-07-26 10:13:14.973: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 328 (0x7ff2eddee750) 2013-07-26 10:13:14.973: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee750 (RPC#328) tag(148002a) sent to node 1 2013-07-26 10:13:14.973: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.973: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.973: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee750 (RPC#328) state 6, flags 0x100 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddee750, ret 0, client 0x7ff2e41af6e0 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 328 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 329 (0x7ff2eddee7f8) 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x149002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 329 2013-07-26 10:13:14.974: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(328) from node(1). Discarding. 2013-07-26 10:13:14.976: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41af6e0), type 3 2013-07-26 10:13:14.976: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 330 (0x7ff2eddee8a0) 2013-07-26 10:13:14.976: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee8a0 (RPC#330) tag(14a002a) sent to node 1 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(28) 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee8a0 (RPC#330) state 6, flags 0x100 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddee8a0, ret 0, client 0x7ff2e41af6e0 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 330 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 331 (0x7ff2eddee948) 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x14b002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 331 2013-07-26 10:13:14.977: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(330) from node(1). Discarding. 2013-07-26 10:13:14.978: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e41af6e0), type 2 2013-07-26 10:13:14.978: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 332 (0x7ff2eddee9f0) 2013-07-26 10:13:14.978: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddee9f0 (RPC#332) tag(14c002a) sent to node 1 2013-07-26 10:13:14.978: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.978: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.978: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:14.978: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddee9f0 (RPC#332) state 6, flags 0x100 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddee9f0, ret 0, client 0x7ff2e41af6e0 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 332 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 333 (0x7ff2eddeea98) 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x14d002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 333 2013-07-26 10:13:14.979: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(332) from node(1). Discarding. 2013-07-26 10:13:14.980: [ CSSD][3989223168]clssgmExitGrock: client 9 (0x7ff2e41af6e0), grock #CLSS_CONFIG_LOCK, member 0 2013-07-26 10:13:14.980: [ CSSD][3989223168]clssgmUnregisterPrimary: Unregistering member 0 (0x7ff2e40419a0) in global grock #CLSS_CONFIG_LOCK 2013-07-26 10:13:14.981: [ CSSD][3989223168]clssgmUnreferenceMember: global grock #CLSS_CONFIG_LOCK member 0 refcount is 0 2013-07-26 10:13:14.981: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 334 (0x7ff2eddeeb40) 2013-07-26 10:13:14.981: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddeeb40 (RPC#334) tag(14e002a) sent to node 1 2013-07-26 10:13:14.982: [ CSSD][3981948672]clssgmHandleMasterMemberExit: [s(1) d(1)] 2013-07-26 10:13:14.982: [ CSSD][3981948672]clssgmRemoveMember: grock #CLSS_CONFIG_LOCK, member number 0 (0x7ff2e40419a0) node number 1 state 0x4 grock type 2 2013-07-26 10:13:14.982: [ CSSD][3981948672]clssgmResetGrock: grock(#CLSS_CONFIG_LOCK) reset=1 2013-07-26 10:13:14.982: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddeeb40 (RPC#334) state 6, flags 0x100 2013-07-26 10:13:14.982: [ CSSD][3981948672]clssgmDelMemCmpl: rpc 0x7ff2eddeeb40, ret 0, client 0x7ff2e41af6e0 member 0x7ff2e40419a0 2013-07-26 10:13:14.982: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 334 2013-07-26 10:13:14.983: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 335 (0x7ff2eddeebe8) 2013-07-26 10:13:14.983: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x14f002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.983: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 335 2013-07-26 10:13:14.983: [ CSSD][3981948672]clssgmDeleteGrock: (0x7ff2e415dbd0) grock(#CLSS_CONFIG_LOCK) not deleted, persistent 4, non-empty 0, awaiting-acks 0, pending-msgs 0 2013-07-26 10:13:14.983: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb38c 2013-07-26 10:13:14.986: [ CSSD][3989223168]clssscSelect: cookie accept request 0x7ff2e4057290 2013-07-26 10:13:14.986: [ CSSD][3989223168]clssscevtypSHRCON: getting client with cmproc 0x7ff2e4057290 2013-07-26 10:13:14.986: [ CSSD][3989223168]clssgmRegisterClient: proc(19/0x7ff2e4057290), client(10/0x7ff2e4243a10) 2013-07-26 10:13:14.986: [ CSSD][3989223168]clssgmJoinGrock: global grock #CLSS_CONFIG_LOCK new client 0x7ff2e4243a10 with con 0x7ff20000b437, requested num -1, flags 0x142000 2013-07-26 10:13:14.987: [ CSSD][3989223168]clssgmAddGrockMember: adding member to grock #CLSS_CONFIG_LOCK 2013-07-26 10:13:14.987: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 336 (0x7ff2eddeec90) 2013-07-26 10:13:14.987: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddeec90 (RPC#336) tag(150002a) sent to node 1 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmHandleMasterMemberAdd: [s(1) d(1)] 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmAddMember: new master 0 for group(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmAddMember: Adding fencing for member 0, group #CLSS_CONFIG_LOCK, death 1, SAGE 0 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmAddMember: member (0/0x7ff2e4081150) added. pbsz(0) prsz(0) flags 0x0 to grock (0x7ff2e415dbd0/#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(1), incarn 1, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmCommonAddMember: global group grock #CLSS_CONFIG_LOCK member(0/Local) node(1) flags 0x0 0xed579070 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddeec90 (RPC#336) state 6, flags 0x100 2013-07-26 10:13:14.987: [ CSSD][3981948672]clssgmAddGrockMemCmpl: rpc 0x7ff2eddeec90, ret 0, client 0x7ff2e4243a10 member 0x7ff2e4081150 2013-07-26 10:13:14.988: [ CSSD][3981948672]clssgmAddGrockMemCmpl: sending type 6, size 540 to 0x7ff2e4243a10 2013-07-26 10:13:14.988: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 336 2013-07-26 10:13:14.988: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 337 (0x7ff2eddeed38) 2013-07-26 10:13:14.988: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x151002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.988: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 337 2013-07-26 10:13:14.989: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e4243a10), type 2 2013-07-26 10:13:14.989: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 338 (0x7ff2eddeede0) 2013-07-26 10:13:14.989: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddeede0 (RPC#338) tag(152002a) sent to node 1 2013-07-26 10:13:14.989: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.989: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.989: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:14.989: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddeede0 (RPC#338) state 6, flags 0x100 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddeede0, ret 0, client 0x7ff2e4243a10 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 338 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 339 (0x7ff2eddeee88) 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x153002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 339 2013-07-26 10:13:14.990: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(338) from node(1). Discarding. 2013-07-26 10:13:14.992: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e4243a10), type 3 2013-07-26 10:13:14.992: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 340 (0x7ff2eddeef30) 2013-07-26 10:13:14.993: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddeef30 (RPC#340) tag(154002a) sent to node 1 2013-07-26 10:13:14.993: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.993: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(29) 2013-07-26 10:13:14.993: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddeef30 (RPC#340) state 6, flags 0x100 2013-07-26 10:13:14.993: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddeef30, ret 0, client 0x7ff2e4243a10 2013-07-26 10:13:14.994: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 340 2013-07-26 10:13:14.994: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 341 (0x7ff2eddeefd8) 2013-07-26 10:13:14.994: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x155002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.994: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 341 2013-07-26 10:13:14.994: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(340) from node(1). Discarding. 2013-07-26 10:13:14.994: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e4243a10), type 2 2013-07-26 10:13:14.995: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 342 (0x7ff2eddef080) 2013-07-26 10:13:14.995: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddef080 (RPC#342) tag(156002a) sent to node 1 2013-07-26 10:13:14.995: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:14.995: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:14.995: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:14.995: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:14.995: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddef080 (RPC#342) state 6, flags 0x100 2013-07-26 10:13:14.995: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddef080, ret 0, client 0x7ff2e4243a10 2013-07-26 10:13:14.996: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 342 2013-07-26 10:13:14.996: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 343 (0x7ff2eddef128) 2013-07-26 10:13:14.996: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x157002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:14.996: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 343 2013-07-26 10:13:14.996: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(342) from node(1). Discarding. 2013-07-26 10:13:14.999: [ CSSD][3989223168]clssgmExecuteClientRequest: VOTEDISKQUERY recvd from proc 19 (0x7ff2e4057290) 2013-07-26 10:13:15.040: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e4243a10), type 2 2013-07-26 10:13:15.040: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 344 (0x7ff2eddef1d0) 2013-07-26 10:13:15.040: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddef1d0 (RPC#344) tag(158002a) sent to node 1 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(0) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(0/0) 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddef1d0 (RPC#344) state 6, flags 0x100 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddef1d0, ret 0, client 0x7ff2e4243a10 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 344 2013-07-26 10:13:15.041: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 345 (0x7ff2eddef278) 2013-07-26 10:13:15.042: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x159002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:15.042: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 345 2013-07-26 10:13:15.042: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(344) from node(1). Discarding. 2013-07-26 10:13:15.044: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e4243a10), type 3 2013-07-26 10:13:15.044: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 346 (0x7ff2eddef320) 2013-07-26 10:13:15.044: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddef320 (RPC#346) tag(15a002a) sent to node 1 2013-07-26 10:13:15.044: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:15.044: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), private data(1088), incarn(30) 2013-07-26 10:13:15.045: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddef320 (RPC#346) state 6, flags 0x100 2013-07-26 10:13:15.045: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddef320, ret 0, client 0x7ff2e4243a10 2013-07-26 10:13:15.045: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 346 2013-07-26 10:13:15.045: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 347 (0x7ff2eddef3c8) 2013-07-26 10:13:15.045: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x15b002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:15.045: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 347 2013-07-26 10:13:15.046: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(346) from node(1). Discarding. 2013-07-26 10:13:15.046: [ CSSD][3989223168]clssgmExecuteClientRequest: Received group property update request from client (0x7ff2e4243a10), type 2 2013-07-26 10:13:15.046: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 348 (0x7ff2eddef470) 2013-07-26 10:13:15.047: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddef470 (RPC#348) tag(15c002a) sent to node 1 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmHandleMasterGrpData: [s(1) d(1)] 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmGrockOpTagProcess: Request to commission member(-1) using key(0) for grock(#CLSS_CONFIG_LOCK) 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmUpdateGrpData: grock(#CLSS_CONFIG_LOCK), commissioner(-1/0) 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmQueueGrockEvent: groupName(#CLSS_CONFIG_LOCK) count(1) master(0) event(18), incarn 0, mbrc 1, to member 0, events 0x80, state 0x0 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddef470 (RPC#348) state 6, flags 0x100 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmGrpDataCompl: rpc 0x7ff2eddef470, ret 0, client 0x7ff2e4243a10 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 348 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 349 (0x7ff2eddef518) 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x15d002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:15.047: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 349 2013-07-26 10:13:15.048: [ CSSD][3981948672]clssgmDispatchCMXMSG: received a timed out RPC id(348) from node(1). Discarding. 2013-07-26 10:13:15.049: [ CSSD][3989223168]clssgmExitGrock: client 10 (0x7ff2e4243a10), grock #CLSS_CONFIG_LOCK, member 0 2013-07-26 10:13:15.049: [ CSSD][3989223168]clssgmUnregisterPrimary: Unregistering member 0 (0x7ff2e4081150) in global grock #CLSS_CONFIG_LOCK 2013-07-26 10:13:15.049: [ CSSD][3989223168]clssgmUnreferenceMember: global grock #CLSS_CONFIG_LOCK member 0 refcount is 0 2013-07-26 10:13:15.049: [ CSSD][3989223168]clssgmAllocateRPCIndex: allocated rpc 350 (0x7ff2eddef5c0) 2013-07-26 10:13:15.049: [ CSSD][3989223168]clssgmRPC: rpc 0x7ff2eddef5c0 (RPC#350) tag(15e002a) sent to node 1 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmHandleMasterMemberExit: [s(1) d(1)] 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmRemoveMember: grock #CLSS_CONFIG_LOCK, member number 0 (0x7ff2e4081150) node number 1 state 0x4 grock type 2 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmResetGrock: grock(#CLSS_CONFIG_LOCK) reset=1 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmRPCDone: rpc 0x7ff2eddef5c0 (RPC#350) state 6, flags 0x100 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmDelMemCmpl: rpc 0x7ff2eddef5c0, ret 0, client 0x7ff2e4243a10 member 0x7ff2e4081150 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 350 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmAllocateRPCIndex: allocated rpc 351 (0x7ff2eddef668) 2013-07-26 10:13:15.050: [ CSSD][3981948672]clssgmRPCBroadcast: rpc(0x15f002a), status(0), sendcount(0), filtered by specific properties: 2013-07-26 10:13:15.051: [ CSSD][3981948672]clssgmFreeRPCIndex: freeing rpc 351 2013-07-26 10:13:15.051: [ CSSD][3981948672]clssgmDeleteGrock: (0x7ff2e415dbd0) grock(#CLSS_CONFIG_LOCK) not deleted, persistent 4, non-empty 0, awaiting-acks 0, pending-msgs 0 2013-07-26 10:13:15.051: [ CSSD][3989223168]clssgmDiscEndpcl: gipcDestroy 0xb437 2013-07-26 10:13:15.783: [ CSSD][3978794752]clssnmSendingThread: sending status msg to all nodes 2013-07-26 10:13:15.783: [ CSSD][3978794752]clssnmSendingThread: sent 5 status msgs to all nodes 2013-07-26 10:13:20.787: [ CSSD][3978794752]clssnmSendingThread: sending status msg to all nodes 2013-07-26 10:13:20.787: [ CSSD][3978794752]clssnmSendingThread: sent 5 status msgs to all nodes 2013-07-26 10:13:25.793: [ CSSD][3978794752]clssnmSendingThread: sending status msg to all nodes 2013-07-26 10:13:25.793: [ CSSD][3978794752]clssnmSendingThread: sent 5 status msgs to all nodes 2013-07-26 10:13:29.797: [ CSSD][3978794752]clssnmSendingThread: sending status msg to all nodes 2013-07-26 10:13:29.798: [ CSSD][3978794752]clssnmSendingThread: sent 4 status msgs to all nodes 2013-07-26 10:13:34.805: [ CSSD][3978794752]clssnmSendingThread: sending status msg to all nodes 2013-07-26 10:13:34.805: [ CSSD][3978794752]clssnmSendingThread: sent 5 status msgs to all nodes
上述问题是由于asm_diskstring参数引起：
4.解决过程：
SQL> create spfile from pfile='/tmp/asm_pfile_0724.txt';File created.
SQL> shutdown immediate
ORA-15100: invalid or missing diskgroup name
ASM instance shutdown
SQL> startup
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> alter system set asm_diskstring='/dev/asm*' scope=both;
System altered.
SQL> startup force mount;
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /apps/grid/gridhome/11.2.0/gri
d/dbs/spfile+ASM1.ora
[root@oeltan1 ~]# crsctl replace votedisk +CRSVOTE
Successful addition of voting disk 54e90342890c4f58bf319f3c57c4acd7.
Successful addition of voting disk 5103d7e756f14f69bf3935ace34e5de7.
Successful addition of voting disk 370645ed11324fe5bf99eac9bf5efb21.
Successfully replaced voting disk group with +CRSVOTE.
CRS-4266: Voting file(s) successfully replaced