RAC一个节点内存故障宕机，无法访问
环境描述 :
应用使用sap ERP6.0
数据库使用11g RAC + ASM
巡检时SAP 里面执行事务码db02
查看clusterware状态,只剩下一个节点
bash-3.00$ ./crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS.dg
ONLINE ONLINE r3prddb01
ora.ARCH.dg
ONLINE ONLINE r3prddb01
ora.DATA.dg
ONLINE ONLINE r3prddb01
ora.LISTENER.lsnr
ONLINE ONLINE r3prddb01
ora.MLOG.dg
ONLINE ONLINE r3prddb01
ora.OLOG.dg
ONLINE ONLINE r3prddb01
ora.RECO.dg
ONLINE ONLINE r3prddb01
ora.VCR.dg
ONLINE ONLINE r3prddb01
ora.acfs.acfs.acfs
ONLINE ONLINE r3prddb01
ora.asm
ONLINE ONLINE r3prddb01
ora.gsd
OFFLINE OFFLINE r3prddb01
ora.net1.network
ONLINE ONLINE r3prddb01
ora.ons
ONLINE ONLINE r3prddb01
ora.registry.acfs
ONLINE ONLINE r3prddb01
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE r3prddb01
ora.cvu
1 ONLINE ONLINE r3prddb01
ora.oc4j
1 ONLINE ONLINE r3prddb01
ora.p01.db
1 ONLINE ONLINE r3prddb01 Open
2 ONLINE OFFLINE
ora.r3prddb01.vip
1 ONLINE ONLINE r3prddb01
ora.r3prddb02.vip
1 ONLINE INTERMEDIATE r3prddb01 FAILED OVER
ora.scan1.vip
1 ONLINE ONLINE r3prddb01
bash-3.00$
db01 数据库实例alert日志，
Sat Aug 23 11:17:20 2014
Reconfiguration started (old inc 4, new inc 6)
List of instances:
1 (myinst: 1)
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Sat Aug 23 11:17:20 2014
LMS 1: 6 GCS shadows cancelled, 0 closed, 0 Xw survived
Sat Aug 23 11:17:20 2014
LMS 0: 5 GCS shadows cancelled, 1 closed, 0 Xw survived
Sat Aug 23 11:17:20 2014
LMS 2: 5 GCS shadows cancelled, 2 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Sat Aug 23 11:17:20 2014
Instance recovery: looking for dead threads
Beginning instance recovery of 1 threads
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
parallel recovery started with 31 processes
Started redo scan
Completed redo scan
read 58319 KB redo, 11434 data blocks need recovery
Started redo application at
Thread 2: logseq 49560, block 81359
Recovery of Online Redo Log: Thread 2 Group 44 Seq 49560 Reading mem 0
Mem# 0: +OLOG/p01/onlinelog/log_g44m1.dbf
Mem# 1: +MLOG/p01/onlinelog/log_g44m2.dbf
Sat Aug 23 11:17:25 2014
Setting Resource Manager plan SCHEDULER[0x447BF]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sat Aug 23 11:17:25 2014
minact-scn: master found reconf/inst-rec before recscn scan old-inc#:6 new-inc#:6
Completed redo application of 48.31MB
Completed instance recovery at
Thread 2: logseq 49560, block 197998, scn 11518227963
10738 data blocks read, 11483 data blocks written, 58319 redo k-bytes read
Thread 2 advanced to log sequence 49561 (thread recovery)
Redo thread 2 internally disabled at seq 49561 (SMON)
Sat Aug 23 11:17:27 2014
Archived Log entry 91800 added for thread 2 sequence 49560 ID 0x592ddd4a dest 1:
Sat Aug 23 11:17:27 2014
ARC0: Archiving disabled thread 2 sequence 49561
Archived Log entry 91801 added for thread 2 sequence 49561 ID 0x592ddd4a dest 1:
minact-scn: master continuing after IR
minact-scn: Master considers inst:2 dead
Sat Aug 23 11:17:28 2014
Beginning log switch checkpoint up to RBA [0xa4e4.2.10], SCN: 11518240393
Thread 1 advanced to log sequence 42212 (LGWR switch)
Current log# 35 seq# 42212 mem# 0: +OLOG/p01/onlinelog/log_g35m1.dbf
Current log# 35 seq# 42212 mem# 1: +MLOG/p01/onlinelog/log_g35m2.dbf
Archived Log entry 91802 added for thread 1 sequence 42211 ID 0x592ddd4a dest 1:
DB01 ASM实例alert日志 ASM01
Sat Aug 23 11:17:20 2014
Reconfiguration started (old inc 4, new inc 6)
List of instances:
1 (myinst: 1)
Global Resource Directory frozen
* dead instance detected - domain 1 invalid = TRUE
* dead instance detected - domain 2 invalid = TRUE
* dead instance detected - domain 3 invalid = TRUE
* dead instance detected - domain 4 invalid = TRUE
* dead instance detected - domain 5 invalid = TRUE
* dead instance detected - domain 6 invalid = TRUE
* dead instance detected - domain 7 invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Sat Aug 23 11:17:20 2014
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Sat Aug 23 11:17:20 2014
NOTE: SMON starting instance recovery for group ACFS domain 1 (mounted)
NOTE: F1X0 found on disk 1 au 49 fcn 0.12248
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
NOTE: starting recovery of thread=2 ckpt=19.43 group=1 (ACFS)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 1 (ACFS)
NOTE: SMON successfully validated lock domain 1
NOTE: advancing ckpt for thread=2 ckpt=19.43
NOTE: SMON did instance recovery for group ACFS domain 1
Sat Aug 23 11:17:20 2014
ASM Volume(VDBG) - Unable to send message 'disk status' to the volume driver.
NOTE: SMON starting instance recovery for group ARCH domain 2 (mounted)
NOTE: F1X0 found on disk 9 au 113 fcn 0.41343439
NOTE: starting recovery of thread=2 ckpt=77.3254 group=2 (ARCH)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (ARCH)
NOTE: SMON successfully validated lock domain 2
NOTE: advancing ckpt for thread=2 ckpt=77.3254
NOTE: SMON did instance recovery for group ARCH domain 2
ASM Volume(VDBG) - Unable to send message 'disk status' to the volume driver.
NOTE: SMON starting instance recovery for group DATA domain 3 (mounted)
NOTE: F1X0 found on disk 15 au 60241 fcn 0.5143392
NOTE: starting recovery of thread=2 ckpt=22.3858 group=3 (DATA)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 3 (DATA)
NOTE: SMON successfully validated lock domain 3
NOTE: advancing ckpt for thread=2 ckpt=22.3858
NOTE: SMON did instance recovery for group DATA domain 3
ASM Volume(VDBG) - Unable to send message 'disk status' to the volume driver.
NOTE: SMON starting instance recovery for group MLOG domain 4 (mounted)
NOTE: F1X0 found on disk 3 au 639 fcn 0.120137
NOTE: starting recovery of thread=2 ckpt=23.2161 group=4 (MLOG)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 4 (MLOG)
NOTE: SMON successfully validated lock domain 4
NOTE: advancing ckpt for thread=2 ckpt=23.2161
NOTE: SMON did instance recovery for group MLOG domain 4
ASM Volume(VDBG) - Unable to send message 'disk status' to the volume driver.
NOTE: SMON starting instance recovery for group OLOG domain 5 (mounted)
NOTE: F1X0 found on disk 3 au 637 fcn 0.121291
NOTE: starting recovery of thread=2 ckpt=23.2261 group=5 (OLOG)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 5 (OLOG)
NOTE: SMON successfully validated lock domain 5
NOTE: advancing ckpt for thread=2 ckpt=23.2261
NOTE: SMON did instance recovery for group OLOG domain 5
ASM Volume(VDBG) - Unable to send message 'disk status' to the volume driver.
NOTE: SMON starting instance recovery for group RECO domain 6 (mounted)
NOTE: F1X0 found on disk 11 au 11 fcn 0.2264
NOTE: starting recovery of thread=2 ckpt=19.6 group=6 (RECO)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 6 (RECO)
NOTE: SMON successfully validated lock domain 6
NOTE: advancing ckpt for thread=2 ckpt=19.6
NOTE: SMON did instance recovery for group RECO domain 6
ASM Volume(VDBG) - Unable to send message 'disk status' to the volume driver.
NOTE: SMON starting instance recovery for group VCR domain 7 (mounted)
NOTE: F1X0 found on disk 2 au 177 fcn 0.1216
NOTE: starting recovery of thread=2 ckpt=16.13 group=7 (VCR)
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 7 (VCR)
NOTE: SMON successfully validated lock domain 7
NOTE: advancing ckpt for thread=2 ckpt=16.13
NOTE: SMON did instance recovery for group VCR domain 7
通过以上日志判断,故障发生在 23日 11:17 ，联系机房同事，检查发现M9000上该机器告警，内存故障，已报修。
机器修理中。。。。
第二天02机器内存故障已修好，启动02操作系统。等待一会儿发现cluserware自动起来了，02数据库实例也恢复
该机器设置的cluster随着操作系统启动自动启动，默认安装好设置也是这样的。
如果设置不自动启动执行以下命令crsctl disable crs，使用crsctl enable crs改回开启自动启动
设置后，需要手工启动执行crsctl start crs命令
bash-3.00$ ./crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.ARCH.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.DATA.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.LISTENER.lsnr
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.MLOG.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.OLOG.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.RECO.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.VCR.dg
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.acfs.acfs.acfs
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.asm
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.gsd
OFFLINE OFFLINE r3prddb01
OFFLINE OFFLINE r3prddb02
ora.net1.network
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.ons
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
ora.registry.acfs
ONLINE ONLINE r3prddb01
ONLINE ONLINE r3prddb02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE r3prddb01
ora.cvu
1 ONLINE ONLINE r3prddb01
ora.oc4j
1 ONLINE ONLINE r3prddb01
ora.p01.db
1 ONLINE ONLINE r3prddb01 Open
2 ONLINE ONLINE r3prddb02 Open
ora.r3prddb01.vip
1 ONLINE ONLINE r3prddb01
ora.r3prddb02.vip
1 ONLINE ONLINE r3prddb02
ora.scan1.vip
1 ONLINE ONLINE r3prddb01
DB02查看已经两个实例
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/27771627/viewspace-1257887/，如需转载，请注明出处，否则将追究法律责任。
转载于:http://blog.itpub.net/27771627/viewspace-1257887/