Oracle 11203 GRID在CRS磁盘组损坏后的处理
11.2.0.3重建CRS磁盘，by ORA老菜，转载请注明作者和出处
本文主要是处理CRS磁盘组(ocr和votedisk所在磁盘组)损坏的情况下，对11.2.0.3 GRID重建CRS磁盘的处理过程
一、故障模拟
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3012
Available space (kbytes) : 259108
ID : 1676783525
Device/File Name : +asmcrs
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
SQL> select inst_name from v$active_instances;
INST_NAME
--------------------------------------------------------------------------------
dbrac1:rac1
dbrac2:rac2
SQL> col name format a30
SQL> col path format a40
SQL> select name,path from v$asm_disk;
NAME PATH
------------------------------ ----------------------------------------
/dev/rhdisk10
/dev/rhdisk11
/dev/rhdisk2
/dev/rhdisk3
/dev/rhdisk9
ASMCRS_0000 /dev/rhdisk4
ASMVG1_0000 /dev/rhdisk5
ASMVG1_0001 /dev/rhdisk6
ASMVG1_0002 /dev/rhdisk7
ASMVG1_0003 /dev/rhdisk8
10 rows selected.
/oracle/asm/cdata/dbrac-cluster
#
#
#
#
# ls -lrt
total 86144
-rw------- 1 root system 7331840 May 17 02:07 week_.ocr
-rw------- 1 root system 7331840 May 18 10:11 day.ocr
-rw------- 1 root system 7331840 May 19 10:11 day_.ocr
-rw------- 1 root system 7331840 May 19 22:11 backup02.ocr
-rw------- 1 root system 7331840 May 20 02:11 backup01.ocr
-rw------- 1 root system 7331840 May 20 06:11 backup00.ocr
-rw------- 1 root system 105559 May 31 02:48 ocr0531.ocr
dd if=/dev/zero of=/dev/rhdisk4 bs=1024 count=2560
# dd if=/dev/zero of=/dev/rhdisk4 bs=1024 count=2560
2560+0 records in.
2560+0 records out.
#
# ./srvctl stop database -d rac
# ./crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbrac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'dbrac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dbrac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.ASMCRS.dg' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.ASMVG1.dg' on 'dbrac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'dbrac1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.dbrac1.vip' on 'dbrac1'
CRS-2677: Stop of 'ora.dbrac1.vip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.dbrac1.vip' on 'dbrac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'dbrac2'
CRS-2677: Stop of 'ora.ASMVG1.dg' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.dbrac1.vip' on 'dbrac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'dbrac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'dbrac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'dbrac2' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.cvu' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'dbrac2'
CRS-2677: Stop of 'ora.oc4j' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'dbrac2'
CRS-2676: Start of 'ora.cvu' on 'dbrac2' succeeded
CRS-2676: Start of 'ora.oc4j' on 'dbrac2' succeeded
CRS-2677: Stop of 'ora.ASMCRS.dg' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbrac1'
CRS-2677: Stop of 'ora.asm' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'dbrac1'
CRS-2677: Stop of 'ora.ons' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'dbrac1'
CRS-2677: Stop of 'ora.net1.network' on 'dbrac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'dbrac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.crf' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.asm' on 'dbrac1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'dbrac1'
CRS-2677: Stop of 'ora.mdnsd' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'dbrac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dbrac1'
CRS-2677: Stop of 'ora.cssd' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'dbrac1'
CRS-2677: Stop of 'ora.gipcd' on 'dbrac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'dbrac1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'dbrac1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'dbrac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbrac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
二、处理过程如下：
在node1启动crs
# ./crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'dbrac1'
CRS-2676: Start of 'ora.mdnsd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'dbrac1'
CRS-2676: Start of 'ora.gpnpd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'dbrac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'dbrac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'dbrac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'dbrac1'
CRS-2676: Start of 'ora.diskmon' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'dbrac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'dbrac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'dbrac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dbrac1'
CRS-2676: Start of 'ora.ctssd' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'dbrac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'dbrac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'dbrac1'
CRS-2676: Start of 'ora.asm' on 'dbrac1' succeeded
# ps -ef|grep smon
root 2228290 7536750 0 02:56:04 pts/5 0:00 grep smon
grid 4063380 1 0 02:54:49 - 0:00 asm_smon_+ASM1
su - grid
为OCR和votedisk创建ASM磁盘组
$ id
uid=301(grid) gid=300(dba) groups=1(staff),600(hagsuser)
$
$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Thu May 31 02:55:52 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL>
SQL>
SQL>
SQL>
SQL> create diskgroup ASMCRS external redundancy disk '/dev/rhdisk4' attribute 'COMPATIBLE.ASM' = '11.2';
Diskgroup created
# id
uid=0(root) gid=0(system) groups=2(bin),3(sys),7(security),8(cron),10(audit),11(lp),600(hagsuser)
# pwd
/oracle/asm/cdata/dbrac-cluster
#
# ls -lrt
total 86136
-rw------- 1 root system 7331840 May 17 02:07 week_.ocr
-rw------- 1 root system 7331840 May 18 10:11 day.ocr
-rw------- 1 root system 7331840 May 19 10:11 day_.ocr
-rw------- 1 root system 7331840 May 19 22:11 backup02.ocr
-rw------- 1 root system 7331840 May 20 02:11 backup01.ocr
-rw------- 1 root system 7331840 May 20 06:11 backup00.ocr
恢复ocr
#/oracle/asm/bin/ocrconfig -restore backup00.ocr
恢复vote disk
# /oracle/asm/bin/crsctl replace votedisk +ASMCRS
Successful addition of voting disk 986d9f44180e4f5cbf1c328c2c7d727f.
Successfully replaced voting disk group with +ASMCRS.
CRS-4266: Voting file(s) successfully replaced
关闭crs
./crsctl stop crs -f
启动crs
# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
检查资源情况
# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.ASMCRS.dg ora....up.type ONLINE ONLINE dbrac1
ora.ASMVG1.dg ora....up.type ONLINE ONLINE dbrac1
ora....ER.lsnr ora....er.type ONLINE ONLINE dbrac1
ora....N1.lsnr ora....er.type ONLINE ONLINE dbrac1
ora.asm ora.asm.type ONLINE ONLINE dbrac1
ora.cvu ora.cvu.type ONLINE ONLINE dbrac1
ora....SM1.asm application ONLINE ONLINE dbrac1
ora....C1.lsnr application ONLINE ONLINE dbrac1
ora.dbrac1.gsd application OFFLINE OFFLINE
ora.dbrac1.ons application ONLINE ONLINE dbrac1
ora.dbrac1.vip ora....t1.type ONLINE ONLINE dbrac1
ora....SM2.asm application ONLINE ONLINE dbrac2
ora....C2.lsnr application ONLINE ONLINE dbrac2
ora.dbrac2.gsd application OFFLINE OFFLINE
ora.dbrac2.ons application ONLINE ONLINE dbrac2
ora.dbrac2.vip ora....t1.type ONLINE ONLINE dbrac2
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE dbrac1
ora.oc4j ora.oc4j.type ONLINE ONLINE dbrac1
ora.ons ora.ons.type ONLINE ONLINE dbrac1
ora.rac.db ora....se.type ONLINE ONLINE dbrac1
ora....ry.acfs ora....fs.type ONLINE ONLINE dbrac1
ora.scan1.vip ora....ip.type ONLINE ONLINE dbrac1
# ./crsctl check cluster -all
**************************************************************
dbrac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
dbrac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
OK，收工