ORACLE 11G RAC ASM磁盘全部丢失后的恢复
一、环境描述
（1）Oracle 11.2.0.3 RAC ON Oracle Linux 6 x86_64，只有一个ASM外部冗余磁盘组——DATA；
（2）OCR，VOTEDISK，DATAFILE，CONTROLFILE，SPFILE全部位于这个磁盘组上；
二、故障描述
（1）存储故障导致ASM磁盘丢失。
（2）CRS因为OCR和VOTEDISK的丢失，除了OHAS还联机外，CLUSTERWARE服务都已经停止。
三、备份情况
（1）RMAN备份：包括controlfile，database，spfile，archivelog，
（2）OCR备份：没有进行过人工备份，在$CRS_HOME/cdata目录下有CRS自动备份文件。
四、操作步骤
说明：准使用CRS自动备份的文件恢复OCR，使用RMAN备份来恢复数据库；准备恢复数据的同时，调整ASM磁盘组，将OCR，VOTEDISK同数据库文件分开存放。
4.1 恢复OCR和VOTEDISK
(1) 在所有RAC节点上停止CRS服务
[root@rac1 ~]# crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.[root@rac2 ~]# crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
(2) 在一个节点上以NOCRS方式启动CRS，此操作会启动ASM实例。
[root@rac1 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
(3) 新添加了三块磁盘，已经使用UDEV进行了绑定，查看磁盘状态。
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 5 17:41:49 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select group_number group#, disk_number disk#, OS_MB, state, path, header_status from v$asm_disk order by 1,2;
GROUP# DISK# OS_MB STATE PATH HEADER_STATUS
---------- ---------- ---------- ---------- -------------------- ----------------------
0 0 1024 NORMAL /dev/asm-diskc CANDIDATE
0 1 5120 NORMAL /dev/asm-diskd CANDIDATE
0 2 20480 NORMAL /dev/asm-diskb CANDIDATE
(4) 创建三个磁盘组，SYSTEMDG给CRS使用，用于存放OCR，VOTEDISK和ASM实例的SPFILE。其余两个给ORACLE使用，DATADG用于存放datafile，controlfile，redolog，spfile；ARCLOGDG存放archivelog。
SQL> create diskgroup SYSTEMDG external redundancy
2 disk '/dev/asm-diskc'
3 ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';
Diskgroup created.
SQL> create diskgroup DATADG external redundancy
2 disk '/dev/asm-diskb'
3 ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';
Diskgroup created.
SQL> create diskgroup ARCLOGDG external redundancy
2 disk '/dev/asm-diskd'
3 ATTRIBUTE 'compatible.rdbms' = '11.2','compatible.asm' = '11.2';
Diskgroup created.
(5) 准备恢复OCR和VOTEDISK，/etc/oracle/ocr.loc中记录了OCR路径，修改ocrconfig_loc的值，以便将OCR恢复到新的磁盘组中。
[root@rac1 ~]# more /etc/oracle/ocr.loc
ocrconfig_loc=+DATA
local_only=FALSE
[root@rac1 ~]# vi /etc/oracle/ocr.loc
ocrconfig_loc=+SYSTEMDG
local_only=FALSE
(6) 恢复OCR
[root@rac1 ~]# ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
rac1 2013/07/05 12:30:00 /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
rac1 2013/07/05 08:30:00 /u01/app/11.2.0/grid/cdata/rac-cluster/backup01.ocr
rac1 2013/07/05 04:30:00 /u01/app/11.2.0/grid/cdata/rac-cluster/backup02.ocr
rac1 2013/07/05 00:29:59 /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocr
rac1 2013/07/05 00:29:59 /u01/app/11.2.0/grid/cdata/rac-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
[root@rac1 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
[root@rac1 ~]#
[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2840
Available space (kbytes) : 259280
ID : 59415097
Device/File Name : +SYSTEMDG
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
(7) 创建VOTEDISK
[root@rac1 ~]# crsctl replace votedisk +SYSTEMDG
CRS-4602: Failed 27 to add voting file afb0ca0f35684f1abfd43d5ec2dc1123.
Failed to replace voting disk group with +SYSTEMDG.
CRS-4000: Command Replace failed, or completed with errors.
以上报错是因为使用UDEV绑定ASM磁盘时需要更改默认磁盘搜索路径为/dev/asm*，修改ASM磁盘搜索路径
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Fri Jul 5 19:03:25 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter asm_diskstring
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring string
SQL>
SQL>
SQL> alter system set asm_diskstring = '/dev/asm*';
System altered.
SQL> create spfile from memory;
create spfile from memory
*
ERROR at line 1:
ORA-00349: failure obtaining block size for
'+DATA/rac-cluster/asmparameterfile/registry.253.819922365'
ORA-15001: diskgroup "DATA" does not exist or is not mounted
SQL> create spfile='+SYSTEMDG' from memory;
File created.
SQL> startup force mount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
在次创建VOTEDISK，成功。
[root@rac1 init]# crsctl replace votedisk +SYSTEMDG
Successful addition of voting disk 8ebb7a63accb4fa8bfa7ab65df7a8c8a.
Successfully replaced voting disk group with +SYSTEMDG.
CRS-4266: Voting file(s) successfully replaced
(8) OCR和VOTEDISK都恢复完成后，重启CRS到正常模式。
[root@rac1 ~]# crsctl stop has -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@rac1 ~]#
4.2 修改CRS注册表中相关配置信息
(1) 挂载新的ASM磁盘组
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 00:16:05 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL>
SQL> select name,state from v$asm_diskgroup;
NAME STATE
------------------------------ -----------
SYSTEMDG MOUNTED
ARCLOGDG DISMOUNTED
DATADG DISMOUNTED
SQL> alter diskgroup ARCLOGDG,DATADG mount;
Diskgroup altered.
(2) 更改CRS配置文件中数据库的磁盘组为DATADG和ARCLOGDG
[root@rac1 ~]# srvctl modify database -d csdb -a "DATADG,ARCLOGDG"
(3) 禁用并删除原来的磁盘组DATA
[root@rac1 ~]# srvctl disable diskgroup -g DATA
[root@rac1 ~]# srvctl remove diskgroup -g DATA
[root@rac1 rac-cluster]# crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora....OGDG.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.DATADG.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.csdb.db ora....se.type 0/2 0/1 ONLINE OFFLINE
ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
(4) 在OCR注册表中修改Oracle数据库参数文件的位置
[root@rac1 ~]# srvctl modify database -d csdb -p +DATADG/csdb/spfilecsdb.ora
4.3 恢复数据库
(1) 查看备份文件路径和名称
[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$
[oracle@rac1 ~]$ cd /u01/app/oracle/backup
[oracle@rac1 backup]$ ll
total 221796
-rw-r----- 1 oracle asmadmin 5357568 Jul 5 15:19 arc_819991156_9.bk
-rw-r----- 1 oracle asmadmin 2560 Jul 5 15:19 arc_819991158_11.bk
-rw-r----- 1 oracle asmadmin 203104256 Jul 5 15:18 CSDB_819991120_5.bk
-rw-r----- 1 oracle asmadmin 18546688 Jul 5 15:19 ctl_file_0coe04jq_1_1_20130705.ctl
-rw-r----- 1 oracle asmadmin 98304 Jul 5 15:19 spfile_0doe04js_1_1_20130705
[oracle@rac1 backup]$
(2) 创建一个基本的启动参数文件，以便启动数据库到nomout状态恢复spfile
[oracle@rac1 ~]$ touch /u01/app/oracle/backup/init.ora
[oracle@rac1 ~]$ vi /u01/app/oracle/backup/init.ora
*.db_name='csdb'
*.remote_login_passwordfile='exclusive'
(3) 使用刚创建的参数文件将数据库启动到nomount状态
[oracle@rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 13:56:06 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup nomount pfile='/u01/app/oracle/backup/init.ora';
ORACLE instance started.
Total System Global Area 238034944 bytes
Fixed Size 2227136 bytes
Variable Size 180356160 bytes
Database Buffers 50331648 bytes
Redo Buffers 5120000 bytes
SQL>
(4) 使用RMAN恢复SPFILE到ASM磁盘组DATADG
[oracle@rac1 ~]$ rman target /
Recovery Manager: Release 11.2.0.3.0 - Production on Sat Jul 6 13:59:26 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: CSDB (not mounted)
RMAN> restore spfile to '+DATADG/csdb/spfilecsdb.ora' from '/u01/app/oracle/backup/spfile_0doe04js_1_1_20130705';
Starting restore at 06-JUL-13
using channel ORA_DISK_1
channel ORA_DISK_1: restoring spfile from AUTOBACKUP /u01/app/oracle/backup/spfile_0doe04js_1_1_20130705
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 06-JUL-13
(5) 使用恢复后spfile启动数据库，并修改control_files，db_recovery_file_dest，log_archive_dest等存在旧路径的参数值。
[oracle@rac1 ~]$ vi $ORACLE_HOME/dbs/initcsdb1.ora
SPFILE='+DATADG/csdb/spfilecsdb.ora'
[oracle@rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 14:11:58 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
SQL> startup nomount force
ORACLE instance started.
Total System Global Area 1653518336 bytes
Fixed Size 2228904 bytes
Variable Size 1073745240 bytes
Database Buffers 570425344 bytes
Redo Buffers 7118848 bytes
SQL>
SQL> show parameter control_files
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_files string +DATA/csdb/control01.ctl, +DAT
A/csdb/control02.ctl
SQL>
SQL> alter system set control_files='+DATADG/csdb/control01.ctl','+DATADG/csdb/control02.ctl' scope=spfile
System altered.
SQL> alter system set db_recovery_file_dest='+DATADG' scope=spfile;
System altered.
SQL> alter system set log_archive_dest_1='LOCATION=+ARCLOGDG' scope=spfile;
System altered.
SQL> startup force nomount;
ORACLE instance started.
Total System Global Area 1653518336 bytes
Fixed Size 2228904 bytes
Variable Size 1073745240 bytes
Database Buffers 570425344 bytes
Redo Buffers 7118848 bytes
(6) 查看数据库的DBID
[oracle@rac1 ~]$ strings /u01/app/oracle/backup/CSDB_819991120_5.bk | grep MAXVALUE,
返回的值类似下面的例子，其中那一窜数字即为DBID。
...
MAXVALUE, MAXVALUE!
3042905279, MAXVALUE,
3042905279, MAXVALUE,
...
(7) 恢复控制文件到新的ASM磁盘组DATADG
[oracle@rac1 ~]$ rman target /
Recovery Manager: Release 11.2.0.3.0 - Production on Sat Jul 6 14:28:28 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: CSDB (not mounted)
RMAN> set dbid=3042905279
executing command: SET DBID
RMAN> restore controlfile from '/u01/app/oracle/backup/ctl_file_0coe04jq_1_1_20130705.ctl';
Starting restore at 06-JUL-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 instance=csdb1 device type=DISK
channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=+DATADG/csdb/control01.ctl
output file name=+DATADG/csdb/control02.ctl
Finished restore at 06-JUL-13
(8) 进入SQLPLUS，查看旧数据文件信息
[oracle@rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sat Jul 6 14:41:12 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> alter database mount;
Database altered.
SQL> col name format a50
SQL> select file#,name from v$datafile;
FILE# NAME
---------- --------------------------------------------------
1 +DATA/csdb/datafile/system.260.819979847
2 +DATA/csdb/datafile/sysaux.261.819979871
3 +DATA/csdb/datafile/undotbs1.262.819979889
4 +DATA/csdb/datafile/undotbs2.264.819979905
5 +DATA/csdb/datafile/users.265.819979913
(9) 使用RMAN恢复数据库
RMAN> run{
2> set newname for datafile 1 to '+DATADG/csdb/datafile/system.260.819979847';
3> set newname for datafile 2 to '+DATADG/csdb/datafile/sysaux.261.819979871';
4> set newname for datafile 3 to '+DATADG/csdb/datafile/undotbs1.262.819979889';
5> set newname for datafile 4 to '+DATADG/csdb/datafile/undotbs2.264.819979905';
6> set newname for datafile 5 to '+DATADG/csdb/datafile/users.265.819979913';
7> restore database;
8> switch datafile all;
9> recover database;
10> }
executing command: SET NEWNAME
released channel: ORA_DISK_1
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
Starting restore at 07-JUL-13
Starting implicit crosscheck backup at 07-JUL-13
allocated channel: ORA_DISK_1
Crosschecked 10 objects
Finished implicit crosscheck backup at 07-JUL-13
Starting implicit crosscheck copy at 07-JUL-13
using channel ORA_DISK_1
Finished implicit crosscheck copy at 07-JUL-13
searching for all files in the recovery area
cataloging files...
no files cataloged
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00002 to +DATADG/csdb/datafile/sysaux.261.819979871
channel ORA_DISK_1: restoring datafile 00003 to +DATADG/csdb/datafile/undotbs1.262.819979889
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/CSDB_819991120_6.bk
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/CSDB_819991120_6.bk tag=ORCL_HOT_DB_BK
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:46
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to +DATADG/csdb/datafile/system.260.819979847
channel ORA_DISK_1: restoring datafile 00004 to +DATADG/csdb/datafile/undotbs2.264.819979905
channel ORA_DISK_1: restoring datafile 00005 to +DATADG/csdb/datafile/users.265.819979913
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/CSDB_819991120_5.bk
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/CSDB_819991120_5.bk tag=ORCL_HOT_DB_BK
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:35
Finished restore at 07-JUL-13
datafile 1 switched to datafile copy
input datafile copy RECID=6 STAMP=820112831 file name=+DATADG/csdb/datafile/system.284.820112797
datafile 2 switched to datafile copy
input datafile copy RECID=7 STAMP=820112831 file name=+DATADG/csdb/datafile/sysaux.282.820112751
datafile 3 switched to datafile copy
input datafile copy RECID=8 STAMP=820112831 file name=+DATADG/csdb/datafile/undotbs1.283.820112751
datafile 4 switched to datafile copy
input datafile copy RECID=9 STAMP=820112831 file name=+DATADG/csdb/datafile/undotbs2.285.820112797
datafile 5 switched to datafile copy
input datafile copy RECID=10 STAMP=820112831 file name=+DATADG/csdb/datafile/users.286.820112797
Starting recover at 07-JUL-13
using channel ORA_DISK_1
starting media recovery
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=15
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/arc_819991156_9.bk
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/arc_819991156_9.bk tag=TAG20130705T151916
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_1_seq_15.256.820112833 thread=1 sequence=15
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=2
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=16
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/arc_819991156_10.bk
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/arc_819991156_10.bk tag=TAG20130705T151916
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_2.257.820112835 thread=2 sequence=2
channel default: deleting archived log(s)
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_1_seq_15.256.820112833 RECID=5 STAMP=820112832
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_1_seq_16.258.820112835 thread=1 sequence=16
channel default: deleting archived log(s)
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_2.257.820112835 RECID=7 STAMP=820112834
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=3
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/arc_819991158_11.bk
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/arc_819991158_11.bk tag=TAG20130705T151916
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_3.257.820112837 thread=2 sequence=3
channel default: deleting archived log(s)
archived log file name=+ARCLOGDG/csdb/archivelog/2013_07_07/thread_2_seq_3.257.820112837 RECID=8 STAMP=820112835
unable to find archived log
archived log thread=2 sequence=4
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 07/07/2013 01:07:16
RMAN-06054: media recovery requesting unknown archived log for thread 2 with sequence 4 and starting SCN of 323980
(10) 更改REDO LOG位置信息
SQL> select member from v$logfile;
MEMBER
--------------------------------------------------------------------------------
+DATA/csdb/redo01.log
+DATA/csdb/redo02.log
+DATA/csdb/redo03.log
+DATA/csdb/redo04.log
SQL> alter database rename file '+DATA/csdb/redo01.log' to '+DATADG/csdb/redo01.log';
Database altered.
SQL> alter database rename file '+DATA/csdb/redo02.log' to '+DATADG/csdb/redo02.log';
Database altered.
SQL> alter database rename file '+DATA/csdb/redo03.log' to '+DATADG/csdb/redo03.log';
Database altered.
SQL> alter database rename file '+DATA/csdb/redo04.log' to '+DATADG/csdb/redo04.log';
Database altered.
(11) 打开数据库
SQL> alter database open resetlogs;
Database altered.
(12) 更改TEMP表空间文件位置
SQL> select name from v$tempfile;
NAME
--------------------------------------------------------------------------------
+DATA/csdb/tempfile/temp.263.819979895
SQL> alter tablespace temp add tempfile '+DATADG';
Tablespace altered.
SQL> alter tablespace temp drop tempfile '+DATA/csdb/tempfile/temp.263.819979895';
Tablespace altered
4.4 完成恢复操作
(1) 在其他RAC节点上更改OCR路径
[root@rac2 ~]# vi /etc/oracle/ocr.loc
ocrconfig_loc=+SYSTEMDG
local_only=FALSE
(2) 在恢复节点上重启CRS
[root@rac1 ~]# crsctl stop crs
[root@rac1 ~]# crsctl start has
(3) 在其他节点上启动CRS
[root@rac2 ~]# crsctl start crs