Oracle10g 裸设备数据文件恢复，ORA-19501、ORA-27063
环境说明：
OS:aix 6.1
DB:Oracle 10gR2 裸设备
使用rman备份整个数据库备份失败，提示71数据文件失败，单独备份这个数据文件
$ rman target /
Recovery Manager: Release 10.2.0.1.0 - Production on Thu Nov 7 09:14:31 2013
Copyright (c) 1982, 2005, Oracle. All rights reserved.
connected to target database: ORCL (DBID=1237792397, not open)
RMAN> backup as copy datafile 71 format '/backup_dir/in71_%U';
Starting backup at 07-NOV-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=1086 devtype=DISK
channel ORA_DISK_1: starting datafile copy
input datafile fno=00071 name=/dev/rdata_03
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 11/07/2013 09:18:10
ORA-19501: read error on file "/dev/rdata_03", blockno 178305 (blocksize=8192)
ORA-27063: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: -1
Additional information: 1048576
RMAN> backup check logical validate datafile 71;
Starting backup at 07-NOV-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00071 name=/dev/rdata_03
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 11/07/2013 09:47:03
ORA-19501: read error on file "/dev/rdata_03", blockno 178305 (blocksize=8192)
ORA-27063: number of bytes read/written is incorrect
IBM AIX RISC System/6000 Error: 5: I/O error
Additional information: -1
Additional information: 1048576
查询坏块个数
SQL> select * from v$database_block_corruption;
no rows selected
没有查询到逻辑坏块，猜测为物理坏块，数据库层面还认为是好的数据库造成了读的时候提示I/O error
检查71号文件的数据块个数
SQL> select bytes/8192 from v$datafile where file#=71;
BYTES/8192
----------
1280000
验证裸设备是否可以读
$ dbv file=/dev/rdata_03 blocksize=8192
DBVERIFY: Release 10.2.0.1.0 - Production on Thu Nov 7 09:48:08 2013
Copyright (c) 1982, 2005, Oracle. All rights reserved.
DBVERIFY - Verification starting : FILE = /dev/rdata_03
DBV-00102: File I/O error on FILE (/dev/rdata_03) during verification read operation (-2)
找出坏块，使用Oracle自带的工具
$ dbv file=/dev/rdata_03 blocksize=8192
DBVERIFY: Release 10.2.0.1.0 - Production on Thu Nov 7 09:48:08 2013
Copyright (c) 1982, 2005, Oracle. All rights reserved.
DBVERIFY - Verification starting : FILE = /dev/rdata_03
DBV-00102: File I/O error on FILE (/dev/rdata_03) during verification read operation (-2)
同时在另一个窗口中查看
$ ps -ef | grep dbv
oracle 598170 593980 38 09:57:42 pts/1 0:00 dbv file=/dev/rdata_03 blocksize=8192
oracle 1712282 581750 0 09:57:46 pts/0 0:00 grep dbv
$ truss -faieo /tmp/strace.out -p 598170
598170: klseek(5, 746045440, 0, 0x0FFFFFFFFFFF9530) = 0
...skipping...
598170: kread(5, " ( 0\011 ???D 5 ??.., 8192) = 8192
598170: klseek(5, 1461190656, 0, 0x0FFFFFFFFFFF9530) = 0
598170: kread(5, " ( 0\011 ???D 5 ??.., 8192) Err#5 EIO
598170: kwrite(2, "\n D B V - 0 0 1 0 2 : ", 12) = 12
598170: kwrite(2, " F i l e I / O e r r".., 24) = 24
598170: kwrite(2, " / d e v / r d a t a _ c".., 15) = 15
598170: kwrite(2, " ) d u r i n g ", 9) = 9
598170: kwrite(2, " v e r i f i c a t i o n".., 17) = 17
598170: kwrite(2, " o p e r a t i o n (", 12) = 12
598170: kwrite(2, " -", 1) = 1
598170: kwrite(2, " 2", 1) = 1
598170: kwrite(2, " )", 1) = 1
598170: kwrite(2, "\n", 1) = 1
598170: close(5) = 0
598170: close(4) = 0
598170: kfcntl(1, F_GETFL, 0x00000000101A2087) = 67110914
598170: kfcntl(2, F_GETFL, 0x00000000101A2087) = 67110914
598170: _exit(1)
...skipping...
598170: kread(5, " ( 0\011 ???D 5 ??.., 8192) = 8192
598170: klseek(5, 1461190656, 0, 0x0FFFFFFFFFFF9530) = 0
598170: kread(5, " ( 0\011 ???D 5 ??.., 8192) Err#5 EIO
598170: kwrite(2, "\n D B V - 0 0 1 0 2 : ", 12) = 12
598170: kwrite(2, " F i l e I / O e r r".., 24) = 24
598170: kwrite(2, " / d e v / r d a t a _ c".., 15) = 15
598170: kwrite(2, " ) d u r i n g ", 9) = 9
598170: kwrite(2, " v e r i f i c a t i o n".., 17) = 17
598170: kwrite(2, " o p e r a t i o n (", 12) = 12
598170: kwrite(2, " -", 1) = 1
598170: kwrite(2, " 2", 1) = 1
598170: kwrite(2, " )", 1) = 1
598170: kwrite(2, "\n", 1) = 1
598170: close(5) = 0
598170: close(4) = 0
598170: kfcntl(1, F_GETFL, 0x00000000101A2087) = 67110914
598170: kfcntl(2, F_GETFL, 0x00000000101A2087) = 67110914
598170: _exit(1)
找到开始的坏块
1461190656/8192=178368
说明第一个坏块为178368，验证最后的坏块，最大的数据块是1280000
$ dbv file=/dev/rdata_03 blocksize=8192 start=1461190656
DBVERIFY: Release 10.2.0.1.0 - Production on Thu Nov 7 10:10:58 2013
Copyright (c) 1982, 2005, Oracle. All rights reserved.
DBV-00102: File I/O error on FILE (/dev/rdata_03) during start read operation (-2)
$ dbv file=/dev/rdata_03 blocksize=8192 start=194719
DBVERIFY: Release 10.2.0.1.0 - Production on Thu Nov 7 10:52:56 2013
Copyright (c) 1982, 2005, Oracle. All rights reserved.
DBVERIFY - Verification starting : FILE = /dev/rdata_03
DBVERIFY - Verification complete
Total Pages Examined : 1085282
Total Pages Processed (Data) : 1272
Total Pages Failing (Data) : 0
Total Pages Processed (Index): 127
Total Pages Failing (Index): 0
Total Pages Processed (Other): 1082867
Total Pages Processed (Seg) : 0
Total Pages Failing (Seg) : 0
Total Pages Empty : 1016
Total Pages Marked Corrupt : 0
Total Pages Influx : 0
Highest block SCN : 1845997064 (2919.1845997064)
采用数据拼接的模式重构这个数据文件
$ dd if=/dev/zero of=/backup_dir/rdata_03_02 bs=8192 count=16343
16343+0 records in.
16343+0 records out.
$ dd if=/dev/rdata_03 of=/backup_dir/rdata_03_03 bs=8192 skip=194710
1116010+0 records in.
1116010+0 records out.
$ dd if=/dev/rdata_03 of=/backup_dir/rdata_03_01 bs=8192 count=178367
178367+0 records in.
178367+0 records out.
$ cat /backup_dir/rdata_03_01 /backup_dir/rdata_03_02 /backup_dir/rdata_03_03 > /backup_dir/rdata_03_ct
拷贝到服务器上，恢复数据
SQL> alter database rename file '/dev/rdata_03' to '/oracle/oradata/orcl/rdata_03_ct';
Database altered.
SQL> select * from v$recover_file;
no rows selected
SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.
SQL> select file_id,file_name,tablespace_name,ONLINE_STATUS from dba_data_files where online_status='OFFLINE';
FILE_ID FILE_NAME TABLESPACE_NAME ONLINE_
---------- ---------------------------------------- ------------------------------ -------
10 /oracle/oradata/orcl/xx xxx OFFLINE
35 /oracle/oradata/orcl/xx xx OFFLINE
56 /oracle/oradata/orcl/xx xxx OFFLINE
发现10,35,56数据文件offline
SQL> alter database datafile 56,35,10 online;
alter database datafile 56,35,10 online
*
ERROR at line 1:
ORA-01190: control file or data file 56 is from before the last RESETLOGS
ORA-01110: data file 56: '/oracle/oradata/orcl/rmdpt141'
SQL> recover datafile 10;
ORA-00279: change 12539662247608 generated at 11/05/2013 09:24:09 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2617_817156252.dbf
ORA-00280: change 12539662247608 for thread 1 is in sequence #2617
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 12539666248541 generated at 11/05/2013 09:45:21 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2618_817156252.dbf
ORA-00280: change 12539666248541 for thread 1 is in sequence #2618
ORA-00278: log file '/oracle/arch/1_2617_817156252.dbf' no longer needed for this recovery
Log applied.
Media recovery complete.
SQL> alter database datafile 10 online;
Database altered.
SQL> recover datafile 35;
ORA-00279: change 12539662247608 generated at 11/05/2013 09:24:09 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2617_817156252.dbf
ORA-00280: change 12539662247608 for thread 1 is in sequence #2617
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 12539666248541 generated at 11/05/2013 09:45:21 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2618_817156252.dbf
ORA-00280: change 12539666248541 for thread 1 is in sequence #2618
ORA-00278: log file '/oracle/arch/1_2617_817156252.dbf' no longer needed for this recovery
Log applied.
Media recovery complete.
SQL> alter database datafile 35;
alter database datafile 35
*
ERROR at line 1:
ORA-01916: keyword ONLINE, OFFLINE, RESIZE, AUTOEXTEND or END/DROP expected
SQL> alter database datafile 35 online ;
Database altered.
SQL> recover datafile 56;
ORA-00279: change 12539662247608 generated at 11/05/2013 09:24:09 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2617_817156252.dbf
ORA-00280: change 12539662247608 for thread 1 is in sequence #2617
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 12539666248541 generated at 11/05/2013 09:45:21 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2618_817156252.dbf
ORA-00280: change 12539666248541 for thread 1 is in sequence #2618
ORA-00278: log file '/oracle/arch/1_2617_817156252.dbf' no longer needed for this recovery
Log applied.
Media recovery complete.
SQL> alter database datafile 56;
alter database datafile 56
*
ERROR at line 1:
ORA-01916: keyword ONLINE, OFFLINE, RESIZE, AUTOEXTEND or END/DROP expected
SQL> alter database datafile 56 online;
Database altered.
SQL> alter database rename file '/dev/rlog11_1g' to '/oracle/oradata/orcl/rlog11_1g';
Database altered.
SQL> alter database rename file '/dev/rlog12_1g' to '/oracle/oradata/orcl/rlog12_1g';
alter database rename file '/dev/rlog21_1g' to '/oracle/oradata/orcl//rlog21_1g';
Database altered.
alter database rename file '/dev/rlog22_1g' to '/oracle/oradata/orcl/rlog22_1g';SQL>
Database altered.
SQL> alter database rename file '/dev/rlog31_1g' to '/oracle/oradata/orcl/rlog31_1g';
Database altered.
SQL>
alter database rename file '/dev/rlog32_1g' to '/oracle/oradata/orcl/rlog32_1g';
Database altered.
SQL>
Database altered.
SQL> recover database using backup CONTROLFILE until cancel;
ORA-00279: change 12539662247608 generated at 11/05/2013 09:24:09 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2617_817156252.dbf
ORA-00280: change 12539662247608 for thread 1 is in sequence #2617
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/oracle/oradata/orcl/rsystems_2g'
ORA-01112: media recovery not started
SQL> recover database using backup CONTROLFILE until cancel;
ORA-00279: change 12539662247608 generated at 11/05/2013 09:24:09 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2617_817156252.dbf
ORA-00280: change 12539662247608 for thread 1 is in sequence #2617
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
auto
ORA-00279: change 12539666248541 generated at 11/05/2013 09:45:21 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2618_817156252.dbf
ORA-00280: change 12539666248541 for thread 1 is in sequence #2618
ORA-00278: log file '/oracle/arch/1_2617_817156252.dbf' no longer needed for this recovery
ORA-00279: change 12539666769915 generated at 11/05/2013 09:48:47 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2619_817156252.dbf
ORA-00280: change 12539666769915 for thread 1 is in sequence #2619
ORA-00278: log file '/oracle/arch/1_2618_817156252.dbf' no longer needed for this recovery
ORA-00308: cannot open archived log '/oracle/arch/1_2619_817156252.dbf'
ORA-27037: unable to obtain file status
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 3
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 3 needs more recovery to be consistent
ORA-01110: data file 3: '/oracle/oradata/orcl/rsysaux_2g'
SQL> recover database using backup CONTROLFILE until cancel;
ORA-00279: change 12539666769915 generated at 11/05/2013 09:48:47 needed for thread 1
ORA-00289: suggestion : /oracle/arch/1_2619_817156252.dbf
ORA-00280: change 12539666769915 for thread 1 is in sequence #2619
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 3 needs more recovery to be consistent
ORA-01110: data file 3: '/oracle/oradata/orcl/rsysaux_2g'
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01194: file 3 needs more recovery to be consistent
ORA-01110: data file 3: '/oracle/oradata/orcl/rsysaux_2g'
_allow_resetlogs_corruption说明：
Oracle隐含参数，可能造成数据库中的数据不一致，某些查询如果涉及到这些不一致的数据会遇到ora-600 错误，打开数据库后建议重建数据库。
使用这个参数后，数据库在使用resetlogs的选项是才会生效，否则会忽略这个参数。
SQL> alter system set "_allow_resetlogs_corruption"=true scope=spfile;
System altered.
SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 9848225792 bytes
Fixed Size 2037000 bytes
Variable Size 1493175032 bytes
Database Buffers 8338276352 bytes
Redo Buffers 14737408 bytes
Database mounted.
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open
SQL> alter database open RESETLOGS;
Database altered.
SQL> select instance_name,status from v$Instance;
INSTANCE_NAME STATUS
---------------- ------------
orcl OPEN
创建临时文件
SQL> alter tablespace temp add tempfile '/oracle/oradata/orcl/temp01.dbf' size 512M;
alter tablespace temp add tempfile '/oracle/oradata/orcl/temp01.dbf' size 512M
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01552: cannot use system rollback segment for non-system tablespace
'GOLDENGATE'
ORA-06512: at line 993
ORA-01552: cannot use system rollback segment for non-system tablespace
'GOLDENGATE'
SQL> create temporary tablespace temp01 tempfile '/oracle/oradata/orcl/temp01.dbf' size 512M;
create temporary tablespace temp01 tempfile '/oracle/oradata/orcl/temp01.dbf' size 512M
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01552: cannot use system rollback segment for non-system tablespace
'GOLDENGATE'
ORA-06512: at line 993
ORA-01552: cannot use system rollback segment for non-system tablespace
'GOLDENGATE'
undo_management参数说明：
决定数据库使用的回滚段是否使用自动管理模式，auto表示自动管理，manual表示手动管理
auto服务器自动管理重做段的创建、分配和调整
manaul手工管理重做段的创建、分配和调整。
SQL> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
undo_management string MANUAL
undo_retention integer 900
undo_tablespace string UNDOTBS1
SQL> alter system set undo_management=auto scope=spfile;
System altered.
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 9848225792 bytes
Fixed Size 2037000 bytes
Variable Size 1493175032 bytes
Database Buffers 8338276352 bytes
Redo Buffers 14737408 bytes
Database mounted.
Database opened.
SQL> create temporary tablespace temp01 tempfile '/oracle/oradata/orcl/temp01.dbf' size 1G;
Tablespace created.
查看损坏数据块中的数据
SQL> select owner,SEGMENT_NAME,SEGMENT_TYPE from dba_extents where file_id=71 and block_id between 178367 and 194710;
OWNER SEGMENT_NAME SEGMENT_TYPE
------------------------------ --------------------------------------------------------------------------------- ------------------
DZG SYS_LOB0000051457C00011$$ LOB PARTITION
DZG SYS_LOB0000051457C00011$$ LOB PARTITION
SQL> select owner,table_name from dba_lobs where SEGMENT_NAME='SYS_LOB0000051457C00011$$';
OWNER TABLE_NAME
------------------------------ ------------------------------
DZG DZG_FILEARC