linux下ora 01110,ORA-00376/ORA-01110 故障处理办法

Oracle数据库实例启动之后发现，9号数据文件发生故障，file 9 cannot be read at this time，

ORACLE Instance ilndb2 (pid = 16) - Error 376 encountered while recovering transaction (8, 21) on object 66190.

Mon Aug 29 12:15:06 EAT 2016

Errors in file /home/oracle/product/admin/ilndb/bdump/ilndb2_smon_14168.trc:

ORA-00376: file 9 cannot be read at this time

ORA-01110: data file 9: '+ILN_DATA/ilndb/datafile/ilearn_indx1.dbf'

ORACLE Instance ilndb2 (pid = 16) - Error 376 encountered while recovering transaction (338, 42) on object 66177.

Mon Aug 29 12:15:08 EAT 2016

Errors in file /home/oracle/product/admin/ilndb/bdump/ilndb2_smon_14168.trc:

ORA-00376: file 9 cannot be read at this time

ORA-01110: data file 9: '+ILN_DATA/ilndb/datafile/ilearn_indx1.dbf'

ORACLE Instance ilndb2 (pid = 16) - Error 376 encountered while recovering transaction (339, 45) on object 66190.

Mon Aug 29 12:15:10 EAT 2016

Errors in file /home/oracle/product/admin/ilndb/bdump/ilndb2_smon_14168.trc:

ORA-00376: file 9 cannot be read at this time

ORA-01110: data file 9: '+ILN_DATA/ilndb/datafile/ilearn_indx1.dbf'

ORACLE Instance ilndb2 (pid = 16) - Error 376 encountered while recovering transaction (340, 23) on object 66190.

Mon Aug 29 12:15:11 EAT 2016

Errors in file /home/oracle/product/admin/ilndb/bdump/ilndb2_smon_14168.trc:

ORA-00376: file 9 cannot be read at this time

ORA-01110: data file 9: '+ILN_DATA/ilndb/datafile/ilearn_indx1.dbf'

ORACLE Instance ilndb2 (pid = 16) - Error 376 encountered while recovering transaction (341, 20) on object 66177.

Mon Aug 29 12:15:13 EAT 2016

Errors in file /home/oracle/product/admin/ilndb/bdump/ilndb2_smon_14168.trc:

ORA-00376: file 9 cannot be read at this time

ORA-01110: data file 9: '+ILN_DATA/ilndb/datafile/ilearn_indx1.dbf'

根据数据库日志的提示，发现9号数据发生故障，尝试介质恢复。

oracle@elndb1[ilndb2]:/home/oracle$ rman target /

Recovery Manager: Release 10.2.0.5.0 - Production on Mon Aug 29 12:13:41 2016

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: ILNDB (DBID=140433242)

RMAN> recover datafile 9;

Starting recover at 29-AUG-16

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=6481 instance=ilndb2 devtype=DISK

allocated channel: ORA_DISK_2

channel ORA_DISK_2: sid=6001 instance=ilndb2 devtype=DISK

allocated channel: ORA_DISK_3

channel ORA_DISK_3: sid=6000 instance=ilndb2 devtype=DISK

allocated channel: ORA_DISK_4

channel ORA_DISK_4: sid=5999 instance=ilndb2 devtype=DISK

starting media recovery

archive log thread 3 sequence 205311 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205311.4265.921154275

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of recover command at 08/29/2016 12:14:34

RMAN-06053: unable to perform media recovery because of missing log

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205310 lowscn 19240848472

RMAN-06102: no channel to restore a backup or copy of log thread 2 seq 147416 lowscn 19240848247

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151889 lowscn 19240848211

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151888 lowscn 19240845047

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205309 lowscn 19240844632

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205308 lowscn 19240742241

RMAN-06102: no channel to restore a backup or copy of log thread 2 seq 147415 lowscn 19240632693

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151887 lowscn 19240631896

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205307 lowscn 19240631896

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151886 lowscn 19240560360

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205306 lowscn 19240500521

RMAN-06102: no channel to restore a backup or copy of log thread 2 seq 147414 lowscn 19240439862

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151885 lowscn 19240308923

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205305 lowscn 19240301377

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151884 lowscn 19240124380

RMAN-06102: no channel to restore a backup or copy of log thread 2 seq 147413 lowscn 19240106434

RMAN-06102: no channel to restore a backup or copy of log thread 1 seq 151883 lowscn 19239991180

RMAN-06102: no channel to restore a backup or copy of log thread 3 seq 205304 lowscn 19239911439

发现缺少需要的日志，应该已经完成备份，在磁盘组中删除了。

联系备份小组，对需要的日志进行恢复，恢复到原有目录。

run {

allocate channel ch00 type 'sbt_tape';

allocate channel ch01 type 'sbt_tape';

SEND 'NSR_ENV=(NSR_SERVER=bksvr2p_svc,NSR_CLIENT=elndb1-vip)';

restore archivelog sequence between 151882 and 151890 thread 1;

restore archivelog sequence between 147312 and 147417 thread 2;

restore archivelog sequence between 205303 and 205311 thread 3;

release channel ch00;

release channel ch01;

}

日志恢复完毕之后，重新执行介质恢复

oracle@elndb1[ilndb2]:/home/oracle/product/admin/ilndb/bdump$ rman target /

Recovery Manager: Release 10.2.0.5.0 - Production on Mon Aug 29 12:53:11 2016

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: ILNDB (DBID=140433242)

RMAN> recover datafile 9;

Starting recover at 29-AUG-16

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=5707 instance=ilndb2 devtype=DISK

allocated channel: ORA_DISK_2

channel ORA_DISK_2: sid=5706 instance=ilndb2 devtype=DISK

allocated channel: ORA_DISK_3

channel ORA_DISK_3: sid=5705 instance=ilndb2 devtype=DISK

allocated channel: ORA_DISK_4

channel ORA_DISK_4: sid=5704 instance=ilndb2 devtype=DISK

starting media recovery

archive log thread 1 sequence 151883 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151883.4080.921155791

archive log thread 1 sequence 151884 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151884.2542.921155791

archive log thread 1 sequence 151885 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151885.2136.921155791

archive log thread 1 sequence 151886 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151886.2115.921155815

archive log thread 1 sequence 151887 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151887.2496.921155815

archive log thread 1 sequence 151888 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151888.2593.921155817

archive log thread 1 sequence 151889 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151889.1397.921155817

archive log thread 1 sequence 151890 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151890.2469.921155571

archive log thread 2 sequence 147413 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147413.2362.921156057

archive log thread 2 sequence 147414 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147414.1828.921156073

archive log thread 2 sequence 147415 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147415.3869.921156067

archive log thread 2 sequence 147416 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147416.2487.921156077

archive log thread 2 sequence 147417 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147417.3331.921155015

archive log thread 2 sequence 147418 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147418.4018.921155341

archive log thread 2 sequence 147419 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147419.3104.921155879

archive log thread 3 sequence 205304 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205304.1826.921156603

archive log thread 3 sequence 205305 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205305.1558.921156603

archive log thread 3 sequence 205306 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205306.1981.921156605

archive log thread 3 sequence 205307 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205307.668.921156605

archive log thread 3 sequence 205308 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205308.632.921156605

archive log thread 3 sequence 205309 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205309.714.921156623

archive log thread 3 sequence 205310 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205310.1391.921156625

archive log thread 3 sequence 205311 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205311.4265.921154275

archive log thread 3 sequence 205312 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205312.3633.921155013

archive log thread 3 sequence 205313 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205313.2490.921155571

archive log thread 3 sequence 205314 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205314.868.921156127

archive log thread 3 sequence 205315 is already on disk as file +ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205315.545.921156559

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147413.2362.921156057 thread=2 sequence=147413

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151883.4080.921155791 thread=1 sequence=151883

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205304.1826.921156603 thread=3 sequence=205304

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151884.2542.921155791 thread=1 sequence=151884

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205305.1558.921156603 thread=3 sequence=205305

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151885.2136.921155791 thread=1 sequence=151885

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147414.1828.921156073 thread=2 sequence=147414

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205306.1981.921156605 thread=3 sequence=205306

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151886.2115.921155815 thread=1 sequence=151886

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_1_seq_151887.2496.921155815 thread=1 sequence=151887

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_3_seq_205307.668.921156605 thread=3 sequence=205307

archive log filename=+ILN_DATA/ilndb/archivelog/2016_08_29/thread_2_seq_147415.3869.921156067 thread=2 sequence=147415

media recovery complete, elapsed time: 00:00:10

Finished recover at 29-AUG-16

9号数据文件介质恢复成功，查看文件状态为OFFLINE，将数据文件online

SQL> select STATUS from v$datafile where TS# =9;

STATUS

---------------------

OFFLINE

SQL>

SQL>

SQL>

SQL> alter database datafile 9 online;

Database altered.

检查数据库所有数据文件的scn是否一致

FILE_NUM FILE_NAME SCN STATUS SEQUENCE

---------- -------------------------------------------------- ---------------- ---------- ----------

1 +ILN_DATA/ilndb/datafile/system.256.7572 19241144540 8196 151891

2 +ILN_DATA/ilndb/datafile/undotbs1.258.75 19241144540 4 151891

3 +ILN_DATA/ilndb/datafile/sysaux.257.7572 19241144540 4 151891

4 +ILN_DATA/ilndb/datafile/users.259.75726 19241144540 4 151891

5 +ILN_DATA/ilndb/datafile/example.265.757 19241144540 4 151891

6 +ILN_DATA/ilndb/datafile/undotbs2.266.75 19241144540 4 151891

7 +ILN_DATA/ilndb/datafile/undotbs3.267.75 19241144540 4 151891

8 +ILN_DATA/ilndb/datafile/ilearn_data1.db 19241144540 4 151891

9 +ILN_DATA/ilndb/datafile/ilearn_indx1.db 19240106434 4 147413

10 +ILN_DATA/ilndb/datafile/ilearn_data2.db 19241144540 4 151891

11 +ILN_DATA/ilndb/datafile/ilearn_data3.db 19241144540 4 151891

12 +ILN_DATA/ilndb/datafile/ilearn_data3.db 19241144540 4 151891

13 +ILN_DATA/ilndb/datafile/ilearn_data4.db 19241144540 4 151891

14 +ILN_DATA/ilndb/datafile/ilearn_data5.db 19241144540 4 151891

15 +ILN_DATA/ilndb/datafile/ilearn_data6.db 19241144540 4 151891

16 +ILN_DATA/ilndb/datafile/ilearn_data7.db 19241144540 4 151891

17 +ILN_DATA/ilndb/datafile/ilearn_data8.db 19241144540 4 151891

18 +ILN_DATA/ilndb/datafile/ilearn_data9.db 19241144540 4 151891

19 +ILN_DATA/ilndb/datafile/ilearn_data10.d 19241144540 4 151891

20 +ILN_DATA/ilndb/datafile/ilearn_data11.d 19241144540 4 151891

21 +ILN_DATA/ilndb/datafile/ilearn_data12.d 19241144540 4 151891

22 +ILN_DATA/ilndb/datafile/ilearn_data13.d 19241144540 4 151891

23 +ILN_DATA/ilndb/datafile/ilearn_data14.d 19241144540 4 151891

24 +ILN_DATA/ilndb/datafile/ilearn_data15.d 19241144540 4 151891

25 +ILN_DATA/ilndb/datafile/ilearn_data16.d 19241144540 4 151891

26 +ILN_DATA/ilndb/datafile/ilearn_data17.d 19241144540 4 151891

27 +ILN_DATA/ilndb/datafile/ilearn_data18.d 19241144540 4 151891

28 +ILN_DATA/ilndb/datafile/ilearn_data19.d 19241144540 4 151891

29 +ILN_DATA/ilndb/datafile/ilearn_data20.d 19241144540 4 151891

30 +ILN_DATA/ilndb/datafile/undotbs4.dbf 19241144540 4 151891

31 +ILN_DATA/ilndb/datafile/undotbs5.dbf 19241144540 4 151891

32 +ILN_DATA/ilndb/datafile/undotbs6.dbf 19241144540 4 151891

33 +ILN_DATA/ilndb/datafile/ilearn_data21.d 19241144540 4 151891

34 +ILN_DATA/ilndb/datafile/ilearn_data22.d 19241144540 4 151891

35 +ILN_DATA/ilndb/datafile/ilearn_data23.d 19241144540 4 151891

36 +ILN_DATA/ilndb/datafile/ilearn_data24.d 19241144540 4 151891

发现刚刚恢复的9号数据文件SCN与其他数据文件SCN不一致，手动触发检查点更新

alter system checkpoint GLOBAL;

再次检查数据文件的SCN一致性，发现已全部更新。

FILE_NUM FILE_NAME SCN STATUS SEQUENCE

---------- -------------------------------------------------- ---------------- ---------- ----------

1 +ILN_DATA/ilndb/datafile/system.256.7572 19241541927 8196 147421

2 +ILN_DATA/ilndb/datafile/undotbs1.258.75 19241541927 4 147421

3 +ILN_DATA/ilndb/datafile/sysaux.257.7572 19241541927 4 147421

4 +ILN_DATA/ilndb/datafile/users.259.75726 19241541927 4 147421

5 +ILN_DATA/ilndb/datafile/example.265.757 19241541927 4 147421

6 +ILN_DATA/ilndb/datafile/undotbs2.266.75 19241541927 4 147421

7 +ILN_DATA/ilndb/datafile/undotbs3.267.75 19241541927 4 147421

8 +ILN_DATA/ilndb/datafile/ilearn_data1.db 19241541927 4 147421

9 +ILN_DATA/ilndb/datafile/ilearn_indx1.db 19241541927 4 147421

10 +ILN_DATA/ilndb/datafile/ilearn_data2.db 19241541927 4 147421

11 +ILN_DATA/ilndb/datafile/ilearn_data3.db 19241541927 4 147421

12 +ILN_DATA/ilndb/datafile/ilearn_data3.db 19241541927 4 147421

13 +ILN_DATA/ilndb/datafile/ilearn_data4.db 19241541927 4 147421

14 +ILN_DATA/ilndb/datafile/ilearn_data5.db 19241541927 4 147421

15 +ILN_DATA/ilndb/datafile/ilearn_data6.db 19241541927 4 147421

16 +ILN_DATA/ilndb/datafile/ilearn_data7.db 19241541927 4 147421

17 +ILN_DATA/ilndb/datafile/ilearn_data8.db 19241541927 4 147421

18 +ILN_DATA/ilndb/datafile/ilearn_data9.db 19241541927 4 147421

19 +ILN_DATA/ilndb/datafile/ilearn_data10.d 19241541927 4 147421

20 +ILN_DATA/ilndb/datafile/ilearn_data11.d 19241541927 4 147421

21 +ILN_DATA/ilndb/datafile/ilearn_data12.d 19241541927 4 147421

22 +ILN_DATA/ilndb/datafile/ilearn_data13.d 19241541927 4 147421

23 +ILN_DATA/ilndb/datafile/ilearn_data14.d 19241541927 4 147421

24 +ILN_DATA/ilndb/datafile/ilearn_data15.d 19241541927 4 147421

25 +ILN_DATA/ilndb/datafile/ilearn_data16.d 19241541927 4 147421

26 +ILN_DATA/ilndb/datafile/ilearn_data17.d 19241541927 4 147421

27 +ILN_DATA/ilndb/datafile/ilearn_data18.d 19241541927 4 147421

28 +ILN_DATA/ilndb/datafile/ilearn_data19.d 19241541927 4 147421

29 +ILN_DATA/ilndb/datafile/ilearn_data20.d 19241541927 4 147421

30 +ILN_DATA/ilndb/datafile/undotbs4.dbf 19241541927 4 147421

31 +ILN_DATA/ilndb/datafile/undotbs5.dbf 19241541927 4 147421

32 +ILN_DATA/ilndb/datafile/undotbs6.dbf 19241541927 4 147421

33 +ILN_DATA/ilndb/datafile/ilearn_data21.d 19241541927 4 147421

34 +ILN_DATA/ilndb/datafile/ilearn_data22.d 19241541927 4 147421

35 +ILN_DATA/ilndb/datafile/ilearn_data23.d 19241541927 4 147421

36 +ILN_DATA/ilndb/datafile/ilearn_data24.d 19241541927 4 147421

至此9号数据文件恢复完成