11gR2 dataguard 备库文件损坏处理

1. 环境模拟：

   主库：

SQL> select DATABASE_ROLE,open_mode from v$database;
DATABASE_ROLE OPEN_MODE
---------------- --------------------
PRIMARY  READ WRITE

        备库：

SQL> select DATABASE_ROLE,open_mode from v$database;
DATABASE_ROLE OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY

2.主库中断日志传输，备库停止日志应用，然后再主库更新数据。

SQL>  ALTER SYSTEM SET log_archive_dest_state_2='reset' SCOPE=BOTH;
System altered.

alert日志可以看到，不再向备库传输日志：

ALTER SYSTEM SET log_archive_dest_state_2='reset' SCOPE=BOTH;
Wed Oct 21 10:41:05 2015
Thread 1 advanced to log sequence 355 (LGWR switch)
  Current log# 1 seq# 355 mem# 0: +DATA/phub/onlinelog/group_1.262.890480943
  Current log# 1 seq# 355 mem# 1: +DATA/phub/onlinelog/group_1.263.890480945
Wed Oct 21 10:41:05 2015
Archived Log entry 707 added for thread 1 sequence 354 ID 0x1fffdaed dest 1:
Thread 1 cannot allocate new log, sequence 356
Checkpoint not complete
  Current log# 1 seq# 355 mem# 0: +DATA/phub/onlinelog/group_1.262.890480943
  Current log# 1 seq# 355 mem# 1: +DATA/phub/onlinelog/group_1.263.890480945
Thread 1 advanced to log sequence 356 (LGWR switch)
  Current log# 2 seq# 356 mem# 0: +DATA/phub/onlinelog/group_2.264.890480945
  Current log# 2 seq# 356 mem# 1: +DATA/phub/onlinelog/group_2.265.890480945
Wed Oct 21 10:41:06 2015
Archived Log entry 708 added for thread 1 sequence 355 ID 0x1fffdaed dest 1:

备库停止日志应用：

SQL> ALTER DATABASE recover managed standby DATABASE cancel;
Database altered.
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY

测试数据：

主库，

SQL> conn scott/scott;
Connected.
SQL> select count(*) from test;
  COUNT(*)
----------
     87065

备库：

SQL> select count(*) from test;

  COUNT(*)

----------

     87065

主库更新test表数据：

SQL> delete from test where rownum<1000;
999 rows deleted.
SQL> update test set owner='SCOTT' where object_id<10000;
8812 rows updated.
SQL> insert into test select * from test;
86066 rows created.
SQL> commit;
Commit complete.

SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.

删除最新归档日志：

ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] > rm -rf thread_1_seq_359.544.893674539
ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] > rm -rf thread_1_seq_358.543.893674457
ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] > rm -rf thread_1_seq_357.542.893674453
ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] >

启用主库日志传输：

SQL> ALTER system SET log_archive_dest_state_2 = 'enable';

System altered.

启用备库日志应用：

SQL> conn / as sysdba
Connected.
SQL> ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION;
Database altered.

查看备库alert日志：

[oracle@dg trace]$ tail -f alert_MECBS.log 

Serial Media Recovery started

Managed Standby Recovery not using Real Time Apply

Waiting for all non-current ORLs to be archived...

All non-current ORLs have been archived.

Completed: ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION

Media Recovery Log +DATA/mecbs/archivelog/2015_10_21/thread_1_seq_354.606.893674715

Media Recovery Log +DATA/mecbs/archivelog/2015_10_21/thread_1_seq_355.607.893674715

Media Recovery Log +DATA/mecbs/archivelog/2015_10_21/thread_1_seq_356.605.893674715

Media Recovery Waiting for thread 1 sequence 357

Fetching gap sequence in thread 1, gap sequence 357-359

Wed Oct 21 11:01:34 2015

FAL[client]: Failed to request gap sequence

 GAP - thread 1 sequence 357-359

 DBID 536511065 branch 890484819

FAL[client]: All defined FAL servers have been attempted.

------------------------------------------------------------

Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization

parameter is defined to a value that's sufficiently large

enough to maintain adequate log switch information to resolve

archivelog gaps.

------------------------------------------------------------

解决办法：

查询主库scn：

1. SQL> SELECT SEQUENCE#,FIRST_CHANGE#,NEXT_CHANGE# FROM v$archived_log WHERE SEQUENCE# > 356 ORDER BY 1;
    SEQUENCE# FIRST_CHANGE# NEXT_CHANGE#
   ---------- ------------- ------------
          357 5501419      5501424
          358 5501424      5501430
          359 5501430      5501524
          360 5501524      5501782
          360 5501524      5501782

2)根据scn，进行rman增量备份

RMAN> backup device type disk incremental from scn 5501419 database format '/home/oracle/data_%U.bak';
Starting backup at 21-OCT-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=400 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=613 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00006 name=+DATA/phub/datafile/llc01.dbf
input datafile file number=00003 name=+DATA/phub/datafile/undotbs1.260.891340857
input datafile file number=00004 name=+DATA/phub/datafile/users.269.891340843
channel ORA_DISK_1: starting piece 1 at 21-OCT-15
channel ORA_DISK_2: starting full datafile backup set
channel ORA_DISK_2: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATA/phub/datafile/sysaux.272.891340857
input datafile file number=00007 name=+DATA/phub/datafile/idx01.dbf
input datafile file number=00001 name=+DATA/phub/datafile/system.271.891340857
input datafile file number=00005 name=+DATA/phub/datafile/example.287.891340843
channel ORA_DISK_2: starting piece 1 at 21-OCT-15
channel ORA_DISK_1: finished piece 1 at 21-OCT-15
piece handle=/home/oracle/data_5lqk8pt4_1_1.bak tag=TAG20151021T111028 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:06
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
channel ORA_DISK_2: finished piece 1 at 21-OCT-15
piece handle=/home/oracle/data_5mqk8pt4_1_1.bak tag=TAG20151021T111028 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time: 00:01:05
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 21-OCT-15
channel ORA_DISK_1: finished piece 1 at 21-OCT-15
piece handle=/home/oracle/data_5nqk8pv6_1_1.bak tag=TAG20151021T111028 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 21-OCT-15

上传备份文件到备库：

[oracle@cwogg ~]$ scp data_5* 172.16.30.228:/home/oracle/
[email protected]'s password: 
data_5lqk8pt4_1_1.bak                                                           100% 1248KB   1.2MB/s   00:00    
data_5mqk8pt4_1_1.bak                                                           100% 7128KB   7.0MB/s   00:00    
data_5nqk8pv6_1_1.bak                                                           100%   10MB  10.2MB/s   00:00

备库上进行recover：

SQL> ALTER DATABASE recover managed standby DATABASE cancel;
Database altered

[oracle@dg ~]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Wed Oct 21 11:19:03 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PHUB (DBID=536511065)

RMAN> catalog start with '/home/oracle/backup/';

using target database control file instead of recovery catalog

searching for all files that match the pattern /home/oracle/backup/

List of Files Unknown to the Database

=====================================

File Name: /home/oracle/backup/data_5lqk8pt4_1_1.bak

File Name: /home/oracle/backup/data_5mqk8pt4_1_1.bak

File Name: /home/oracle/backup/data_5nqk8pv6_1_1.bak

Do you really want to catalog the above files (enter YES or NO)? yes

cataloging files...

cataloging done

List of Cataloged Files

=======================

File Name: /home/oracle/backup/data_5lqk8pt4_1_1.bak

File Name: /home/oracle/backup/data_5mqk8pt4_1_1.bak

File Name: /home/oracle/backup/data_5nqk8pv6_1_1.bak

恢复备库：

[oracle@dg ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 21 11:22:44 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Data Mining

and Real Application Testing options

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area  835104768 bytes

Fixed Size    2257840 bytes

Variable Size  541068368 bytes

Database Buffers  289406976 bytes

Redo Buffers    2371584 bytes

Database mounted.

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Data Mining

and Real Application Testing options

[oracle@dg ~]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Wed Oct 21 11:23:39 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PHUB (DBID=536511065, not open)

RMAN> recover database noredo;

Starting recover at 21-OCT-15

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=36 device type=DISK

allocated channel: ORA_DISK_2

channel ORA_DISK_2: SID=37 device type=DISK

channel ORA_DISK_1: starting incremental datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

destination for restore of datafile 00001: +DATA/mecbs/datafile/system.259.891103927

destination for restore of datafile 00002: +DATA/mecbs/datafile/sysaux.260.891104071

destination for restore of datafile 00005: +DATA/mecbs/datafile/example.261.891104187

destination for restore of datafile 00007: +DATA/mecbs/datafile/idx.410.891688925

channel ORA_DISK_1: reading from backup piece /home/oracle/backup/data_5mqk8pt4_1_1.bak

channel ORA_DISK_2: starting incremental datafile backup set restore

channel ORA_DISK_2: specifying datafile(s) to restore from backup set

destination for restore of datafile 00003: +DATA/mecbs/datafile/undotbs1.262.891104243

destination for restore of datafile 00004: +DATA/mecbs/datafile/users.263.891104267

destination for restore of datafile 00006: +DATA/mecbs/datafile/llc.258.891103925

channel ORA_DISK_2: reading from backup piece /home/oracle/backup/data_5lqk8pt4_1_1.bak

channel ORA_DISK_1: piece handle=/home/oracle/backup/data_5mqk8pt4_1_1.bak tag=TAG20151021T111028

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:03

channel ORA_DISK_2: piece handle=/home/oracle/backup/data_5lqk8pt4_1_1.bak tag=TAG20151021T111028

channel ORA_DISK_2: restored backup piece 1

channel ORA_DISK_2: restore complete, elapsed time: 00:00:03

Finished recover at 21-OCT-15

查看备库alert日志：

Wed Oct 21 11:23:45 2015

Incremental restore complete of datafile 3 +DATA/mecbs/datafile/undotbs1.262.891104243

  checkpoint is 5502788

  last deallocation scn is 1354205

Wed Oct 21 11:23:45 2015

Incremental restore complete of datafile 5 +DATA/mecbs/datafile/example.261.891104187

  checkpoint is 5502790

  last deallocation scn is 1114995

Incremental restore complete of datafile 4 +DATA/mecbs/datafile/users.263.891104267

  checkpoint is 5502788

  last deallocation scn is 3

Incremental restore complete of datafile 1 +DATA/mecbs/datafile/system.259.891103927

  checkpoint is 5502790

  last deallocation scn is 1095967

Incremental restore complete of datafile 6 +DATA/mecbs/datafile/llc.258.891103925

  checkpoint is 5502788

  last deallocation scn is 1099825

Incremental restore complete of datafile 7 +DATA/mecbs/datafile/idx.410.891688925

  checkpoint is 5502790

Incremental restore complete of datafile 2 +DATA/mecbs/datafile/sysaux.260.891104071

  checkpoint is 5502790

  last deallocation scn is 994406

开启日志应用：

SQL> ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION;

Database altered.

主库切换日志：

SQL> ALTER system switch logfile;
System altered.
SQL> /
System altered.
SQL> SELECT MAX(al.SEQUENCE#) "Last Seq Recieved", MAX(lh.SEQUENCE#) "Last Seq Applied" FROM v$archived_log al, v$log_history lh;  
Last Seq Recieved Last Seq Applied
----------------- ----------------
      363       363

备库：

SQL> SELECT MAX(al.SEQUENCE#) "Last Seq Recieved", MAX(lh.SEQUENCE#) "Last Seq Applied" FROM v$archived_log al, v$log_history lh;  

Last Seq Recieved Last Seq Applied

----------------- ----------------

     363       356

此时备库仍然显示有gap：

Completed: ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION
Media Recovery Waiting for thread 1 sequence 357
Fetching gap sequence in thread 1, gap sequence 357-359
Wed Oct 21 11:27:29 2015
RFS[1]: Selected log 4 for thread 1 sequence 363 dbid 536511065 branch 890484819
Wed Oct 21 11:27:29 2015
Archived Log entry 333 added for thread 1 sequence 362 ID 0x1fffdaed dest 1:
Wed Oct 21 11:27:49 2015
RFS[1]: Selected log 5 for thread 1 sequence 364 dbid 536511065 branch 890484819
Wed Oct 21 11:27:49 2015
Archived Log entry 334 added for thread 1 sequence 363 ID 0x1fffdaed dest 1:
Wed Oct 21 11:27:52 2015
FAL[client]: Failed to request gap sequence
 GAP - thread 1 sequence 357-359
 DBID 536511065 branch 890484819
FAL[client]: All defined FAL servers have been attempted.
------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that's sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
------------------------------------------------------------

停止备库日志应用，在重启，

两边数据以及同步了：

SQL> archive log list;

Database log mode       Archive Mode

Automatic archival       Enabled

Archive destination       +DATA

Oldest online log sequence     365

Next log sequence to archive   0

Current log sequence       367

SQL> archive log list;

Database log mode       Archive Mode

Automatic archival       Enabled

Archive destination       +DATA

Oldest online log sequence     365

Next log sequence to archive   367

Current log sequence       367

但是备库alertlog仍然报错：找不到那3个归档文件：

FAL[client]: Failed to request gap sequence

 GAP - thread 1 sequence 357-359

 DBID 536511065 branch 890484819

FAL[client]: All defined FAL servers have been attempted.

重建备库的控制文件：

SQL> ALTER DATABASE CREATE standby controlfile AS '/tmp/standby.ctl';   
Database altered.
[oracle@cwogg tmp]$ scp standby.ctl 172.16.30.228:/home/oracle/
[email protected]'s password: 
standby.ctl                                                                     100%   10MB  10.1MB/s   00:00

[oracle@dg ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 21 11:40:43 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount;

ORACLE instance started.

Total System Global Area  835104768 bytes

Fixed Size    2257840 bytes

Variable Size  541068368 bytes

Database Buffers  289406976 bytes

Redo Buffers    2371584 bytes

RMAN> restore controlfile from '/home/oracle/standby.ctl';

Starting restore at 21-OCT-15

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=24 device type=DISK

channel ORA_DISK_1: copied control file copy

output file name=+DATA/mecbs/controlfile/control01.ctl

output file name=+DATA/mecbs/controlfile/control02.ctl

Finished restore at 21-OCT-15

RMAN> startup mount;

database is already started

database mounted

released channel: ORA_DISK_1

待续，，，