记 一次dg断档恢复案例

一、运行环境

操作系统：HPUNIX

数据库版本：11.2.0.3

数据库配置：RAC

数据文件存储方式：asm

二、故障描述

生产环境主备库的数据文件存储方式为ASM，发现备库的归档文件路径磁盘空间不足，删除部分归档日志释放一些空间，但是经过排查发现重启备库后没有开启DG恢复模式，只是在接受主库传过来的归档日志，并没有在备端应用，导致备库删除归档日志文件脚本执行失败，造成归档空间不足。由于已经删除一部分没应用到备库的归档文件，导致灾备dg同步中断。

三、处理过程

由于归档丢失，备库恢复同步方式由如下两种：

1.通过主库全库备份来恢复备库。

2.通过基于SCN备份主库来恢复备库

第一种方式适合数据量较少的库，这里选择第二种方式，步骤如下：

 

• 备库确认丢失归档信息

>select  *  from  v$archive_gap

 

THREAD#   LOW_SEQUENCE#     HIGH_SEQUENCE#

----------       -------------           --------------

       2            27893                  28555

 

>select process, client_process,sequence#,status from v$managed_standby

 

PROCESS   CLIENT_P  SEQUENCE# STATUS

--------- -------- ---------- ------------

ARCH      ARCH          30775 CLOSING

ARCH      ARCH          29360 CLOSING

ARCH      ARCH          29361 CLOSING

ARCH      ARCH          30771 CLOSING

ARCH      ARCH          29357 CLOSING

ARCH      ARCH          30772 CLOSING

ARCH      ARCH          30773 CLOSING

ARCH      ARCH          29358 CLOSING

ARCH      ARCH          30774 CLOSING

ARCH      ARCH          29359 CLOSING

RFS       UNKNOWN           0 IDLE

RFS       LGWR          29362 IDLE

RFS       N/A               0 IDLE

RFS       UNKNOWN           0 IDLE

MRP0      N/A           27893 WAIT_FOR_GAP

RFS       UNKNOWN           0 IDLE

RFS       LGWR          30776 IDLE

RFS       N/A               0 IDLE

 

>select name, values from v$dataguard_stats

 

NAME                             VALUE

-------------------------------- ------------------------------

apply finish time                +00 16:52:12.9

apply lag                        +94 04:21:02

estimated startup time           22

standby has been open            N

transport lag                    +00 00:00:07

 

• 备库确认最小scn号

在备库上执行：

>alter database recover managed standby database cancel；--停止日志应用服务

>select current_scn from v$DATABASE;

 

CURRENT_SCN

----------------

     39597750709

>select min(f.checkpoint_change#) from v$datafile_header f,v$datafile d

where f.file#=d.file# and d.enabled !='READ ONLY';

 

CURRENT_SCN

----------------

     39597657793

分别查出了备库当前的scn和文件头部的最小scn，采用两者最小的scn在主库进行增量备份

 

>alter system set standby_file_management='AUTO'  SCOPE=BOTH;  --在备库端开启数据文件自动创建参数

 

在主库上执行：

select file#,to_char(creation_time,'yyyy-mm-dd hh24:mi:ss' ) creation_time,status,last_time,name from v$datafile where creation_change#>39597657793

 

FILE# CREATION_TIME     STATUS  LAST_TIME NAME

---------- --------------------  -------------- --------------------  ---- 

  151 2017-07-13 online +DATA/ebsback/datafile/apps_pts_cux6_02.dbf

  152 2017-08-18 online +DATA/ebsback/datafile/cuxd_05

  153 2017-08-18 online +DATA/ebsback/datafile/apps_pts_cux2_03

  154 2017-08-18 online +DATA/ebsback/datafile/apps_pts_cux3_03

  155 2017-08-18 online +DATA/ebsback/datafile/apps_pts_cux0.03

通过查询得知在备库最下的scn之后，主库又新增了5个数据文件 151～155

在备库上执行：

select file#,name from v$datafile;  --备库并未查得主库新增的数据文件

• 主库通过基于scn方式rman增量备份数据，备份主库的控制文件，并且把两个备份集scp到备库

RMAN>backup as compressed backupset incremental from scn 39597657793 

database  format'/备份文件_U%'

SQL>alter database create standby controlfile as '/控制文件备份.ctl'

scp 两个备份文件到备库

• 备库恢复数据库

SQL>startup mount         --备库首先启动至mount状态

RMAN>backup current controlfile format '/beifenwenjian.ctl'  --备份当前备库的控制文件

RMAN>catalog start with ‘/主库备份文件的所在备库的路径’

RMAN>recover database noredo;

• ⭐️备库使用旧的控制文件采集数据文件路径信息

备库数据文件恢复完成后，使用当前的备库控制文件查询数据库内的数据文件个数，查询结果还是150个

• 备库恢复新的控制文件

SQL>startup nomount  --启动数据库到nomount状态

RMAN>restore controlfile from '/主库控制文件的备份文件'。 

SQL>alter database mount

• 备库使用catalog datafilecopy 命令扫描数据文件

为使备库新恢复的控制文件能够有原来备库上的数据文件信息（由于启动了OMF，主库的数据文件名是不一致的），所以，需要使用⭐️步骤中收集的数据文件信息来编辑下面脚本中catalog datafilecopy 后面的信息，将备库的数据文件和名称注册到新的控制文件中。

⚠️OMF(oracle managed files) 能简化管理员的管理工作，不要指定文件的名字、大小、路径。其名字，大小，路径由oracle自动分配，可以通过DB_CREATE_FILE_DEST参数来获得。当参数值为空时表示未启动omf，若是设置为文件夹的名 则是开启omf。当主备库的数据文件存储方式为asm，并且ASM管理数据文件名称的机制为omf时，会导致主备库的文件名称不一致。

 

rman target /

catalog datafilecopy'+DATA/ebsback/datafile/system.285.837193785';

catalog datafilecopy'+DATA/ebsback/datafile/system.331.837194971';

catalog datafilecopy'+DATA/ebsback/datafile/system.317.837194443';

catalog datafilecopy'+DATA/ebsback/datafile/system.293.837193791';

catalog datafilecopy'+DATA/ebsback/datafile/system.332.837194971';

catalog datafilecopy'+DATA/ebsback/datafile/system.318.837194445';

catalog datafilecopy'+DATA/ebsback/datafile/system.294.837193797';

catalog datafilecopy'+DATA/ebsback/datafile/system.333.837194971';

catalog datafilecopy'+DATA/ebsback/datafile/system.319.837194445';

catalog datafilecopy'+DATA/ebsback/datafile/system.295.837193797';

catalog datafilecopy'+DATA/ebsback/datafile/system.300.837193803';

catalog datafilecopy'+DATA/ebsback/datafile/apps_undots1.261.837193733';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_archive.304.837193805';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_interface.306.837193807';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_media.327.837194907';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_nologging.303.837193805';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_queues.307.837193807';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_queues.308.837193807';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_seed.284.837193785';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_seed.291.837193789';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_summary.305.837193807';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.264.837193733';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.259.837193733';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.268.837193735';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.326.837194907';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.299.837193801';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.301.837193803';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.296.837193801';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.297.837193801';

catalog datafilecopy'+DATA/ebsback/datafile/ctxd.322.837194611';

catalog datafilecopy'+DATA/ebsback/datafile/odm.328.837194907';

catalog datafilecopy'+DATA/ebsback/datafile/olap.310.837193809';

catalog datafilecopy'+DATA/ebsback/datafile/owapub.313.837193811';

catalog datafilecopy'+DATA/ebsback/datafile/portal.314.837193811';

catalog datafilecopy'+DATA/ebsback/datafile/sysaux.325.837194905';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tools.311.837193809';

catalog datafilecopy'+DATA/ebsback/datafile/interim.312.837193809';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.265.837193735';

catalog datafilecopy'+DATA/ebsback/datafile/apps_undots1.286.837193787';

catalog datafilecopy'+DATA/ebsback/datafile/cuxd.323.837194903';

catalog datafilecopy'+DATA/ebsback/datafile/cuxi.260.837193733';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux0.267.837193735';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux1.263.837193733';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux2.266.837193735';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux3.269.837193735';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux4.329.837194969';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux5.315.837194441';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux6.274.837193759';

catalog datafilecopy'+DATA/ebsback/datafile/apps_pts_cux7.281.837193761';

catalog datafilecopy'+DATA/ebsback/datafile/apps_undots1.292.837193789';

catalog datafilecopy'+DATA/ebsback/datafile/undotbs2.298.837193801';

catalog datafilecopy'+DATA/ebsback/datafile/apps_undots2.273.837193757';

catalog datafilecopy'+DATA/ebsback/datafile/apps_undots2.288.837193787';

catalog datafilecopy'+DATA/ebsback/datafile/apps_undots2.287.837193787';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_interface.334.837194973';

catalog datafilecopy'+DATA/ebsback/datafile/cuxd.278.837193759';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_summary.320.837194445';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_media.302.837193803';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.321.837194607';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.309.837193807';

catalog datafilecopy'+DATA/ebsback/datafile/cuxi.282.837193763';

catalog datafilecopy'+DATA/ebsback/datafile/cuxd.324.837194905';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_data.271.837193735';

catalog datafilecopy'+DATA/ebsback/datafile/apps_ts_tx_idx.277.837193759';

• 备库使用set newname 命令修改数据文件路径

 

rman target /

run {

set newname for datafile          1 to '+DATA/ebsback/datafile/system.285.837193785';                             

set newname for datafile          2 to '+DATA/ebsback/datafile/system.331.837194971';                             

set newname for datafile          3 to '+DATA/ebsback/datafile/system.317.837194443';                             

set newname for datafile          4 to '+DATA/ebsback/datafile/system.293.837193791';                             

set newname for datafile          5 to '+DATA/ebsback/datafile/system.332.837194971';                             

set newname for datafile          6 to '+DATA/ebsback/datafile/system.318.837194445';                             

set newname for datafile          7 to '+DATA/ebsback/datafile/system.294.837193797';                             

set newname for datafile          8 to '+DATA/ebsback/datafile/system.333.837194971';                             

set newname for datafile          9 to '+DATA/ebsback/datafile/system.319.837194445';                             

set newname for datafile         10 to '+DATA/ebsback/datafile/system.295.837193797';                             

set newname for datafile         11 to '+DATA/ebsback/datafile/system.300.837193803';                             

set newname for datafile         12 to '+DATA/ebsback/datafile/apps_undots1.261.837193733';                       

set newname for datafile         13 to '+DATA/ebsback/datafile/apps_ts_archive.304.837193805';                    

set newname for datafile         14 to '+DATA/ebsback/datafile/apps_ts_interface.306.837193807';                  

set newname for datafile         15 to '+DATA/ebsback/datafile/apps_ts_media.327.837194907';                      

set newname for datafile         16 to '+DATA/ebsback/datafile/apps_ts_nologging.303.837193805';                  

set newname for datafile         17 to '+DATA/ebsback/datafile/apps_ts_queues.307.837193807';                     

set newname for datafile         18 to '+DATA/ebsback/datafile/apps_ts_queues.308.837193807';                     

set newname for datafile         19 to '+DATA/ebsback/datafile/apps_ts_seed.284.837193785';                       

set newname for datafile         20 to '+DATA/ebsback/datafile/apps_ts_seed.291.837193789';                       

set newname for datafile         21 to '+DATA/ebsback/datafile/apps_ts_summary.305.837193807';                    

set newname for datafile         22 to '+DATA/ebsback/datafile/apps_ts_tx_data.264.837193733';                    

set newname for datafile         23 to '+DATA/ebsback/datafile/apps_ts_tx_data.259.837193733';                    

set newname for datafile         24 to '+DATA/ebsback/datafile/apps_ts_tx_data.268.837193735';                    

set newname for datafile         25 to '+DATA/ebsback/datafile/apps_ts_tx_idx.326.837194907';                     

set newname for datafile         26 to '+DATA/ebsback/datafile/apps_ts_tx_idx.299.837193801';                     

set newname for datafile         27 to '+DATA/ebsback/datafile/apps_ts_tx_idx.301.837193803';                     

set newname for datafile         28 to '+DATA/ebsback/datafile/apps_ts_tx_idx.296.837193801';                     

set newname for datafile         29 to '+DATA/ebsback/datafile/apps_ts_tx_idx.297.837193801';                     

set newname for datafile         30 to '+DATA/ebsback/datafile/ctxd.322.837194611';                               

set newname for datafile         31 to '+DATA/ebsback/datafile/odm.328.837194907';                                

set newname for datafile         32 to '+DATA/ebsback/datafile/olap.310.837193809';                               

set newname for datafile         33 to '+DATA/ebsback/datafile/owapub.313.837193811';                             

set newname for datafile         34 to '+DATA/ebsback/datafile/portal.314.837193811';                             

set newname for datafile         35 to '+DATA/ebsback/datafile/sysaux.325.837194905';                             

set newname for datafile         36 to '+DATA/ebsback/datafile/apps_ts_tools.311.837193809';                      

set newname for datafile         37 to '+DATA/ebsback/datafile/interim.312.837193809';                            

set newname for datafile         38 to '+DATA/ebsback/datafile/apps_ts_tx_data.265.837193735';                    

set newname for datafile         39 to '+DATA/ebsback/datafile/apps_undots1.286.837193787';                       

set newname for datafile         40 to '+DATA/ebsback/datafile/cuxd.323.837194903';                               

set newname for datafile         41 to '+DATA/ebsback/datafile/cuxi.260.837193733';                               

set newname for datafile         42 to '+DATA/ebsback/datafile/apps_pts_cux0.267.837193735';                      

set newname for datafile         43 to '+DATA/ebsback/datafile/apps_pts_cux1.263.837193733';                      

set newname for datafile         44 to '+DATA/ebsback/datafile/apps_pts_cux2.266.837193735';                      

set newname for datafile         45 to '+DATA/ebsback/datafile/apps_pts_cux3.269.837193735';                      

set newname for datafile         46 to '+DATA/ebsback/datafile/apps_pts_cux4.329.837194969';                      

set newname for datafile         47 to '+DATA/ebsback/datafile/apps_pts_cux5.315.837194441';                      

set newname for datafile         48 to '+DATA/ebsback/datafile/apps_pts_cux6.274.837193759';                      

set newname for datafile         49 to '+DATA/ebsback/datafile/apps_pts_cux7.281.837193761';                      

set newname for datafile         50 to '+DATA/ebsback/datafile/apps_undots1.292.837193789';                       

set newname for datafile         51 to '+DATA/ebsback/datafile/undotbs2.298.837193801';                           

set newname for datafile         52 to '+DATA/ebsback/datafile/apps_undots2.273.837193757';                       

set newname for datafile         53 to '+DATA/ebsback/datafile/apps_undots2.288.837193787';                       

set newname for datafile         54 to '+DATA/ebsback/datafile/apps_undots2.287.837193787';                       

set newname for datafile         55 to '+DATA/ebsback/datafile/apps_ts_interface.334.837194973';                  

set newname for datafile         56 to '+DATA/ebsback/datafile/cuxd.278.837193759';                               

set newname for datafile         57 to '+DATA/ebsback/datafile/apps_ts_summary.320.837194445';                    

set newname for datafile         58 to '+DATA/ebsback/datafile/apps_ts_media.302.837193803';                      

set newname for datafile         59 to '+DATA/ebsback/datafile/apps_ts_tx_data.321.837194607';                    

set newname for datafile         60 to '+DATA/ebsback/datafile/apps_ts_tx_idx.309.837193807';                     

set newname for datafile         61 to '+DATA/ebsback/datafile/cuxi.282.837193763';                               

set newname for datafile         62 to '+DATA/ebsback/datafile/cuxd.324.837194905';                               

set newname for datafile         63 to '+DATA/ebsback/datafile/apps_ts_tx_data.271.837193735';

}

 

备库开启同步机制：

SQL>alter database recover managed database standby database disconnect from session;

• 主库备份缺少的数据文件

RMAN>backup datafile 后接主库新加的文件号

• 备库恢复新增加的数据文件

RMAN>catalog start with '/主库备份文件拷贝到备库的所在路径';

RMAN>restore datafile 文件号

• 开启DG同步机制

alter database recover managed standby database using current logfile disconnect from session