ORA-19909与ORA-01110导致DG备库应用停止

现象分析

DG备库MRP日志应用进程停止，重启进程后马上又会挂掉。

检查告警日志中的相关告警信息，注意到错误ORA-19909和ORA-1110：

[oracle@oraclehost ~]$ tail -n 2000 $ORACLE_BASE/diag/rdbms/orcldb_2/orcldb/trace/alert_orcldb.log | grep -i mrp -A3 -B3
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 1008702072870) is orphaned on incarnation#=1
PR00 (PID:226047): MRP0: Detected orphaned datafiles! 
2023-07-23T13:54:25.806095+08:00
Errors in file /oracle/app/diag/rdbms/orcldb_2/orcldb/trace/orcldb_pr00_226047.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
--
2023-07-23T13:54:26.954298+08:00
Recovery Slave PR00 previously exited with exception 19909
2023-07-23T13:54:26.954816+08:00
Errors in file /oracle/app/diag/rdbms/orcldb_2/orcldb/trace/orcldb_mrp0_297018.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/oradata/ORCLDB_2/datafile/o1_mf_system_lcdp3t0s_.dbf'
2023-07-23T13:54:36.169316+08:00
--
 rfs (PID:42281): transaction.  This error can be ignored.  The control
 rfs (PID:42281): file transaction will be retried.
2023-07-23T13:54:46.961393+08:00
MRP0 (PID:297018): Recovery coordinator encountered one or more errors during automatic flashback on standby
2023-07-23T13:54:46.961550+08:00
Background Media Recovery process shutdown (orcldb)
2023-07-23T13:55:17.434026+08:00

手动拉起MRP进程后，MRP进程也会很快挂掉：

[oracle@oraclehost ~]$ tail -n 1000 $ORACLE_BASE/diag/rdbms/orcldb_2/orcldb/trace/alert_orcldb.log | grep -i mrp -A5 -B5
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
alter database recover managed standby database using current logfile disconnect from session
2023-07-24T09:22:59.589432+08:00
Attempt to start background Managed Standby Recovery process (orcldb)
Starting background process MRP0
2023-07-24T09:22:59.628920+08:00
MRP0 started with pid=443, OS id=350718 
2023-07-24T09:22:59.631005+08:00
Background Managed Standby Recovery process started (orcldb)
2023-07-24T09:23:04.677939+08:00
 Started logmerger process
2023-07-24T09:23:04.689215+08:00
--
PR00 (PID:350870): Managed Standby Recovery starting Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 1008702072870) is orphaned on incarnation#=1
PR00 (PID:350870): MRP0: Detected orphaned datafiles! 
2023-07-24T09:23:04.696998+08:00
Errors in file /oracle/app/diag/rdbms/orcldb_2/orcldb/trace/orcldb_pr00_350870.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/oradata/ORCLDB_2/datafile/o1_mf_system_lcdp3t0s_.dbf'
PR00 (PID:350870): Managed Standby Recovery not using Real Time Apply
--
2023-07-24T09:23:05.644643+08:00
Completed: alter database recover managed standby database using current logfile disconnect from session
2023-07-24T09:23:05.834535+08:00
Recovery Slave PR00 previously exited with exception 19909
2023-07-24T09:23:05.881092+08:00
Errors in file /oracle/app/diag/rdbms/orcldb_2/orcldb/trace/orcldb_mrp0_350718.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/oradata/ORCLDB_2/datafile/o1_mf_system_lcdp3t0s_.dbf'
2023-07-24T09:23:25.886958+08:00
MRP0 (PID:350718): Recovery coordinator encountered one or more errors during automatic flashback on standby
2023-07-24T09:23:25.887118+08:00
Background Media Recovery process shutdown (orcldb)

怀疑是当前备库曾经切换为SNAPSHOT模式，期间的某些操作导致切回物理备库模式时，切到了与主库不同的化身（INCARNATION）。

问题处理

检查主库的化身：

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       ORCLDB      3339609200       CURRENT 1          2019-08-27 11:06:24
4       4       ORCLDB      3339609200       ORPHAN  593356961202 2021-04-10 08:07:15
2       2       ORCLDB      3339609200       ORPHAN  593655142697 2021-04-10 14:57:24
3       3       ORCLDB      3339609200       ORPHAN  593735608020 2021-04-11 19:34:12

检查问题备库的化身：

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       ORCLDB      3339609200       PARENT  1          2019-08-27 11:06:24
4       4       ORCLDB      3339609200       ORPHAN  593356961202 2021-04-10 08:07:15
2       2       ORCLDB      3339609200       ORPHAN  593655142697 2021-04-10 14:57:24
3       3       ORCLDB      3339609200       ORPHAN  593735608020 2021-04-11 19:34:12
5       5       ORCLDB      3339609200       CURRENT 998409557353 2023-07-02 10:04:30

可以看到，当前（CURRENT）两者的化身不同。主库是Inc Key=1，备库是Inc Key=5。

重启备库到MOUNT状态，修改备库的化身与主库一致：

RMAN> shutdown immediate;

database closed
database dismounted
Oracle instance shut down

RMAN> startup mount;

connected to target database (not started)
Oracle instance started
database mounted

Total System Global Area  205084686320 bytes

Fixed Size                    37222384 bytes
Variable Size              28991029248 bytes
Database Buffers          175019917312 bytes
Redo Buffers                1036517376 bytes

RMAN> list incarnation;

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       ORCLDB      3339609200       PARENT  1          2019-08-27 11:06:24
4       4       ORCLDB      3339609200       ORPHAN  593356961202 2021-04-10 08:07:15
2       2       ORCLDB      3339609200       ORPHAN  593655142697 2021-04-10 14:57:24
3       3       ORCLDB      3339609200       ORPHAN  593735608020 2021-04-11 19:34:12
5       5       ORCLDB      3339609200       CURRENT 998409557353 2023-07-02 10:04:30

RMAN> reset database to incarnation 1;

database reset to incarnation 1

RMAN> list incarnation;


List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       ORCLDB      3339609200       CURRENT 1          2019-08-27 11:06:24
4       4       ORCLDB      3339609200       ORPHAN  593356961202 2021-04-10 08:07:15
2       2       ORCLDB      3339609200       ORPHAN  593655142697 2021-04-10 14:57:24
3       3       ORCLDB      3339609200       ORPHAN  593735608020 2021-04-11 19:34:12
5       5       ORCLDB      3339609200       ORPHAN  998409557353 2023-07-02 10:04:30

RMAN> alter database open;

Statement processed

RMAN> exit;

最后打开备库，观察同步状态即可。