[2021-01-04] ogg部分表初始化
--//如果replicate进程因为某些表异常（如：ORA-01403），并且表的数据差异很多，这个时候就需要对这些表重新初始化,初始化大致分为如下5个步骤
1. 停止目标端应用进程(或已经abend了)
2. 源库查询SCN号
3. 源端通过SCN号备份问题表，传送到目标端还原
4. 修改目标端进程配置，从SCN开始恢复
5. 开启OGG进程
--//测试过程
--//源端
SQL> select * from t5;
ID
----------
10
11
SQL> --//源端ogg进程
GGSCI (QXY.localdomain) 2> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING EORA_HR 00:00:00 00:00:04
EXTRACT RUNNING PORA_HR 00:00:00 00:00:05
GGSCI (QXY.localdomain) 3>
--//目标端应用进程
GGSCI (QXY1.localdomain) 1> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING RORA_HR 00:00:00 00:00:06 --//目标端T5表
SQL> select * from t5;
ID
----------
10
11
SQL> --//目标端先删除id=11的记录
SQL> delete from t5 where id = 11;
1 row deleted.
SQL> commit;
Commit complete.
SQL>
--//源端删除id=11的记录
SQL> delete from t5 where id = 11;
1 row deleted.
SQL> commit;
Commit complete.
SQL>--//源端ogg进程
GGSCI (QXY.localdomain) 3> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
EXTRACT RUNNING EORA_HR 00:00:00 00:00:05
EXTRACT RUNNING PORA_HR 00:00:00 00:00:03
GGSCI (QXY.localdomain) 4> --//发现目标端的ogg进程还是正常运行
GGSCI (QXY1.localdomain) 4> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING RORA_HR 00:00:00 00:00:06
GGSCI (QXY1.localdomain) 6> stats rora_hr
Sending STATS request to REPLICAT RORA_HR ...
Start of Statistics at 2020-12-31 20:48:39.
Replicating from POSTGRES.T5 to POSTGRES.T5:
*** Total statistics since 2020-12-31 20:40:24 ***
Total inserts 2.00
Total updates 0.00
Total deletes 0.00
Total discards 1.00 <====discard记录加1
Total operations 2.00
*** Daily statistics since 2020-12-31 20:40:24 ***
Total inserts 2.00
Total updates 0.00
Total deletes 0.00
Total discards 1.00
Total operations 2.00
*** Hourly statistics since 2020-12-31 20:40:24 ***
Total inserts 2.00
Total updates 0.00
Total deletes 0.00
Total discards 1.00
Total operations 2.00
--//查看目标端dsc日志
Oracle GoldenGate Delivery for Oracle process started, group RORA_HR discard file opened: 2020-12-31 20:40:21.009968
Current time: 2020-12-31 20:45:04
OCI Error ORA-01403: no data found, SQL
Operation failed at seqno 2 rba 2582
Discarding record on action DISCARD on error 1403
Problem replicating POSTGRES.T5 to POSTGRES.T5
Record not found
Mapping problem with delete record (target format)...
*
ID = 11
000000: 31 31 |11 |
* --//查看目标端的rep进程参数文件
GGSCI (QXY1.localdomain) 7> view params rora_hr
replicat rora_hr
setenv (ORACLE_SID=QXY2)
setenv (ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1)
setenv (NLS_LANG=AMERICAN_AMERICA.AL32UTF8)
userid goldengate,password goldengate
--handlecollisions
assumetargetdefs
discardfile ./dirrpt/rora_hr.dsc,purge
reperror default, discard <======因为配置了reperror
ddlerror default ignore
tableexclude hr.t4
map postgres.* ,target postgres.*;--//注释reperror之后，重新启动目标端的rep进程
GGSCI (QXY1.localdomain) 8> shell vi ./dirprm/rora_hr.prm
replicat rora_hr
setenv (ORACLE_SID=QXY2)
setenv (ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1)
setenv (NLS_LANG=AMERICAN_AMERICA.AL32UTF8)
userid goldengate,password goldengate
--handlecollisions
assumetargetdefs
discardfile ./dirrpt/rora_hr.dsc,purge
--reperror default, discard <======注释reperror
ddlerror default ignore
tableexclude hr.t4
map postgres.* ,target postgres.*;
--//重新启动
GGSCI (QXY1.localdomain) 10> stop rep rora_hr
Sending STOP request to REPLICAT RORA_HR ...
Request processed.
GGSCI (QXY1.localdomain) 12> start rep rora_hr
Sending START request to MANAGER ...
REPLICAT RORA_HR starting
GGSCI (QXY1.localdomain) 13>
GGSCI (QXY1.localdomain) 14> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING RORA_HR 00:00:00 00:00:05
GGSCI (QXY1.localdomain) 15>
--//源端和目标端T5表还剩下id=10的记录
--//首先在目标端删除id=5的记录
SQL> delete from t5 where id = 10;
1 row deleted.
SQL> commit;
Commit complete.
SQL> --//源端删除id=10的记录
SQL> delete from t5 where id = 10;
1 row deleted.
SQL> commit;
Commit complete.
SQL>
--//再次查看目标端的rep进程已经abend
GGSCI (QXY1.localdomain) 15> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT ABENDED RORA_HR 00:00:07 00:00:15
GGSCI (QXY1.localdomain) 16>
--//reprot
GGSCI (QXY1.localdomain) 16> view report rora_hr
2020-12-31 20:53:41 WARNING OGG-01004 Aborted grouped transaction on 'POSTGRES.T5', Database error 1403 (OCI Error ORA-01403: no da
ta found, SQL ).
2020-12-31 20:53:41 WARNING OGG-01003 Repositioning to rba 2705 in seqno 2.
2020-12-31 20:53:41 WARNING OGG-01154 SQL error 1403 mapping POSTGRES.T5 to POSTGRES.T5 OCI Error ORA-01403: no data found, SQL .
2020-12-31 20:53:41 WARNING OGG-01003 Repositioning to rba 2705 in seqno 2.
Source Context :
SourceModule : [er.errors]
SourceID : [/scratch/aime1/adestore/views/aime1_adc4150472/oggcore/OpenSys/src/app/er/errors.cpp]
SourceFunction : [take_rep_err_action]
SourceLine : [682]
ThreadBacktrace : [12] elements
: [/home/oracle/OGG/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f3e9464bdee]]
: [/home/oracle/OGG/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x31c) [
0x7f3e94648c2c]]
: [/home/oracle/OGG/libgglog.so(_MSG_ERR_MAP_TO_TANDEM_FAILED(CSourceContext*, ggs::gglib::ggapp::CQualDBOb
jName<(DBObjType)1> const&, ggs::gglib::ggapp::CQualDBObjName<(DBObjType)1> const&, CMessageFactory::MessageDisposition)+0x53) [0x7f3
e9463ff95]]
: [/home/oracle/OGG/replicat(take_rep_err_action(short, int, char const*, extr_ptr_def*, __std_rec_hdr*, ch
ar*, file_def*, bool)+0x175a) [0x56d2d2]]
: [/home/oracle/OGG/replicat(process_extract_loop()+0x2dd5) [0x58ee95]]
: [/home/oracle/OGG/replicat(replicat_main(int, char**)+0x8aa) [0x5b702a]]
: [/home/oracle/OGG/replicat(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x4f) [0x6778cf]]
: [/home/oracle/OGG/replicat(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thre
ad::ThreadArgs*)+0x104) [0x677b24]]
: [/home/oracle/OGG/replicat(ggs::gglib::MultiThreading::MainThread::Run(int, char**)+0x8b) [0x677ceb]]
: [/home/oracle/OGG/replicat(main+0x3f) [0x5b672f]]
: [/lib64/libc.so.6(__libc_start_main+0x100) [0x3b5281f930]]
: [/home/oracle/OGG/replicat() [0x4ff979]]
2020-12-31 20:53:41 ERROR OGG-01296 Error mapping from POSTGRES.T5 to POSTGRES.T5. --//dsc文件
[oracle@QXY1 dirrpt]$ vi rora_hr.dsc
Oracle GoldenGate Delivery for Oracle process started, group RORA_HR discard file opened: 2020-12-31 20:51:23.283775
Current time: 2020-12-31 20:53:41
Discarded record from action ABEND on error 1403
OCI Error ORA-01403: no data found, SQL
Aborting transaction on ./dirdat/pg beginning at seqno 2 rba 2705
error at seqno 2 rba 2705
Problem replicating POSTGRES.T5 to POSTGRES.T5
Record not found
Mapping problem with delete record (target format)...
*
ID = 10
000000: 31 30 |10 |
*
Process Abending : 2020-12-31 20:53:41 --//源端T5表插入9行记录
SQL> insert into t5 select level lv from dual connect by level < 10;
10 rows created.
SQL> commit;
Commit complete.
SQL> --//目标端的rep进程rora_hr还是abend
GGSCI (QXY1.localdomain) 2> info rora_hr
REPLICAT RORA_HR Last Started 2020-12-31 20:57 Status ABENDED
Checkpoint Lag 00:04:05 (updated 00:02:20 ago)
Log Read Checkpoint File ./dirdat/pg000002
2020-12-31 20:53:34.452114 RBA 2705
GGSCI (QXY1.localdomain) 3>
--//使用logdump查看/dirdat/pg000002文件的rba=2705的地方
[oracle@QXY1 OGG]$ ./logdump
Oracle GoldenGate Log File Dump Utility for Oracle
Version 12.1.2.0.0 17185003 OGGCORE_12.1.2.0.0_PLATFORMS_130924.1316
Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.
Logdump 199 >open ./dirdat/pg000002
Current LogTrail is /home/oracle/OGG/dirdat/pg000002
Logdump 200 > GHDR ON
Logdump 201 > DETAIL DATA
Logdump 202 > DETAIL DATA
Logdump 203 > USERTOKEN ON
Logdump 204 > GGSTOKEN ON
Logdump 205 > RECLEN length
Reclen set to 0
Logdump 206 >
Logdump 206 >
Logdump 206 >pos 2705
Reading forward from RBA 2705
Logdump 207 >n
___________________________________________________________________
Hdr-Ind : E (x45) Partition : . (x04)
UndoFlag : . (x00) BeforeAfter: B (x42)
RecLength : 10 (x000a) IO Time : 2020/12/31 20:53:34.452.114
IOType : 3 (x03) OrigNode : 255 (xff)
TransInd : . (x03) FormatType : R (x52)
SyskeyLen : 0 (x00) Incomplete : . (x00)
AuditRBA : 4 AuditPos : 733712
Continued : N (x00) RecCount : 1 (x01)
2020/12/31 20:53:34.452.114 Delete Len 10 RBA 2705
Name: POSTGRES.T5
Before Image: Partition 4 G s
0000 0006 0000 0002 3130 | ........10
Column 0 (x0000), Len 6 (x0006)
0000 0002 3130 | ....10
GGS tokens:
5200 0014 4141 4156 7744 4141 4541 4141 4271 2b41 | R...AAAVwDAAEAAABq+A
4141 0001 4c00 0007 3139 3931 3631 3036 0000 0836 | AA..L...19916106...6
2e35 2e31 3632 39 | .5.1629
Logdump 208 >
--//源端重新出事数据
--//抽取进程和投递进程不用停止
--//数据导出导入
--//源端导出数据,基于scn
SQL> col current_scn for 9999999999999999999
SQL> select current_scn from v$database;
CURRENT_SCN
--------------------
1992341
SQL>
SQL> col DIRECTORY_PATH for a80
SQL> col DIRECTORY_NAME for a30
SQL> select DIRECTORY_NAME,DIRECTORY_PATH from DBA_DIRECTORIES;
DIRECTORY_NAME DIRECTORY_PATH
------------------------------ --------------------------------------------------------------------------------
SUBDIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/schema/order_entry//2002/Sep
SS_OE_XMLDIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/schema/order_entry/
DATADUMP /u01/app/datadump
LOG_FILE_DIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/schema/log/
MEDIA_DIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/schema/product_media/
DATA_FILE_DIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/schema/sales_history/
XMLDIR /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/xml
ORACLE_OCM_CONFIG_DIR /u01/app/oracle/product/11.2.0/dbhome_1/ccr/hosts/QXY.localdomain/state
DATA_PUMP_DIR /u01/app/admin/QXY/dpdump/
ORACLE_OCM_CONFIG_DIR2 /u01/app/oracle/product/11.2.0/dbhome_1/ccr/state
10 rows selected.
SQL> --//导出T5表
cat > expdp_postgres_T5_20210104.par
userid="/ as sysdba"
directory=DATADUMP
dumpfile=expdp_postgres_T5_20210104_%u.dmp
logfile=expdp_postgres_T5_20210104.log
compression=all
cluster=n
parallel=8
CONTENT=data_only
tables=(postgres.T5)
flashback_scn=1992341
nohup expdp parfile= expdp_postgres_T5_20210104.par> expdp_postgres_T5_20210104.par.out &
[oracle@QXY ~]$ nohup expdp parfile= expdp_postgres_T5_20210104.par> expdp_postgres_T5_20210104.par.out &
[1] 9660
[oracle@QXY ~]$ jobs
[1]+ Running nohup expdp parfile= expdp_postgres_T5_20210104.par > expdp_postgres_T5_20210104.par.out &
[oracle@QXY ~]$ tail -100f expdp_postgres_T5_20210104.par.out &
[2] 9672
[1] Done nohup expdp parfile= expdp_postgres_T5_20210104.par > expdp_postgres_T5_20210104.par.out
[oracle@QXY ~]$
Export: Release 11.2.0.4.0 - Production on Mon Jan 4 11:00:15 2021
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Starting "SYS"."SYS_EXPORT_TABLE_01": /******** AS SYSDBA parfile=
Estimate in progress using BLOCKS method...
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 64 KB
. . exported "POSTGRES"."T5" 4.765 KB 10 rows
Master table "SYS"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SYS.SYS_EXPORT_TABLE_01 is:
/u01/app/datadump/expdp_postgres_T5_20210104_01.dmp
/u01/app/datadump/expdp_postgres_T5_20210104_02.dmp
Job "SYS"."SYS_EXPORT_TABLE_01" successfully completed at Mon Jan 4 11:00:17 2021 elapsed 0 00:00:01
--//ftp dmp文件目标库
--//查询目标的directory目录
SQL> col DIRECTORY_PATH for a80
SQL> col DIRECTORY_NAME for a30
SQL> select DIRECTORY_NAME,DIRECTORY_PATH from DBA_DIRECTORIES;
DIRECTORY_NAME DIRECTORY_PATH
------------------------------ -------------------------------------------------
SUBDIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/sche
SS_OE_XMLDIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/sche
DATADUMP /u01/app/datadump
LOG_FILE_DIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/sche
MEDIA_DIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/sche
DATA_FILE_DIR /u01/app/oracle/product/11.2.0/dbhome_1/demo/sche
XMLDIR /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/xml
ORACLE_OCM_CONFIG_DIR /u01/app/oracle/product/11.2.0/dbhome_1/ccr/hosts
DATA_PUMP_DIR /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/log
ORACLE_OCM_CONFIG_DIR2 /u01/app/oracle/product/11.2.0/dbhome_1/ccr/state
10 rows selected.
--//从源端传输到目标端
scp /u01/app/datadump/expdp_postgres_T5* [email protected]:/u01/app/datadump
[oracle@QXY ~]$ scp /u01/app/datadump/expdp_postgres_T5* [email protected]:/u01/app/datadump
expdp_postgres_T5_20210104_01.dmp 100% 12KB 12.0KB/s 00:00
expdp_postgres_T5_20210104_02.dmp 100% 20KB 20.0KB/s 00:00
[oracle@QXY ~]$
--目标端导入数据
cat > imp_postgres_T5_20210104.par
userid="/ as sysdba"
directory=DATADUMP
dumpfile=expdp_postgres_T5_20210104_%u.dmp
logfile=expdp_postgres_T5_20210104.log
cluster=n
parallel=8
content=data_only
nohup impdp parfile= imp_postgres_T5_20210104.par> imp_postgres_T5_20210104.par.out &
[oracle@QXY1 ~]$ nohup impdp parfile= imp_postgres_T5_20210104.par> imp_postgres_T5_20210104.par.out &
[1] 6758
[oracle@QXY1 ~]$ jobs
[1]+ Running nohup impdp parfile= imp_postgres_T5_20210104.par > imp_postgres_T5_20210104.par.out &
[oracle@QXY1 ~]$ tail -100f imp_postgres_T5_20210104.par.out
Import: Release 11.2.0.4.0 - Production on Thu Dec 31 21:18:29 2020
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
Master table "SYS"."SYS_IMPORT_FULL_01" successfully loaded/unloaded
Starting "SYS"."SYS_IMPORT_FULL_01": /******** AS SYSDBA parfile=
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
. . imported "POSTGRES"."T5" 4.765 KB 10 rows
Job "SYS"."SYS_IMPORT_FULL_01" successfully completed at Thu Dec 31 21:18:39 2020 elapsed 0 00:00:07--//目标库查询T5确认
[oracle@QXY1 ~]$ sqlplus postgres/postgres
SQL*Plus: Release 11.2.0.4.0 Production on Thu Dec 31 21:19:23 2020
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select * from t5;
ID
----------
1
2
3
4
5
6
7
8
9--//如果目标库的rep进程的参数不做任何修改，目标库启动ogg的rep进程还是会abend
GGSCI (QXY1.localdomain) 9> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT ABENDED RORA_HR 00:00:00 00:00:05
--//report日志
GGSCI (QXY1.localdomain) 7> view report rora_hr
2020-12-31 21:42:48 WARNING OGG-01004 Aborted grouped transaction on 'POSTGRES.T5', Database error 1403 (OCI Error ORA-01403: no da
ta found, SQL ).
2020-12-31 21:42:48 WARNING OGG-01003 Repositioning to rba 2705 in seqno 2.
...skipping one line
2020-12-31 21:42:48 WARNING OGG-01154 SQL error 1403 mapping POSTGRES.T5 to POSTGRES.T5 OCI Error ORA-01403: no data found, SQL .
2020-12-31 21:42:48 WARNING OGG-01003 Repositioning to rba 2705 in seqno 2. --//修改rep进程的参数，使用filter跳过初始化之前的事务
--//11版本的ogg transaction使用双引号， 12版本之后使用单引号，否则会报could not find column "transaction".
map postgres.t5, target postgres.t5, filter (@GETENV('transaction','csn') >1992341);
GGSCI (QXY1.localdomain) 1> edit params rora_hr
map postgres.t5, target postgres.t5, filter (@GETENV("transaction","csn") >1992341);
replicat rora_hr
setenv (ORACLE_SID=QXY2)
setenv (ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1)
setenv (NLS_LANG=AMERICAN_AMERICA.AL32UTF8)
userid goldengate,password goldengate
--handlecollisions
assumetargetdefs
discardfile ./dirrpt/rora_hr.dsc,purge
--reperror default, discard
ddlerror default ignore
tableexclude hr.t4
map postgres.t5, target postgres.t5, filter (@GETENV('transaction','csn') >1992341);
map postgres.* ,target postgres.*;
--//重启启动replicat进程，发现可以正常启动
GGSCI (QXY1.localdomain) 4> info all
Program Status Group Lag at Chkpt Time Since Chkpt
MANAGER RUNNING
REPLICAT RUNNING RORA_HR 00:00:00 00:00:03
GGSCI (QXY1.localdomain) 5> stats rora_hr
Sending STATS request to REPLICAT RORA_HR ...
Start of Statistics at 2020-12-31 22:29:07.
Replicating from POSTGRES.T5 to POSTGRES.T5:
*** Total statistics since 2020-12-31 22:28:56 ***
No database operations have been performed.
*** Daily statistics since 2020-12-31 22:28:56 ***
No database operations have been performed.
*** Hourly statistics since 2020-12-31 22:28:56 ***
No database operations have been performed.
*** Latest statistics since 2020-12-31 22:28:56 ***
No database operations have been performed.
End of Statistics.注意：
初始化时要禁止replicat进程启动，这样可能导致数据不一致（比如目标是因为delete from t5 where id = 10导致的abend，如果初始化的时候正好包含了id=10的记录，这个时候rep进程是能启动起来的，但是这个不是想要的结果），可以把rep进程的参数注释下或者在mgr里面把autorestart禁止。等重新初始化完成，并且rep参数已经配置了filter之后再启动。