oracle 12.2.0.1 + oracle goldengate 12.3.0.1.4 之一 dml

os: centos7.4
db: oracle 12.2.0.1 + dbf
ogg:12.3.0.1.4

Oracle GoldenGate简称 ogg，主要包含Manager进程、Extract进程、Pump进程、Replicat进程。下面简单描述下，详细的可以看官方文档。

Manager进程：
GoldenGate的控制进程，它主要作用有以下几个方面：启动、监控、重启GoldenGate的其他进程，报告错误及时间，分配数据存储空间，发布阀值报告等。

Extract进程：
运行在数据库源端，负责从源端数据表或日志中捕获数据。Extract进程利用其内在的checkpoint机制，周期性地检查并记录其读写的位置，通常是写入到本地的trail文件。这种机制是为了保证如果Extract进程终止或者操作系统宕机，我们重启Extract进程后，GoldenGate能够恢复到以前的状态，从上一个断点处继续往下运行，而不会有任何数据损失。

Pump进程：
运行在数据库源端，其作用非常简单。如果源端使用了本地trail文件，那么Pump进程就会把Trail文件以数据块的形式通过TCP/IP协议发送到目标端，我们下面的配置都是这种方式。Pump进程本质是Extract进程的一种特殊形式，如果不使用Trail文件，那么Extract进程在抽取完数据后，直接投递到目标端。

Collector进程：
与Pump进程相对应的进程，它运行在目标端。这个进程不需要引起我们关注，因为在实际操作过程中无需对其进行任何配置，任务就是把Extract/Pump进程投递过来的数据块重新组装成Trail文件。

Replicat进程：
运行在目标端，是数据投递的最后一站，负责读取目标端Trail文件中的内容，并将解析其解析为DML或DDL语句，然后应用到目标数据库中。

Classic Capture is the default mode of the Oracle GoldenGate. whereas Integrated is introduced from Oracle GoldenGate 11g.
There are two types of Capture Modes in Oracle GoldenGate.

1. Classic Capture

Classic Capture Mode - The Extract process captures data changes from the Oracle redo or archive log files on the source system and writes it to the Trail files (Local Trail).

2. Integrated Capture

Integrated Capture Mode - The extract process does not directly read the redo log files. The job is done by the logmining server residing in the Oracle database. Since Integrated Capture is fully Integrated with the Database, more data types as well as compressed data are supported by this Mode.

ip规划

                source                      target
os              centos 7.4                  centos 7.5
ip              192.168.56.101              192.168.56.102
hostname        node1                       node2

db              12.2.0.1                    12.2.0.1
db_name         orcl                        orcl
db_unique_name  orcl                        orcl
instance_name   rac01                       rac0x

ogg_home        /u01/app/oracle/ogg/12.3    /u01/app/oracle/ogg/12.3
ogg admin       c##ggsadmin                 ggsadmin
obj user        peiyb                       peiyb

ogg 下载、安装

ogg 的下载要与运行的平台一致，本次下载的是 Oracle GoldenGate 12.3.0.1.4 for Oracle on Linux x86-64 (324 MB)
下载地址为：http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

使用传统的命令配置方式配置，也可以使用 Oracle GoldenGate Studio 来配置。
node1、node2 节点都需要操作

# su - oracle

$ mkdir -p /u01/app/oracle/ogg/12.3
$ vi ~/.bash_profile
#for ogg
export OGG_HOME=$ORACLE_BASE/ogg/12.3
export PATH=$PATH:$OGG_HOME
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$OGG_HOME

$ source ~/.bash_profile

$ cd /tmp
$ rz
$ unzip 123014_fbo_ggs_Linux_x64_shiphome.zip
$ cd /tmp/fbo_ggs_Linux_x64_shiphome/Disk1
$ ./runInstaller 

可以粗略看下 OGG-12.3.0.1.4-README.txt、OGG_WinUnix_Rel_Notes_12.3.0.1.4.pdf 这两个文件

创建 ogg 工作目录

node1、node2 节点都执行

GGSCI (node1) 3> create subdirs

Creating subdirectories under current directory /u01/app/oracle/ogg/12.3

Parameter file                 /u01/app/oracle/ogg/12.3/dirprm: already exists.
Report file                    /u01/app/oracle/ogg/12.3/dirrpt: already exists.
Checkpoint file                /u01/app/oracle/ogg/12.3/dirchk: already exists.
Process status files           /u01/app/oracle/ogg/12.3/dirpcs: already exists.
SQL script files               /u01/app/oracle/ogg/12.3/dirsql: already exists.
Database definitions files     /u01/app/oracle/ogg/12.3/dirdef: already exists.
Extract data files             /u01/app/oracle/ogg/12.3/dirdat: already exists.
Temporary files                /u01/app/oracle/ogg/12.3/dirtmp: already exists.
Credential store files         /u01/app/oracle/ogg/12.3/dircrd: already exists.
Masterkey wallet files         /u01/app/oracle/ogg/12.3/dirwlt: already exists.
Dump files                     /u01/app/oracle/ogg/12.3/dirdmp: already exists.

创建 c##ggsadmin 用户

source node1 节点都在cdb层级创建 c##ggsadmin 用户(方便以后操作，两个节点都创建)
源端要启动 force logging ,supplemental log, archive log.如果是生产环境配置ADG的话,这三个基本都是启用的.

supplemental log 可以参考<>,<>

# su - oracle
$ sqlplus / as sysdba
SQL> show pdbs;
SQL> alter system set enable_goldengate_replication=true;

SQL> alter database force logging;
SQL> alter database add supplemental log data;
alter database add supplemental log data (all,primary key,unique index,foreign key) columns ;
alter database add supplemental log data for procedural replication;

SQL> select * from DBA_SUPPLEMENTAL_LOGGING ;
MIN PRI UNI FOR ALL PRO SUB
--- --- --- --- --- --- ---
YES YES YES YES YES YES	NO

SQL> alter system archive log current;
SQL> select name,open_mode,force_logging,supplemental_log_data_min from v$database;

SQL> alter session set container=RAC01PDB;
SQL> create tablespace tbs_ggs datafile '/u01/app/oracle/oradata/orcl/rac01pdb/ggs001.dbf' size 500m autoextend on maxsize 30g;

创建 common user for ogg

SQL> alter session set container=CDB$ROOT;
SQL> create tablespace tbs_ggs datafile '/u01/app/oracle/oradata/orcl/ggs001.dbf' size 500m autoextend on maxsize 30g;

SQL> create user c##ggsadmin identified by ggsadminggsadmin 
default tablespace tbs_ggs;
grant dba to c##ggsadmin container=all;
grant ggsys_role to c##ggsadmin container=all;

SQL> begin 
  dbms_goldengate_auth.grant_admin_privilege('c##ggsadmin',container=>'ALL');
  commit; 
end;
/

关闭 recyclebin

SQL> alter system set recyclebin=off scope=spfile;
SQL> show parameter recyclebin

source 库需要在root容器中创建ogg管理用户，由于12c使用intergated extract，所以需要在cdb创建管理用户。
12c在cdb创建用户设置 default tablespace 时，必须确保cdb,各个pdb都要有指定的表空间，否则会报错。

target node2 节点的pdb层级创建 ggsadmin 用户(方便以后操作，两个节点都创建)
注意一个是rac01pdb，一个是rac0xpdb

SQL> alter session set container=RAC01PDB;
SQL> create user ggsadmin identified by ggsadminggsadmin 
default tablespace tbs_ggs;
grant dba to ggsadmin;
grant ggsys_role to ggsadmin;

注意:
12c架构的原因，c##ggsadmin 为cdb层面的用户(common user)，ggsadmin为pdb层面的用户(local user)。

配置 tnsname

node1、node2两个节点都配置 tnsname.ora

tns_node1_orcl =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )
  )

tns_node1_rac01pdb =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = node1)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = rac01pdb)
    )
  )

tns_node2_orcl =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = orcl)
    )
  )

tns_node2_rac0xpdb =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = node2)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = rac0xpdb)
    )
  )

source端配置

source端配置全局参数

$ ggsci
GGSCI (node1) 1> edit params ./GLOBALS

ENABLEMONITORING
CHECKPOINTTABLE CDB$ROOT.c##ggsadmin.checkpoint 
GGSCHEMA c##ggsadmin

source端配置认证证书

$ ggsci
GGSCI (node1) 1> add credentialstore
GGSCI (node1) 2> alter credentialstore add user c##ggsadmin@tns_node1_orcl,password ggsadminggsadmin alias ogg1
GGSCI (node1) 3> info credentialstore
GGSCI (node1) 4> dblogin useridalias ogg1

source端配置 manager

$ cd $OGG_HOME
$ ./ggsci
GGSCI (node1) 1> edit params mgr

PORT 7809
DYNAMICPORTLIST 7810-7890
AUTOSTART EXTRACT *
AUTORESTART EXTRACT *,RETRIES 5, WAITMINUTES 3,RESETMINUTES 5
#PURGEOLDEXTRACTS /u01/app/oracle/ogg/12.3/dirdat/*,usecheckpoints, minkeepdays 7
#LAGREPORTHOURS 1
#LAGINFOMINUTES 30
#LAGCRITICALMINUTES 45

GGSCI (node1) 2> stop mgr
GGSCI (node1) 3> start mgr
GGSCI (node1) 3> info all

source端配置 extract

创建 integrated 类型

GGSCI (node1) 4> dblogin useridalias ogg1

GGSCI (node1 as c##ggsadmin@rac01/CDB$ROOT) 4> add schematrandata RAC01PDB.peiyb
GGSCI (node1 as c##ggsadmin@rac01/CDB$ROOT) 5> add trandata RAC01PDB.peiyb.*

GGSCI (node1 as c##ggsadmin@rac01/CDB$ROOT) 4> info schematrandata RAC01PDB.peiyb
GGSCI (node1 as c##ggsadmin@rac01/CDB$ROOT) 6> info trandata RAC01PDB.peiyb.*
GGSCI (node1 as c##ggsadmin@rac01/CDB$ROOT) 7> add checkpointtable CDB$ROOT.c##ggsadmin.checkpoint 


GGSCI (node1) 1> add extract ext,integrated tranlog,begin now
GGSCI (node1) 2> add exttrail ./dirdat/ea,extract ext

或者

GGSCI (node1) 2> add exttrail ./dirdat/ea,extract ext,THREADS 2 (RAC环境使用)
GGSCI (node1) 3> dblogin useridalias ogg1
GGSCI (node1) 4> register extract ext database container (RAC01PDB) -- 提示 This EXTRACT EXT is already registered
GGSCI (node1) 5> edit params ext

EXTRACT ext
setenv (NLS_LANG=AMERICAN_AMERICA.AL32UTF8)
USERIDALIAS ogg1
TRANLOGOPTIONS DBLOGREADER     --rac时使用
FETCHOPTIONS NOUSESNAPSHOT
GETTRUNCATES
EXTTRAIL ./dirdat/ea
DISCARDFILE ./dirrpt/ea.dsc, APPEND, MEGABYTES 40
WARNLONGTRANS 1H, CHECKINTERVAL 5M
CACHEMGR CACHESIZE 1024MB, CACHEDIRECTORY ./dirtmp

LOGALLSUPCOLS
NOCOMPRESSUPDATES
UPDATERECORDFORMAT FULL
--ddl include all
--ddloptions report
REPORTCOUNT EVERY 5 SECONDS, RATE
table RAC01PDB.peiyb.*;

GGSCI (node1) 4> start extract ext 

管理extract抓取进程命令
GGSCI > start extract ext –启动ext进程
GGSCI > stop extract ext –关闭ext进程
GGSCI > kill extract ext --强制停止ext进程
GGSCI > info extract ext --查看进程参数信息
GGSCI > stats extract ext --查看进程运行信息
GGSCI > view report ext —查看运行报告

source端配置 pump

GGSCI (node1) 2> add extract pump,exttrailsource ./dirdat/ea
GGSCI (node1) 3> add rmttrail ./dirdat/pa,extract pump
GGSCI (node1) 4> edit params pump

extract pump
USERIDALIAS ogg1
rmthost 192.168.56.102,mgrport 7809,compress
rmttrail ./dirdat/pa
passthru
table RAC01PDB.peiyb.*;

管理pump投递进程

GGSCI > start extract pump –启动pump进程  
GGSCI > stop extract pump –关闭pump进程  
GGSCI > kill extract pump --强制停止pump进程  
GGSCI > info pump --查看进程参数信息  
GGSCI > stats pump --查看进程运行信息  
GGSCI > view report pump ---查看运行报告 

target端配置

必须针对每个pdb配置一个 replicat

target端配置全局参数

$ cd $OGG_HOME
$ ./ggsci
GGSCI (node2) 1> edit params ./GLOBALS

ENABLEMONITORING
CHECKPOINTTABLE ggsadmin.chkpt 
GGSCHEMA ggsadmin

target端配置认证证书

$ ./ggsci
GGSCI (node2) 1> add credentialstore
GGSCI (node2) 2> alter credentialstore add user ggsadmin@tns_node2_rac0xpdb,password ggsadminggsadmin alias ogg2
GGSCI (node2) 3> info credentialstore
GGSCI (node2) 4> dblogin useridalias ogg2

target端配置 manager

$ ./ggsci
GGSCI (node2) 1> edit params mgr

PORT 7809
DYNAMICPORTLIST 7810-7890
AUTOSTART REPLICAT *
AUTORESTART REPLICAT *,RETRIES 5, WAITMINUTES 3,RESETMINUTES 5
#PURGEOLDEXTRACTS /u01/app/oracle/ogg/12.3/dirdat/*,usecheckpoints, minkeepdays 7
#LAGREPORTHOURS 1
#LAGINFOMINUTES 30
#LAGCRITICALMINUTES 45

GGSCI (node1) 2> stop mgr
GGSCI (node1) 3> start mgr
GGSCI (node1) 3> info all

target端配置 replicat

GGSCI (node2) 21> dblogin useridalias ogg2
GGSCI (node2 as ggsadmin@rac0x/RAC0XPDB) 22> add checkpointtable ggsadmin.chkpt
GGSCI (node2 as ggsadmin@rac0x/RAC0XPDB) 22> add replicat rep,exttrail ./dirdat/pa checkpointtable ggsadmin.chkpt

GGSCI (node2 as ggsadmin@rac0x/RAC0XPDB) 23> edit params rep

REPLICAT rep
USERIDALIAS ogg2
setenv (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
HANDLECOLLISIONS
ASSUMETARGETDEFS
ddloptions report
discardfile ./dirout/RAMER.DSC, purge
MAP RAC01PDB.peiyb.*, TARGET RAC0XPDB.peiyb.*;

一些相关命令

GGSCI > start replicat rep –启动rep进程  
GGSCI > stop replicat rep –关闭rep进程  
GGSCI > kill replicat rep --强制停止e1进程  
GGSCI > info replicat rep --查看进程参数信息  
GGSCI > stats replicat rep --查看进程运行信息  
GGSCI > view report rep ---查看运行报告 

测试

source端启动相关进程

GGSCI (node1) 1> start mgr
GGSCI (node1) 2> start extract ext 
GGSCI (node1) 3> start extract pump 

GGSCI (node1) 4> info all 

target端启动相关进程

GGSCI (node2) 1> start mgr
GGSCI (node2) 2> start replicat rep

GGSCI (node2) 3> info all

数据初始化

使用 expdp、impdp初始化后，通过添加 afterscn 参数启动 rep

GGSCI (node2) 3> start replicat rep,aftercsn 7863029544

测试后发现，数据是可以正常同步，但是ddl语句无法传输。原来ddl传输还需要额外设置，在另外一篇blog介绍。

参考：
http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

https://docs.oracle.com/goldengate/c1230/gg-winux/OGGRN/toc.htm
https://docs.oracle.com/goldengate/c1230/gg-winux/index.html

https://community.oracle.com/thread/4052820
https://docs.oracle.com/goldengate/1212/gg-winux/GIORA/process_mode.htm#GIORA212

http://blog.itpub.net/8520577/viewspace-2119349/