Oracle GoldenGate19c & Oracle GoldenGate Veridata12c 实验（一）

Oracle GoldenGate19c & Oracle GoldenGate Veridata12c 实验

  **概述**                                  

---

文档类别	测试文档
文档时间	2021-02
文档标题	对Oracle GoldenGate及Ogg Veridata功能验证（一）
作者	Ora_Ckpt_Lu

  **目的**                                  

---

1.Oracle GoldGate 19c使用集成抽取模式是否能够应用至Oracle database 11.2.0.4.0；
2.使用Oracle GoldGate 19c集成抽取、集成复制对CLOB,BLOB,LONG,XML数据类型支持程度；
3.Oracle GoldenGate veridata 数据验证工具对Oracle GoldenGate同步数据一致性校验真实性；
4.Oracle GoldenGate veridata 数据验证工具对Out-of-sync数据的修复方法可行性；

   **结果**                                 

---

1、Oracle GoldenGate 19c集成抽取模式可以适用于Oracle database 11.2.0.4.0未打补丁版本，
官方推荐如果适用集成抽取则需要打一些推荐补丁，本实验环境未应用任何数据库补丁

2、Oracle GoldenGate 19c集成抽取模式对Oracle数据库表含有CLOB,BLOB,XML数据类型是支持的，
但是对于LONG对象类型表存在同步丢失内容（丢失long列的字节长度）问题，具体原因还在继续排查

3、使用Oracle GoldenGate veridata 数据验证工具可以运行在没有Oracle
GoldenGate同步环境中，支持异构数据
数据比较，数据比对效率高，通过主键以及其他列hash比较方式进行数据一致性比较，同时支持Oracle lob之间快速比较

4、使用Oracle GoldenGate veridata 数据验证工具 可以对源与目标数据不一致的地方进行数据修复，数据修复同样适用于
Oracle lob类型，并且高效，可以用作同步时对同步失败表进行修复，但是需要做大量预先定义工作。

   **目录**                                 

---

一、环境准备

1.主机规划

2.数据准备

使用Python的Cx_Oracle模块创建相关表

#-*- coding:utf-8 -*-
import cx_Oracle
import os
import sys
os.environ['NLS_LANG']='SIMPLIFIED CHINESE_CHINA.UTF8'


def CrtTab(conn):
    cursor = conn.cursor()
    #创建BLOB表
    sqlstr1 = '''
            CREATE TABLE TAB_BLOB(
            name  VARCHAR2(24) primary key,
            content  BLOB
            )
              '''
    #创建CLOB表   
    sqlstr2 = """
            CREATE TABLE TAB_CLOB(
            name  VARCHAR2(24) primary key,
            content  CLOB
            )
              """  
      #创建LONG表   
    sqlstr3 = """
            CREATE TABLE TAB_LONG(
            name  VARCHAR2(24) primary key,
            content  LONG
            )
              """           
        #创建XML表   
    sqlstr4 = """
            CREATE TABLE TAB_XML(
            id  VARCHAR2(24) primary key,
            content  xmltype
            )
              """        
    cursor.execute(sqlstr1)
    cursor.execute(sqlstr2)
    cursor.execute(sqlstr3)
    cursor.execute(sqlstr4)
    cursor.close()            
def main():
    conn = cx_Oracle.connect('scott', 'scott', '192.168.3.103:1521/pri')
    print("已连接到数据库pri！")
    CrtTab(conn)
    print("已创建相关表！")
    conn.close()
    
if __name__ == '__main__':
    main()  

• 使用python插入原始数据

#-*- coding:utf-8 -*-
import cx_Oracle
import os
import sys
os.environ['NLS_LANG']='SIMPLIFIED CHINESE_CHINA.UTF8'





def IstTab(conn):
    cursor = conn.cursor()
    #将本地图片插入TAB_BLOB
    path = 'E:\\人社厅硬件交接\\巡检\\巡检报告\\7月2日巡检\\5500\\'
    for file in os.listdir(path):
        name = file
        content = open(path + file,'rb')
        content = content.read()
        sqlstr = "insert into tab_blob(name,content) values('%s',:blobData)" %(file)
        cursor.setinputsizes(blobData=cx_Oracle.BLOB)
        cursor.execute(sqlstr,{'blobData':content})
        cursor.execute('commit')
    #将本地文本文件插入TAB_CLOB
    path1 = 'E:\工作\手记\\'
    for file in os.listdir(path1):
        name = file
        content = open(path1 + name, 'rb')
        content = content.read()
        sqlstr = "insert into tab_clob(name,content) values('%s',:clobData)" % (file)
        cursor.setinputsizes(clobData=cx_Oracle.CLOB)
        cursor.execute(sqlstr, {'clobData': content})
        cursor.execute('commit')

    #插入long对象类型至TAB_LONG
    for i in range(1, 32):
        id = i + 1
        content = 'This_is_Long_Object.' * 1000 * i
        sqlstr = "insert into tab_LONG(name,content) values('%s',:LongData)" % (id)
        cursor.setinputsizes(LongData=cx_Oracle.LONG_STRING)
        cursor.execute(sqlstr, {'LongData': content})
        cursor.execute('commit')
    #插入Xml对象到表TAB_XML
    for i in range(1, 20):
        i = i + 1
        con = "
this_is_the_xml_type
"
        sqlstr = '''insert into tab_xml(id,content) values('%s',sys.xmlType.createXML('%s'))''' % (i, con)
        cursor.execute(sqlstr)
        cursor.execute('commit')
    cursor.close()


def main():
    conn = cx_Oracle.connect('scott', 'scott', '192.168.3.103:1521/pri')
    print('已连接到数据库！')
    IstTab(conn)
    print('已插入相关数据！')
    conn.close()

if __name__ == '__main__':
    main()

• 查看数据

3.查看受支持的表

• 根据官方文档，可以先查询DBA_GOLDENGATE_SUPPORT_MODE视图

select * from DBA_GOLDENGATE_SUPPORT_MODE where owner='SCOTT';
OWNER	OBJECT_NAME	SUPPORT_MODE
SCOTT	BONUS	         FULL
SCOTT	DEPT	         FULL
SCOTT	EMP	         FULL
SCOTT	SALGRADE	FULL
SCOTT	TAB_BLOB	FULL
SCOTT	TAB_CLOB	FULL
SCOTT	TAB_LONG	FULL
SCOTT	TAB_SEQ	        FULL
SCOTT	TAB_XML	        FULL

• 采用Integrated Capture方式，对以上表的数据同步是全支持的

二、部署Oracle GoldenGate 19c

1.先决条件

• Oracle GoldenGate_19c官方兼容矩阵
  
• 设置主机Oracle环境变量

确保将ORACLE_HOME和ORACLE_SID环境变量设置为正确的Oracle实例。连接到数据库时，Oracle GoldenGate进程会引用它们。
more /home/oracle/bash_profile
ORACLE_SID=pri; export ORACLE_SID
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; 

如果系统上有一个Oracle数据库实例，请在系统级别设置ORACLE_HOME和ORACLE_SID环境变量。如果无法以这种方式设置它们，请SETENV在将要连接到实例的每个“提取和复制”组的参数文件中使用以下语句。这些SETENV参数将覆盖系统设置，并允许Oracle
GoldenGate进程在连接到数据库时在会话级别设置变量。

SETENV (ORACLE_HOME = path_to_Oracle_home_location) SETENV (ORACLE_SID
= SID)

• 数据库要求

确保您的数据库启用了最少的补充日志记录。 在使用Oracle GoldenGate for Oracle
Database中的建立Oracle GoldenGate凭证中说明了数据库用户特权和配置要求。
如果数据库配置为使用遗留连接，则该sqlnet.ora文件必须包含该bequeath_detach=true设置。 Oracle
Databases必须处于ARCHIVELOG模式，以便Extract可以处理日志文件。 Oracle Databases必须处于FORCE
LOGGING模式下，以确保将所有事务数据都写入Redo。 Oracle
GoldenGate绑定恢复功能还需要磁盘空间。有限恢复是常规Extract检查点工具的组成部分。它以特定的时间间隔将长时间运行的未处理事务缓存到磁盘，以在重新启动Extract时实现快速恢复。在每个有界恢复间隔（由受控
BRINTERVAL的选择BR参数），所需的磁盘如下：对于每个具有缓存数据的事务，所需的磁盘空间通常为64k，再加上缓存数据的大小四舍五入为64k。并非每个长时间运行的事务都保留在磁盘上。有关有限恢复的完整信息，请参见《Oracle
GoldenGate参考》中的BR 参数。

alter system set log_archive_dest_1='location=/data/arch';
alter database force logging;
alter database add supplemental log data;
shutdown immediate;
startup mount;
alter database archivelog;
alter database open;

2.安装Oracle GoldenGate 19c

• 使用静默安装方式安装

 #准备安装包
 191004_fbo_ggs_Linux_x64_shiphome.zip
 
 #解压
 unzip  191004_fbo_ggs_Linux_x64_shiphome.zip -d ogg_tmp
 
 #创建安装目录，并指定环境变量
 mkdir /data/soft/ogg
 vi .bash_profile
 export OGG_HOME=/data/soft/ogg
 export PATH=$OGG_HOME:$PATH
 
 #编辑rsp响应文件
 [oracle@centos7 response]$ grep -v -E "^#|^$" ogg_install.rsp 
oracle.install.responseFileVersion=/oracle/install/rspfmt_ogginstall_response_schema_v19_1_0
INSTALL_OPTION=ORA11g
SOFTWARE_LOCATION=/data/soft/ogg
START_MANAGER=
MANAGER_PORT=
DATABASE_LOCATION=
INVENTORY_LOCATION=/u01/app/oraInventory
UNIX_GROUP_NAME=oinstall
[oracle@centos7 response]$ pwd
/data/soft/ogg_tmp/fbo_ggs_Linux_x64_shiphome/Disk1/response

#开始静默安装
./runInstaller -silent -nowait -responseFile /data/soft/ogg_tmp/fbo_ggs_Linux_x64_shiphome/Disk1/response/ogg_install.rsp
The installation of Oracle GoldenGate Core was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2021-01-28_05-04-31AM.log' for more details.


#验证
[oracle@centos7 ogg]$ ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Oct 17 2019 23:13:12
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.



GGSCI (centos7) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED   

                              
                                                           
#创建子目录

GGSCI (centos7) 1> create subdirs 

Creating subdirectories under current directory /data/soft/ogg

Parameter file                 /data/soft/ogg/dirprm: created.
Report file                    /data/soft/ogg/dirrpt: created.
Checkpoint file                /data/soft/ogg/dirchk: created.
Process status files           /data/soft/ogg/dirpcs: created.
SQL script files               /data/soft/ogg/dirsql: created.
Database definitions files     /data/soft/ogg/dirdef: created.
Extract data files             /data/soft/ogg/dirdat: created.
Temporary files                /data/soft/ogg/dirtmp: created.
Credential store files         /data/soft/ogg/dircrd: created.
Masterkey wallet files         /data/soft/ogg/dirwlt: created.
Dump files                     /data/soft/ogg/dirdmp: created.

目标端使用同样的方法安装

3.简单配置Oracle GoldenGate

3.1为Oracle GoldenGate准备数据库

如果要使用集成捕获和集成复制，则每个都需要tnsnames.ora文件中的专用服务器连接。（源和目标） STD =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.104)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = std)
) )

PRI = (DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.3.103)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = pri)
) )

• 源端配置数据库的最小日志以及强制日志（源）

ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
ALTER DATABASE FORCE LOGGING;

SELECT supplemental_log_data_min, force_logging FROM v$database;
YES   YES



如果使用DDL，强烈建议启用Schema-level-trandata（源）

GGSCI>dblogin userid xxxx,passwd xxxx
GGSCI>add schematrandata scott.*

• 对特定表启用Table-level-trandata（源）

GGSCI>add trandata scott.TAB_LOB
....

• 启用复制参数(源和目标）

alter system set ENABLE_GOLDENGATE_REPLICATION=true;  --如果是集群，保证每个实例都启用

• 设置用户闪回查询，并设置合适的UNDO参数（源）

By default, Oracle GoldenGate uses Flashback Query to fetch the values from the undo (rollback) tablespaces. That way, Oracle GoldenGate can reconstruct a read-consistent row image as of a specific time or SCN to match the redo record.
For best fetch results, configure the source database as follows:

alter system set UNDO_MANAGEMENT=AUTO;
alter system set UNDO_RETENTION=86400;  --设置合适大小

undo_space = UNDO_RETENTION * UPS + overhead

GRANT FLASHBACK ANY TABLE TO db_ogg_user;
GRANT FLASHBACK ON schema.table TO db_ogg_user;

• 设置流池大小（源端）

By default, one integrated capture Extract requests the logmining server to run with MAX_SGA_SIZE of 1GB. Thus, if you are running three Extracts in integrated capture mode in the same database instance, you need at least 3 GB of memory allocated to the Streams pool. As a best practice, keep 25 percent of the Streams pool available. For example, if there are 3 Extracts in integrated capture mode, set STREAMS_POOL_SIZE for the database to the following value:

alter system set streams_pool_size=150M scope=both; --这里我设置为sga的四分之一

• 保证源与目标中要同步表行的唯一性

Note:
If there are other, non-usable keys on a table or if there are no keys at all on the table, Oracle GoldenGate logs an appropriate message to the report file. Constructing a key from all of the columns impedes the performance of Oracle GoldenGate on the source system. On the target, this key causes Replicat to use a larger, less efficient WHERE clause.
If a table does not have an appropriate key, or if you prefer the existing key(s) not to be used, you can define a substitute key if the table has columns that always contain unique values. You define this substitute key by including a KEYCOLS clause within the Extract TABLE parameter and the Replicat MAP parameter. The specified key will override any existing primary or unique key that Oracle GoldenGate finds. For more information, see Reference for Oracle GoldenGate.

拟同步表最好有主键、唯一约束或者唯一索引，对于没有唯一或者主键的表，消耗抽取端大量性能，Oracle
GoldGate会自动选择，或者使用KEYCOLS指定唯一列。

对于现有表没有合适的键值，具体改造方法可以参考How to Handle Tables Without Primary Keys or
Unique Indexes With Oracle GoldenGate (Doc ID 1271578.1)

3.2为Oracle GoldenGate准备数据库用户以及登陆凭证

• 整理集成抽取所需要的数据库权限如下：

grant create session,resource,alter system,alter user ,select any transaction to ogg identified by ogg;
BEGIN  
dbms_goldengate_auth.grant_admin_privilege(grantee=>'OGG', privilege_type=>'capture',grant_select_privileges=>true, do_grants=>TRUE);
END;
/
GRANT FLASHBACK ANY TABLE TO ogg;
grant dba to ogg;

• 创建ogg用户专用表空间

create tablespace ogg datafile '/u01/app/oracle/oradata/PRI/datafile/ogg.dbf' size 50m;
alter user ogg default tablespace ogg;

• 创建登陆凭据并保存

[oracle@centos7 ogg]$ ./ggsci
GGSCI (centos7) 4> create subdirs
GGSCI (centos7) 5> ADD CREDENTIALSTORE

Credential store created.

GGSCI (centos7) 2> ALTER CREDENTIALSTORE ADD USER ogg@pri alias ogg_pri
Password: 

Credential store altered.
GGSCI (centos7) 3> dblogin useridalias ogg_pri
Successfully logged into database.

三、配置Oracle GoldGate集成抽取及复用方式

3.1配置集成抽取

• 通过配置Manager进程在源系统上创建Oracle GoldenGate实例

GGSCI (centos7) 1> edit param mgr
port 7809
DYNAMICPORTLIST 7810-7860
PURGEOLDEXTRACTS ./dirdat/*,usecheckpoints, minkeepdays 3
LAGREPORTHOURS 1
LAGINFOMINUTES 30
LAGCRITICALMINUTES 45
AUTORESTART EXTRACT *,RETRIES 15,WAITMINUTES 3,RESETMINUTES 180
AUTOSTART EXTRACT *

GGSCI (centos7) 1> start mgr

• 配置表级别trandata

dblogin useridalias ogg_pri
add trandata SCOTT.BONUS	        
add trandata SCOTT.DEPT	      
add trandata SCOTT.EMP	     
add trandata SCOTT.SALGRADE
add trandata SCOTT.TAB_BLOB	
add trandata SCOTT.TAB_CLOB	
add trandata SCOTT.TAB_LONG	
add trandata SCOTT.TAB_SEQ	        
add trandata SCOTT.TAB_XML	 

• 注册抽取进程exp1并配置

dblogin useridalias ogg_pri
GGSCI (centos7 as ogg@pri) 11> register extract exp1 database

2021-01-30 06:37:35  INFO    OGG-02003  Extract EXP1 successfully registered with database at SCN 1153220.

• 编辑配置文件

edit param exp1
Extract exp1
Useridalias ogg_pri
Exttrail ./dirdat/e1
Logallsupcols
Updaterecordformat compact
Table scott.*;

• 添加主抽取进程

add extract exp1, integrated tranlog, begin now 
add exttrail ./dirdat/e1, extract exp1, megabytes 10 

#启动出现报错 OGG-02912 Patch 17030189 is required on your Oracle mining database for trail format RELEASE 12.2 or later.

#解决方法： SQL> @prvtlmpg.plb

Oracle GoldenGate Workaround prvtlmpg

This script provides a temporary workaround for bug 17030189. It is
strongly recommended that you apply the official Oracle Patch for bug
17030189 from My Oracle Support instead of using this workaround.

This script must be executed in the mining database of Integrated
Capture. You will be prompted for the username of the mining user. Use
a double quoted identifier if the username is case sensitive or
contains special characters. In a CDB environment, this script must be
executed from the CDB$ROOT container and the mining user must be a
common user.

Enter Integrated Capture mining user: ogg

Installing workaround… No errors. No errors. No errors. Installation
completed. SQL>

• 再次启动

GGSCI (centos7) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXP1        00:00:04      00:00:06  

• 注册pmp进程并配置

GGSCI (centos7) 2> dblogin useridalias ogg_pri
Successfully logged into database.

GGSCI (centos7 as ogg@pri) 3> register extract pmp1 database
2021-01-30 06:51:39  INFO    OGG-02003  Extract PMP1 successfully registered with database at SCN 1173320.

• 编辑配置文件

Extract  pmp1
Useridalias ogg_pri
rmthost 192.168.3.104, mgrport 7809
rmttrail ./dirdat/d1
passthru
table scott.*;

• 添加进程文件

add extract pmp1, exttrailsource ./dirdat/e1
add rmttrail ./dirdat/d1, extract pmp1, megabytes 10

#目标端启动mgr
#源端启动pmp

[oracle@centos7 dirdat]$ ls -l
total 4
-rw-r----- 1 oracle oinstall 1412 Jan 30 06:47 e1000000000

[oracle@Centos7-STD dirdat]$ ls -l
total 0
-rw-r----- 1 oracle oinstall 0 Jan 30 18:38 d1000000000

3.2配置集成复制

• 注册目标端复用进程并配置

#先创建检查点
dblogin useridalias ogg_std
GGSCI (Centos7-STD as ogg@std) 2> ADD CHECKPOINTTABLE ogg.checkpoint1

Successfully created checkpoint table ogg.checkpoint1.

GGSCI (Centos7-STD as ogg@std) 4> EDIT PARAMS ./GLOBALS
CHECKPOINTTABLE ogg.checkpoint1

#注册复用进程
dblogin useridalias ogg_std
register replicat rep1 database
2021-01-30 18:49:41  INFO    OGG-02528  REPLICAT REP1 successfully registered with database as inbound server OGG$REP1.

#编辑参数文件
edit param rep1

REPLICAT rep1
DBOPTIONS INTEGRATEDPARAMS(parallelism 1)
USERIDALIAS ogg_std
ASSUMETARGETDEFS
MAP scott.* ,target tim.*;

#添加进程
Add Replicat rep1 ,Integrated,  exttrail ./dirdat/d1 ,checkpointtable ogg.checkpoint1

• 导出数据作初始化

SQL> col current_scn for 999999999999999999999
SQL> select current_scn from v$database;

           CURRENT_SCN
----------------------
               1196800
               
expdp "'/ as sysdba'" directory=d1 dumpfile=scott.dmp logfile=scott.log schemas=scott flashback_scn=1196800   

            
                        
#再去目标端导入
 impdp "'/ as sysdba'" directory=d1 dumpfile=scott.dmp remap_schema=scott:tim

• 启动复用进程

start replicat rep1 ,aftercsn 1196800