使用OGG同步ORACLE ACTIVE DATAGUARD 数据到CLOUDERA HBASE测试

使用OGG同步ORACLE ACTIVE DATAGUARD 数据到CLOUDERA HBASE测试

环境介绍：

机器名	数据库版本	架构	OGG版本	IP	OS版本
uatdbrac1	Oracle 11.2.0.4	rac	NULL	172.16.49.191	REDHAT6.4
Uatdbrac2	Oracle 11.2.0.4	rac	NULL	172.16.49.193	REDHAT6.4
uagracdg	Oracle 11.2.0.4	ADG	OGG 12.2.0.1	172.16.49.138	REDHAT6.4
master1	Hbase	cdh5.6.0	OGG FOR BIG DATA 12.2.0.1	172.16.49.35	REDHAT6.4

测试目的：

以ORACLE ACTIVE DATAGUARD 作为数据抽取源，使用ORACLE GOLDENGATE将ORACLE数据库中变化数据同步到HBASE集群。

 

一．部署ORACLE RAC的ACTIVE DATAGUARD

  请参照之前ADG EXAMPLE 文档

 

二．在主库RAC上的准备工作

 

1. 创建GoldenGate表空间

2. 创建GoldenGate用户并授权

3. 开启force logging（做Active Datagurad时候已经配好）

4. 启用附加日志

alter database add supplemental log data;

alter system switch logfile;

select supplemental_log_data_min from v$database;

5. 启用enable_goldengate_replication

alter system set enable_goldengate_replication=true;

 

三．在ADG上部署goldengate 12.2

1. 创建golden os用户

 

2. 创建ogg软件安装目录 /goldengate/gg12c

 

3. 编辑golden用户profile

Vi .bash_profile 添加如下内容：

export ORACLE_BASE=/u01/app/oracle

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

export ORACLE_SID=uatracdg  --ACTIVE DATAGUARD ORACLE SID

export LD_LIBRARY_PATH=/goldengate/gg12c:$ORACLE_HOME/lib:$ORACLE_HOME/lib32

export OGG_HOME=/goldengate/gg12c

PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin:/goldengate/gg12c:.

export PATH

cd $OGG_HOME

 

4. 下载goldengate 12.2 for linux 64bit版本

最新软件包名 V100692-01.zip

 

5. 解压安装包

Unzip V100692-01.zip得到下列文件：

fbo_ggs_Linux_x64_shiphome

OGG-12.2.0.1.1-ReleaseNotes.pdf  

OGG-12.2.0.1-README.txt

 

6. 编辑静默安装参数文件

cd /goldengate/fbo_ggs_Linux_x64_shiphome/Disk1/response

编辑response文件，更改下列参数

INSTALL_OPTION=ORA11g

SOFTWARE_LOCATION=/goldengate/gg12c

START_MANAGER=true

MANAGER_PORT=7801

DATABASE_LOCATION=/u01/app/oracle/product/11.2.0/dbhome_1

INVENTORY_LOCATION=/goldengate/gg12c/oraInventory

UNIX_GROUP_NAME=oinstall

 

7. 静默安装OGG

cd /goldengate/fbo_ggs_Linux_x64_shiphome/Disk1

./runInstaller -silent -responseFile /goldengate/fbo_ggs_Linux_x64_shiphome/Disk1/response/oggcore.rsp

 

8.  添加表的附加日志

因为ADG是只读模式，添加附加日志需要更改数据字典，所以此操作需要在ggsci命令行中连接主库RAC进行

ggsci dblogin userid goldengate@uatrac passowrd

add trandata HADP.OGGT1

add trandata HADP.OGGT2

 

9.  添加配置extract进程

add extract ext01, tranlog, begin yyyy-MM-dd HH:mm:ss

add exttrail ./dirdat/et, extract ext01

edit param ext01

EXTRACT ext01

setenv ( NLS_LANG="SIMPLIFIED CHINESE_CHINA.ZHS16GBK" )

setenv ( ORACLE_SID=uatracdg )

userid goldengate, password golden123

exttrail ./dirdat/et

gettruncates

--dynamicresolution

discardfile ./dirrpt/discardext01.txt, append, megabytes 50

--DBOPTIONS ALLOWUNUSEDCOLUMN

--TRANLOGOPTIONS CONVERTUCS2CLOBS

EOFDELAY 3

getupdatebefores

TRANLOGOPTIONS MINEFROMACTIVEDG --FOR ADG

TABLE HADP.OGGT2;

TABLE HADP.OGGT1;

10.  添加配置PUMP进程

add extract pup1, exttrailsource ./dirdat/et

add rmttrail ./dirdat/rt, extract pup1

Edit param pup1

extract pup1

setenv ( NLS_LANG="SIMPLIFIED CHINESE_CHINA.ZHS16GBK" )

userid goldengate password golden123

rmthost 172.16.49.35, mgrport 7809, TCPBUFSIZE 5000000

rmttrail ./dirdat/rt

DYNAMICRESOLUTION

NUMFILES 3000

ALLOCFILES 200

passthru

TABLE HADP.OGGT2;

TABLE HADP.OGGT1;

 

 

四．在CDH上部署goldengate for big data 12.2

1. 创建golden os用户

2. 创建ogg软件安装目录 /goldengate/gg12c

3. 编辑golden用户profile

export JAVA_HOME=/usr/local/jdk1.7.0_80

PATH=$PATH:$HOME/bin

export PATH

export NLS_LANG=AMERICAN_AMERICA.ZHS16BGK

export PATH=$JAVA_HOME/bin:$PATH:/goldengate/gg12c:/usr/local/apache-maven-3.1.1/bin:.

export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:/goldengate/gg12c:/goldengate/gg12c/ggjava/resources/ggplugins/hdfs:.

export CLASSPATH=/goldengate/gg12c/ggjava/*:/goldengate/gg12c/dirprm/*:/goldengate/gg12c/lib/*:/etc/hadoop/conf/core-site.xml:/etc/hadoop/conf/hdfs-site.xml:/opt/cloudera/parcels/

CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/*:/

opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/client/*:/goldengate/gg12c/dirprm/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/lib/*:.

export OGG_HOME=/goldengate/gg12c

4.下载ogg for big data

最新软件包V100447-01.zip

5.安装ogg for big data

解压在安装目录即可

6.创建子目录

Ggsci > create subdirs

7.配置管理进程MGR

Edit param mgr

PORT 7809

dynamicportlist 7820-7850

8.添加配置复制进程

将/goldengate/gg12c/AdapterExamples/big-data/hbase下文件copy到$OGG_HOME/dirprm下，

编辑hbase.props

更改下列参数

gg.handler.hbase.hBaseColumnFamilyName=ID

gg.classpath=/etc/hbase/conf.cloudera.hbase:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hbase/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hbase/lib/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/client/*:/goldengate/gg12c/dirprm/*:/opt/cloudera/parcels/CDH-5.6.0-1.cdh5.6.0.p0.45/lib/hadoop/lib/*:.

 

ggsci>add replicat rhbase, nodbcheckpoint, exttrail ./dirdat/rt

ggsci>Edit param rhbase

REPLICAT rhbase

SOURCEDEFS ./dirsql/hdfs.sql

-- Trail file for this example is located in "AdapterExamples/trail" directory

-- Command to add REPLICAT

-- add replicat rhbase, exttrail AdapterExamples/trail/tr

setenv (NLS_LANG=AMERICAN_AMERICA.ZHS16GBK)

TARGETDB LIBFILE /goldengate/gg12c/libggjava.so SET property=/goldengate/gg12c/dirprm/hbase.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 10000

MAP HADP.OGGT2, TARGET HADP.OGGTEST,

RESOLVECONFLICT(INSERTROWEXISTS, (DEFAULT, OVERWRITE)),

COLMAP (ID=ID, COL1=COL1, COL2=COL2);

MAP HADP.OGGT1, TARGET HADP.OGGTEST,

RESOLVECONFLICT(INSERTROWEXISTS, (DEFAULT, OVERWRITE)),

COLMAP (ID=ID, COLA=COLA, COLB=COLB);

9.登陆HBASE创建测试表

创建namespace HADP：

hbase shell

create namespace ‘HADP’

list_namespace

describe_namespace ‘HADP’

创建表：

create ‘HADP:OGGTEST’,’ID’,’COL1’,’COL2’,’COLA’,’COLB’

查看表数据：

scan ‘HADP:OGGTEST’

10.查看OGG FOR BIG DATA日志

cd /goldengate/gg12c/dirrpt

more *hbase*.log

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/308563/viewspace-2136329/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/308563/viewspace-2136329/