oracle数据落地到gp,使用ora2pg迁移oracle数据到greenplum

安装ora2pg条件

1、perl版本5.10以上

2、安装DBD::Oracle

3、安装DBI

4、安装DBD::Pg

一、安装相关软件支持

1、安装DBI

#下载

https://metacpan.org/release/DBI

上传到服务器并解压出来DBI-1.637.tar.gz

[ root@bigdata01 ~]$ ll

total 592

-rw-r--r-- 1 gpadmin gpadmin 596423 Dec 4 15:05 DBI-1.637.tar.gz

drwxr-xr-x 2 gpadmin gpadmin 4096 Dec 1 16:12 gpAdminLogs

drwxr-xr-x 2 gpadmin gpadmin 4096 Dec 1 16:11 gpconfigs

[gpadmin@bigdata01 ~]$ tar -zxvf DBI-1.637.tar.gz

DBI-1.637/

DBI-1.637/Changes

DBI-1.637/dbd_xsh.h

DBI-1.637/DBI.pm

DBI-1.637/DBI.xs

DBI-1.637/dbi_sql.h

DBI-1.637/dbilogstrip.PL

[root@bigdata01 ~]$

[root@bigdata01 ~]$ ll

total 596

drwxr-x--- 5 gpadmin gpadmin 4096 Aug 16 16:45 DBI-1.637

-rw-r--r-- 1 gpadmin gpadmin 596423 Dec 4 15:05 DBI-1.637.tar.gz

drwxr-xr-x 2 gpadmin gpadmin 4096 Dec 1 16:12 gpAdminLogs

drwxr-xr-x 2 gpadmin gpadmin 4096 Dec 1 16:11 gpconfigs

[root@bigdata01 ~]$ cd DBI-1.637

[root@bigdata01 DBI-1.637]$ perl Makefile.PL

*** Your LANG environment variable is set to 'en_US.UTF-8'

*** This may cause problems for some perl installations.

*** If you get test failures, please try again with LANG unset.

*** If that then works, please email [email protected] with details

*** including the output of 'perl -V'

…………

[root@bigdata01 DBI-1.637]$

[root@bigdata01 DBI-1.637]$ make

/usr/bin/perl -MExtUtils::Command -e 'mkpath' -- blib/lib/DBI

rm -f blib/lib/DBI/Changes.pm

cp Changes blib/lib/DBI/Changes.pm

cp Driver_xst.h blib/arch/auto/DBI/Driver_xst.h

cp lib/DBD/Proxy.pm blib/lib/DBD/Proxy.pm

cp lib/DBI/Gofer/Response.pm blib/lib/DBI/Gofer/Response.pm

cp lib/DBI/Util/_accessor.pm blib/lib/DBI/Util/_accessor.pm

cp lib/DBI/Gofer/Transport/Base.pm blib/lib/DBI/Gofer/Transport/Base.pm

cp lib/DBD/DBM.pm blib/lib/DBD/DBM.pm

cp DBIXS.h blib/arch/auto/DBI/DBIXS.h

cp dbixs_rev.pl blib/lib/dbixs_rev.pl

cp lib/DBI/Gofer/Serializer/DataDumper.pm blib/lib/DBI/Gofer/Serializer/DataDumper.pm

cp lib/DBI/Const/GetInfoType.pm blib/lib/DBI/Const/GetInfoType.pm

cp lib/DBI/DBD/Metadata.pm blib/lib/DBI/DBD/Metadata.pm

…………

[root@bigdata01 DBI-1.637]$

[root@bigdata01 DBI-1.637]$ make test

PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t

t/01basics.t .................... # --- Perl 5.010001 on x86_64-linux-thread-multi

t/01basics.t .................... ok

t/02dbidrv.t .................... ok

t/03handle.t .................... ok

t/04mods.t ...................... ok

t/05concathash.t ................ ok

t/06attrs.t ..................... ok

…………

[root@bigdata01 DBI-1.637]#

[root@bigdata01 DBI-1.637]# make install

Manifying blib/man1/dbiproxy.1

Manifying blib/man1/dbilogstrip.1

Files found in blib/arch: installing files in blib/lib into architecture dependent library tree

Installing /usr/local/lib64/perl5/auto/DBI/DBIXS.h

Installing /usr/local/lib64/perl5/auto/DBI/dbixs_rev.h

…………

2、安装DBD::Oracle

#下载

http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html

上传到服务器并安装

oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm

oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm

oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm

oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm

[root@bigdata01 ~]# ll

total 54480

drwxr-xr-x 5 sa_cluster sa_group 4096 Dec 6 15:46 DBD-Pg-1.32

-rw-r--r-- 1 root root 96233 Dec 4 15:21 DBD-Pg-1.32.tar.gz

drwxr-x--- 6 ogg games 4096 Dec 6 15:40 DBI-1.637

drwxr-xr-x 2 root root 4096 Dec 1 15:47 gpAdminLogs

-rw-r--r-- 1 root root 52826628 Dec 4 16:16 oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm

-rw-r--r-- 1 root root 606864 Dec 4 16:14 oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm

-rw-r--r-- 1 root root 1524160 Dec 4 16:28 oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm

-rw-r--r-- 1 root root 708104 Dec 4 16:28 oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm

[root@bigdata01 ~]#

[root@bigdata01 ~]# rpm -ivh oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm

Preparing... ########################################### [100%]

1:oracle-instantclient12.########################################### [100%]

[root@bigdata01 ~]# rpm -ivh oracle-instantclient12.2-devel-12.2.0.1.0-1.x86_64.rpm

Preparing... ########################################### [100%]

1:oracle-instantclient12.########################################### [100%]

[root@bigdata01 ~]# rpm -ivh oracle-instantclient12.2-jdbc-12.2.0.1.0-1.x86_64.rpm

Preparing... ########################################### [100%]

1:oracle-instantclient12.########################################### [100%]

[root@bigdata01 ~]# rpm -ivh oracle-instantclient12.2-sqlplus-12.2.0.1.0-1.x86_64.rpm

Preparing... ########################################### [100%]

1:oracle-instantclient12.########################################### [100%]

[root@bigdata01 ~]#

#下载

http://search.cpan.org/~pythian/DBD-Oracle-1.74/lib/DBD/Oracle.pm

上传到服务器并解压出来DBD-Oracle-1.74.tar.gz

[root@bigdata01 ~]# tar -zxvf DBD-Oracle-1.74.tar.gz

DBD-Oracle-1.74/

DBD-Oracle-1.74/lib/

DBD-Oracle-1.74/lib/DBD/

DBD-Oracle-1.74/lib/DBD/Oracle.pm

DBD-Oracle-1.74/lib/DBD/Oracle/

…………

[root@bigdata01 DBD-Oracle-1.74]#

[root@bigdata01 ~]# cd DBD-Oracle-1.74

[root@bigdata01 DBD-Oracle-1.74]# export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib

[root@bigdata01 DBD-Oracle-1.74]# export ORACLE_HOME=/usr/lib/oracle/12.2/client64/lib

[root@bigdata01 DBD-Oracle-1.74]# perl Makefile.PL -V 12.2

Using DBI 1.637 (for perl 5.010001 on x86_64-linux-thread-multi) installed in /usr/local/lib64/perl5/auto/DBI/

Configuring DBD::Oracle for perl 5.010001 on linux (x86_64-linux-thread-multi)

Remember to actually *READ* the README file! Especially if you have any problems.

…………

[root@bigdata01 DBD-Oracle-1.74]# make

cp lib/DBD/Oracle/Troubleshooting/Cygwin.pod blib/lib/DBD/Oracle/Troubleshooting/Cygwin.pod

cp lib/DBD/Oracle.pm blib/lib/DBD/Oracle.pm

cp Oracle.h blib/arch/auto/DBD/Oracle/Oracle.h

cp lib/DBD/Oracle/Troubleshooting/Hpux.pod blib/lib/DBD/Oracle/Troubleshooting/Hpux.pod

cp lib/DBD/Oracle/Troubleshooting/Vms.pod blib/lib/DBD/Oracle/Troubleshooting/Vms.pod

…………

[root@bigdata01 DBD-Oracle-1.74]# make test

PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t

t/000-report-versions-tiny.t .. #

#

# Generated by Dist::Zilla::Plugin::ReportVersions::Tiny v1.10

# perl: 5.010001 (wanted 5.006) on linux from /usr/bin/perl

#

# B => 1.22 (want any version)

# Carp => 1.11 (want any version)

# DBI => 1.637 (want 1.51)

# Data::Dumper => 2.124 (want any version)

# Devel::Peek => 1.04 (want any version)

…………

[root@bigdata01 DBD-Oracle-1.74]# make install

Files found in blib/arch: installing files in blib/lib into architecture dependent library tree

Installing /usr/local/lib64/perl5/auto/DBD/Oracle/mk.pm

Installing /usr/local/lib64/perl5/auto/DBD/Oracle/Oracle.h

Installing /usr/local/lib64/perl5/auto/DBD/Oracle/Oracle.so

Installing /usr/local/lib64/perl5/auto/DBD/Oracle/Oracle.bs

Installing /usr/local/lib64/perl5/auto/DBD/Oracle/ocitrace.h

Installing /usr/local/lib64/perl5/auto/DBD/Oracle/dbdimp.h

Installing /usr/local/lib64/perl5/DBD/Oracle.pm

…………

3、安装DBD::Pg

#下载

http://search.cpan.org/~rudy/DBD-Pg/Pg.pm

上传到服务器并解压出来DBD-Pg-1.32.tar.gz

[root@bigdata01 ~]# ll

total 104

-rw-r--r-- 1 root root 96233 Dec 4 15:21 DBD-Pg-1.32.tar.gz

drwxr-x--- 6 ogg games 4096 Dec 6 15:40 DBI-1.637

drwxr-xr-x 2 root root 4096 Dec 1 15:47 gpAdminLogs

[root@bigdata01 ~]# tar -zxvf DBD-Pg-1.32.tar.gz

DBD-Pg-1.32/

DBD-Pg-1.32/t/

DBD-Pg-1.32/t/02attribs.t

DBD-Pg-1.32/t/99_pod.t

DBD-Pg-1.32/t/01setup.t

DBD-Pg-1.32/t/01constants.t

DBD-Pg-1.32/t/12placeholders.t

…………

[root@bigdata01 ~]#

[root@bigdata01 ~]# cd DBD-Pg-1.32

[root@bigdata01 DBD-Pg-1.32]# perl Makefile.PL

Configuring Pg

Remember to actually read the README file!

Path to pg_config?

OS: linux

PostgreSQL version: ..

****************

WARNING! DBD::Pg no longer supports versions less than 7.2.

It is highly recommended that you upgrade PostgreSQL to a newer version.

****************

Using DBI 1.637 (for perl 5.010001 on x86_64-linux-thread-multi) installed in /usr/local/lib64/perl5/auto/DBI/

Checking if your kit is complete...

Looks good

Using DBI 1.637 (for perl 5.010001 on x86_64-linux-thread-multi) installed in /usr/local/lib64/perl5/auto/DBI/

Writing Makefile for DBD::Pg

[root@bigdata01 DBD-Pg-1.32]#

[root@bigdata01 DBD-Pg-1.32]# make

Skip blib/lib/DBD/Pg.pm (unchanged)

gcc -c -I/opt/greenplum-db-4.3.17.1/include -I/usr/local/lib64/perl5/auto/DBI -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DHAVE_PQescapeString -DHAVE_PQescapeBytea -DHAVE_PQunescapeBytea -DHAVE_PQfreemem -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -DPERL_EXTMALLOC_DEF -Dmalloc=Perl_malloc -Dfree=Perl_mfree -Drealloc=Perl_realloc -Dcalloc=Perl_calloc -DVERSION=\"1.32\" -DXS_VERSION=\"1.32\" -fPIC "-I/usr/lib64/perl5/CORE" Pg.c

[root@bigdata01 DBD-Pg-1.32]# make test

PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t

t/00basic.t ......... ok

t/01connect.t ....... Use of uninitialized value $DBD::Pg::VERSION in concatenation (.) or string at t/01connect.t line 64.

Use of uninitialized value $pgversion in concatenation (.) or string at t/01connect.t line 64.

[root@bigdata01 DBD-Pg-1.32]# make install

Files found in blib/arch: installing files in blib/lib into architecture dependent library tree

Installing /usr/local/lib64/perl5/auto/DBD/Pg/Pg.bs

Installing /usr/local/lib64/perl5/auto/DBD/Pg/Pg.so

Installing /usr/local/lib64/perl5/DBD/Pg.pm

Installing /usr/local/share/man/man3/DBD::Pg.3pm

Appending installation info to /usr/lib64/perl5/perllocal.pod

[root@bigdata01 DBD-Pg-1.32]#

二、安装ora2pg

#下载

https://github.com/darold/ora2pg/releases

ora2pg-18.2.tar.gz

[root@bigdata01 ~]# tar -zxvf ora2pg-18.2.tar.gz

ora2pg-18.2/

ora2pg-18.2/INSTALL

ora2pg-18.2/LICENSE

ora2pg-18.2/MANIFEST

ora2pg-18.2/Makefile.PL

…………

[root@bigdata01 ~]# cd ora2pg-18.2/

[root@bigdata01 ora2pg-18.2]# perl Makefile.PL

Checking if your kit is complete...

Looks good

Writing Makefile for Ora2Pg

Done...

------------------------------------------------------------------------------

Please read documentation at http://ora2pg.darold.net/ before asking for help

------------------------------------------------------------------------------

Now type: make && make install

[root@bigdata01 ora2pg-18.2]#

[root@bigdata01 ora2pg-18.2]# make && make install

cp lib/Ora2Pg.pm blib/lib/Ora2Pg.pm

cp lib/Ora2Pg/GEOM.pm blib/lib/Ora2Pg/GEOM.pm

cp lib/Ora2Pg/PLSQL.pm blib/lib/Ora2Pg/PLSQL.pm

cp lib/Ora2Pg/MySQL.pm blib/lib/Ora2Pg/MySQL.pm

cp scripts/ora2pg blib/script/ora2pg

/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg

cp scripts/ora2pg_scanner blib/script/ora2pg_scanner

/usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/ora2pg_scanner

Manifying blib/man3/ora2pg.3

Installing /usr/local/share/perl5/Ora2Pg.pm

Installing /usr/local/share/perl5/Ora2Pg/GEOM.pm

Installing /usr/local/share/perl5/Ora2Pg/MySQL.pm

Installing /usr/local/share/perl5/Ora2Pg/PLSQL.pm

Installing /usr/local/share/man/man3/ora2pg.3

Installing /usr/local/bin/ora2pg_scanner

Installing /usr/local/bin/ora2pg

Installing default configuration file (ora2pg.conf.dist) to /etc/ora2pg

Appending installation info to /usr/lib64/perl5/perllocal.pod

[root@bigdata01 ora2pg-18.2]#

#检查所有软件是否安装成功

[root@bigdata01 ora2pg-18.2]# vi list.pl

[root@bigdata01 ora2pg-18.2]# cat list.pl

#!/usr/bin/perl

use strict;

use ExtUtils::Installed;

my $inst= ExtUtils::Installed->new();

my @modules = $inst->modules();

foreach(@modules)

{

my $ver = $inst->version($_) || "???";

printf("%-12s -- %s\n", $_, $ver);

}

exit;

[root@bigdata01 ora2pg-18.2]#

[root@bigdata01 ora2pg-18.2]# perl list.pl

DBD::Oracle -- 1.74

DBD::Pg -- 1.32

DBI -- 1.637

Ora2Pg -- 18.2

Perl -- 5.10.1

[root@bigdata01 ora2pg-18.2]#

三、配置

[root@bigdata01 ora2pg]# vi /etc/ora2pg/ora2pg.conf.dist

ORACLE_HOME /usr/lib/oracle/12.2/client64/lib

ORACLE_DSN dbi:Oracle:host=xxx.xxx.xxx.xxx;sid=orcl;port=1521

ORACLE_USER xxx

ORACLE_PWD xxx

SCHEMA 数据库名

NLS_LANG AMERICAN_AMERICA.AL32UTF8

TYPE TABLE DATA

OUTPUT output.sql

[root@bigdata01 ora2pg]# ora2pg -c ora2pg.conf.dist

Can't locate Time/HiRes.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/share/perl5/Ora2Pg.pm line 33.

BEGIN failed--compilation aborted at /usr/local/share/perl5/Ora2Pg.pm line 33.

Compilation failed in require at /usr/local/bin/ora2pg line 30.

BEGIN failed--compilation aborted at /usr/local/bin/ora2pg line 30.

[root@bigdata01 ora2pg]#

报这个错是perl没有完全安装导致

下载相应版本并上传到服务器http://www.cpan.org/src/5.0/

perl-5.10.1.tar.gz

[root@bigdata01 ~]#tar -zxvf perl-5.10.1.tar.gz

[root@bigdata01 ~]#cd perl-5.10.1

[root@bigdata01 perl-5.10.1]#

[root@bigdata01 perl-5.10.1]# ./Configure -d -Dusethreads

[root@bigdata01 perl-5.10.1]# make

[root@bigdata01 perl-5.10.1]# make test

[root@bigdata01 perl-5.10.1]# make install

四、导出oracle数据

#进入/etc/ora2pg目录下

[ogg@bigdata01 ora2pg]$ pwd

/etc/ora2pg

#编辑配置文件

ORACLE_HOME /usr/lib/oracle/12.2/client64/lib

# Set Oracle database connection (datasource, user, password)

ORACLE_DSN dbi:Oracle:host=xx.xx.xx.xx;sid=xxxx;port=1521

ORACLE_USER xxxxxx

ORACLE_PWD xxxxxx

SCHEMA xxxxxxx #导出xxxxxxx用户下的表

TYPE COPY #导出的对象比如VIEW、TABLE 等，具体请参考readme

ALLOW xxxxxx #允许导出xxxxxx用户下的xxxxxx表

OUTPUT outputcopy.sql #导出的数据文件名称

[root@bigdata01 ora2pg]# ora2pg -c ora2pg.mizu.conf.dist

[========================>] 167/167 tables (100.0%) end of scanning.

[========================>] 78/78 rows (100.0%) Table advertmanagement (78 recs/sec)

[> ] 78/48491814 total rows (0.0%) - (0 sec., avg: 78 recs/sec).

[========================>] 126/126 rows (100.0%) Table auth_back_member (126 recs/sec)

[> ] 204/48491814 total rows (0.0%) - (0 sec., avg: 204 recs/sec).

[> ] 204/48491814 rows (0.0%) on total estimated data (1 sec., avg: 204 recs/sec)

[> ] 204/48491814 total rows (0.0%) - (0 sec., avg: 204 recs/sec).

[========================>] 142/142 rows (100.0%) Table auth_back_member_role (142 recs/sec)

[> ] 346/48491814 total rows (0.0%) - (0 sec., avg: 346 recs/sec).

[> ] 346/48491814 rows (0.0%) on total estimated data (1 sec., avg: 346 recs/sec)

[> ] 346/48491814 total rows (0.0%) - (0 sec., avg: 346 recs/sec).

[========================>] 1/1 rows (100.0%) Table auth_button_resource (1 recs/sec)

[> ] 347/48491814 total rows (0.0%) - (0 sec., avg: 347 recs/sec).

[> ] 347/48491814 rows (0.0%) on total estimated data (1 sec., avg: 347 recs/sec)

[> ] 347/48491814 total rows (0.0%) - (0 sec., avg: 347 recs/sec).

[========================>] 79/79 rows (100.0%) Table auth_menu_resource (79 recs/sec)

[> ] 426/48491814 total rows (0.0%) - (0 sec., avg: 426 recs/sec).

[> ] 426/48491814 rows (0.0%) on total estimated data (1 sec., avg: 426 recs/sec)

[> ] 426/48491814 total rows (0.0%) - (0 sec., avg: 426 recs/sec).

[========================>] 16/16 rows (100.0%) Table auth_role (16 recs/sec)

[> ] 442/48491814 total rows (0.0%) - (0 sec., avg: 442 recs/sec).

五、导入到greenplum

[gpadmin@bigdata01 ~]$ psql -d test

psql (8.2.15)

Type "help" for help.

test=# \i /etc/ora2pg/outputcopy.sql