Datax 数据抽取同步
介绍一款开源的ETL工具
https://github.com/alibaba/DataX
下载软件包：
wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
解压并赋予权限：
tar -zxvf datax.tar.gz
chmod 755 datax
同步实例：
创建表hank，并同步到表hank_bak
硬件限制，只在一个数据库实例测试功能
hank=> \d hank
Table "hank.hank"
Column | Type | Modifiers
--------+---------+-----------
a | integer |
hank=> \d hank_bak
Table "hank.hank_bak"
Column | Type | Modifiers
--------+---------+-----------
a | integer |
hank=> select * from hank;
a
----
10
10
(2 rows)
hank=> truncate hank_bak ;
TRUNCATE TABLE
进入datax目录，配置同步模板pg2pg.json
#cd /home/postgres/datax/
#ls
bin conf job lib log log_perf pg2pg.json plugin script tmp
# cat pg2pg.json
{
"job": {
"setting": {
"speed": {
"channel": 1
}
},
"content": [
{
"reader": {
"name": "postgresqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": ["jdbc:postgresql://192.168.101.115:1921/hank"],
"table": ["hank"]
}
],
"column":["a"],
"password": "hank",
"username": "hank"
}
},
"writer": {
"name": "postgresqlwriter",
"parameter": {
"username": "hank",
"password": "hank",
"column": [
"a"
],
"preSql": [
"delete from hank_bak"
],
"connection": [
{
"jdbcUrl": "jdbc:postgresql://192.168.101.115:1921/hank",
"table": [
"hank_bak"
]
}
]
}
}
}
]
}
}
进行同步
#python ./bin/datax.py pg2pg.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2017-12-04 14:22:14.376 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2017-12-04 14:22:14.387 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.151-b12
jvmInfo: Linux amd64 2.6.32-642.3.1.el6.x86_64
cpu num: 8
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2017-12-04 14:22:14.417 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"postgresqlreader",
"parameter":{
"column":[
"a"
],
"connection":[
{
"jdbcUrl":[
"jdbc:postgresql://192.168.101.115:1921/hank"
],
"table":[
"hank"
]
}
],
"password":"****",
"username":"hank"
}
},
"writer":{
"name":"postgresqlwriter",
"parameter":{
"column":[
"a"
],
"connection":[
{
"jdbcUrl":"jdbc:postgresql://192.168.101.115:1921/hank",
"table":[
"hank_bak"
]
}
],
"password":"****",
"preSql":[
"delete from hank_bak"
],
"username":"hank"
}
}
}
],
"setting":{
"speed":{
"channel":1
}
}
}
2017-12-04 14:22:14.445 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2017-12-04 14:22:14.447 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2017-12-04 14:22:14.448 [main] INFO JobContainer - DataX jobContainer starts job.
2017-12-04 14:22:14.451 [main] INFO JobContainer - Set jobId = 0
2017-12-04 14:22:14.673 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:postgresql://192.168.101.115:1921/hank.
2017-12-04 14:22:14.702 [job-0] INFO OriginalConfPretreatmentUtil - table:[hank] has columns:[a].
2017-12-04 14:22:14.750 [job-0] INFO OriginalConfPretreatmentUtil - table:[hank_bak] all columns:[
a
].
2017-12-04 14:22:14.768 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (a) VALUES(?)
], which jdbcUrl like:[jdbc:postgresql://192.168.101.115:1921/hank]
2017-12-04 14:22:14.769 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2017-12-04 14:22:14.770 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do prepare work .
2017-12-04 14:22:14.771 [job-0] INFO JobContainer - DataX Writer.Job [postgresqlwriter] do prepare work .
2017-12-04 14:22:14.778 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[delete from hank_bak]. context info:jdbc:postgresql://192.168.101.115:1921/hank.
2017-12-04 14:22:14.780 [job-0] INFO JobContainer - jobContainer starts to do split ...
2017-12-04 14:22:14.780 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2017-12-04 14:22:14.787 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] splits to [1] tasks.
2017-12-04 14:22:14.788 [job-0] INFO JobContainer - DataX Writer.Job [postgresqlwriter] splits to [1] tasks.
2017-12-04 14:22:14.820 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2017-12-04 14:22:14.828 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2017-12-04 14:22:14.832 [job-0] INFO JobContainer - Running by standalone Mode.
2017-12-04 14:22:14.846 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2017-12-04 14:22:14.853 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2017-12-04 14:22:14.853 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2017-12-04 14:22:14.870 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2017-12-04 14:22:14.875 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select a from hank
] jdbcUrl:[jdbc:postgresql://192.168.101.115:1921/hank].
2017-12-04 14:22:14.891 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select a from hank
] jdbcUrl:[jdbc:postgresql://192.168.101.115:1921/hank].
2017-12-04 14:22:14.972 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[104]ms
2017-12-04 14:22:14.973 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2017-12-04 14:22:24.865 [job-0] INFO StandAloneJobContainerCommunicator - Total 2 records, 4 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2017-12-04 14:22:24.866 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2017-12-04 14:22:24.867 [job-0] INFO JobContainer - DataX Writer.Job [postgresqlwriter] do post work.
2017-12-04 14:22:24.868 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do post work.
2017-12-04 14:22:24.868 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2017-12-04 14:22:24.869 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /home/postgres/datax/hook
2017-12-04 14:22:24.871 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
PS Scavenge | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s
2017-12-04 14:22:24.872 [job-0] INFO JobContainer - PerfTrace not enable!
2017-12-04 14:22:24.872 [job-0] INFO StandAloneJobContainerCommunicator - Total 2 records, 4 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2017-12-04 14:22:24.874 [job-0] INFO JobContainer -
任务启动时刻 : 2017-12-04 14:22:14
任务结束时刻 : 2017-12-04 14:22:24
任务总计耗时 : 10s
任务平均流量 : 0B/s
记录写入速度 : 0rec/s
读出记录总数 : 2
读写失败总数 : 0
hank=> select * from hank_bak ;
a
----
10
10
(2 rows)
同步完成。