使用Datax抽取ORACLE数据到MYSQL或TiDB

一，系统环境
Linux
JDK(1.8以上，推荐1.8)
Python(推荐Python2.6.X)
Apache Maven 3.x (Compile DataX)


二，软件安装
2.1 java安装
rpm -ivh jdk-8u151-linux-x64.rpm 
java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)


2.2 apache-maven安装
jar -xvf apache-maven-3.5.2-bin.zip 
加入/etc/profile环境变量
PATH=$PATH:/home/tidb/apache-maven-3.5.2/bin


2.3 查看python版本
[root@localhost ~]# python -V
Python 2.7.5


2.4 DataX安装
tar -zxvf datax.tar.gz 
chmod -R 755 datax/
[tidb@localhost bin]$ pwd
/home/tidb/datax/bin


三，DataX的使用
3.1 测试运行样例
[tidb@localhost bin]$ python datax.py ../job/job.json


DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.




2018-02-11 11:23:44.828 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2018-02-11 11:23:44.840 [main] INFO  Engine - the machine info  => 


osInfo: Oracle Corporation 1.8 25.151-b12
jvmInfo: Linux amd64 3.10.0-693.el7.x86_64
cpu num: 4


totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1


GC Names [PS MarkSweep, PS Scavenge]


MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         




2018-02-11 11:23:44.867 [main] INFO  Engine - 
{
"content":[
{
"reader":{
"name":"streamreader",
"parameter":{
"column":[
{
"type":"string",
"value":"DataX"
},
{
"type":"long",
"value":19890604
},
{
"type":"date",
"value":"1989-06-04 00:00:00"
},
{
"type":"bool",
"value":true
},
{
"type":"bytes",
"value":"test"
}
],
"sliceRecordCount":100000
}
},
"writer":{
"name":"streamwriter",
"parameter":{
"encoding":"UTF-8",
"print":false
}
}
}
],
"setting":{
"errorLimit":{
"percentage":0.02,
"record":0
},
"speed":{
"byte":10485760
}
}
}


2018-02-11 11:23:44.894 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2018-02-11 11:23:44.896 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2018-02-11 11:23:44.897 [main] INFO  JobContainer - DataX jobContainer starts job.
2018-02-11 11:23:44.900 [main] INFO  JobContainer - Set jobId = 0
2018-02-11 11:23:44.922 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2018-02-11 11:23:44.923 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do prepare work .
2018-02-11 11:23:44.923 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2018-02-11 11:23:44.924 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2018-02-11 11:23:44.925 [job-0] INFO  JobContainer - Job set Max-Byte-Speed to 10485760 bytes.
2018-02-11 11:23:44.926 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2018-02-11 11:23:44.927 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2018-02-11 11:23:44.955 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2018-02-11 11:23:44.962 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2018-02-11 11:23:44.965 [job-0] INFO  JobContainer - Running by standalone Mode.
2018-02-11 11:23:44.976 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2018-02-11 11:23:44.982 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2018-02-11 11:23:44.982 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2018-02-11 11:23:44.999 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2018-02-11 11:23:45.200 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[204]ms
2018-02-11 11:23:45.201 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2018-02-11 11:23:54.989 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.065s |  All Task WaitReaderTime 0.078s | Percentage 100.00%
2018-02-11 11:23:54.989 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2018-02-11 11:23:54.990 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2018-02-11 11:23:54.991 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
2018-02-11 11:23:54.991 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2018-02-11 11:23:54.992 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /home/tidb/datax/hook
2018-02-11 11:23:54.994 [job-0] INFO  JobContainer - 
[total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
                        


[total gc info] => 
NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             


2018-02-11 11:23:54.995 [job-0] INFO  JobContainer - PerfTrace not enable!
2018-02-11 11:23:54.995 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.065s |  All Task WaitReaderTime 0.078s | Percentage 100.00%
2018-02-11 11:23:54.997 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2018-02-11 11:23:44
任务结束时刻                    : 2018-02-11 11:23:54
任务总计耗时                    :                 10s
任务平均流量                    :          253.91KB/s
记录写入速度                    :          10000rec/s
读出记录总数                    :              100000
读写失败总数                    :                   0


3.2 测试oracle表t1同步数据到MySQL或者TiDB库的表t1
3.2.1创建表结构一样的t1表
[tidb@tidb-163 ~]$ mysql -ucqgds -h192.168.40.161 -P4000 -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 2377
Server version: 5.7.1-TiDB-v1.0.2-2-g07db8cd MySQL Community Server (Apache License 2.0)


Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.


Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> use cqgs
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
MySQL [cqgs]> create table T1
    -> (
    ->   id   INTEGER not null,
    ->   name VARCHAR(10)
    -> );
MySQL [cqgs]> select count(*) from t1;
+----------+
| count(*) |
+----------+
|        0 |
+----------+




3.2.2 查看oracle同步MySQL配置模板
[tidb@localhost bin]$ python datax.py -r oraclereader -w mysqlwriter


3.2.3 编辑oracle同步MySQL配置
[tidb@localhost bin]$ vi oracle2mysql4.json 
{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "oraclereader",
                    "parameter": {
                        "column": ["id","name"],
                        "connection": [
                            {
                                "jdbcUrl": ["jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))"],
                                "table": ["t1"]
                            }
                        ],
                        "password": "test",
                        "username": "test"
                    }
                },
                "writer": {
                    "name": "mysqlwriter",
                    "parameter": {
                        "column": ["id","name"],
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://192.168.40.161:4000/cqgs",
                                "table": ["t1"]
                            }
                        ],
                        "password": "SWZJ@cqgds",
                        "username": "cqgds",
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}


3.2.4 执行oracle同步MySQL配置
[tidb@localhost bin]$ python datax.py ./oracle2mysql4.json 


DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.




2018-02-11 15:02:40.640 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2018-02-11 15:02:40.651 [main] INFO  Engine - the machine info  => 


osInfo: Oracle Corporation 1.8 25.151-b12
jvmInfo: Linux amd64 3.10.0-693.el7.x86_64
cpu num: 4


totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1


GC Names [PS MarkSweep, PS Scavenge]


MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         




2018-02-11 15:02:40.679 [main] INFO  Engine - 
{
"content":[
{
"reader":{
"name":"oraclereader",
"parameter":{
"column":[
"id",
"name"
],
"connection":[
{
"jdbcUrl":[
"jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))"
],
"table":[
"t1"
]
}
],
"password":"****",
"username":"test"
}
},
"writer":{
"name":"mysqlwriter",
"parameter":{
"column":[
"id",
"name"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://192.168.40.161:4000/cqgs",
"table":[
"t1"
]
}
],
"password":"**********",
"username":"cqgds"
}
}
}
],
"setting":{
"speed":{
"channel":"1"
}
}
}


2018-02-11 15:02:40.704 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2018-02-11 15:02:40.707 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2018-02-11 15:02:40.707 [main] INFO  JobContainer - DataX jobContainer starts job.
2018-02-11 15:02:40.710 [main] INFO  JobContainer - Set jobId = 0
2018-02-11 15:02:41.142 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1))).
2018-02-11 15:02:41.266 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t1] has columns:[ID,NAME].
2018-02-11 15:02:44.722 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t1] all columns:[
id,name
].
2018-02-11 15:02:47.336 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (id,name) VALUES(?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.40.161:4000/cqgs?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2018-02-11 15:02:47.337 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2018-02-11 15:02:47.338 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2018-02-11 15:02:47.338 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2018-02-11 15:02:47.339 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2018-02-11 15:02:47.339 [job-0] INFO  JobContainer - Job set Channel-Number to 1 channels.
2018-02-11 15:02:47.345 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2018-02-11 15:02:47.346 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2018-02-11 15:02:47.372 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2018-02-11 15:02:47.377 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2018-02-11 15:02:47.380 [job-0] INFO  JobContainer - Running by standalone Mode.
2018-02-11 15:02:47.392 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2018-02-11 15:02:47.398 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2018-02-11 15:02:47.398 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2018-02-11 15:02:47.411 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2018-02-11 15:02:47.416 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select id,name from t1 
] jdbcUrl:[jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))].
2018-02-11 15:02:47.476 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select id,name from t1 
] jdbcUrl:[jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))].
2018-02-11 15:02:52.721 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[5312]ms
2018-02-11 15:02:52.722 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2018-02-11 15:02:57.410 [job-0] INFO  StandAloneJobContainerCommunicator - Total 19 records, 65 bytes | Speed 6B/s, 1 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2018-02-11 15:02:57.410 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2018-02-11 15:02:57.411 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2018-02-11 15:02:57.411 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do post work.
2018-02-11 15:02:57.412 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2018-02-11 15:02:57.413 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /home/tidb/datax/hook
2018-02-11 15:02:57.415 [job-0] INFO  JobContainer - 
[total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
                        


[total gc info] => 
NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             


2018-02-11 15:02:57.415 [job-0] INFO  JobContainer - PerfTrace not enable!
2018-02-11 15:02:57.416 [job-0] INFO  StandAloneJobContainerCommunicator - Total 19 records, 65 bytes | Speed 6B/s, 1 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2018-02-11 15:02:57.417 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2018-02-11 15:02:40
任务结束时刻                    : 2018-02-11 15:02:57
任务总计耗时                    :                 16s
任务平均流量                    :                6B/s
记录写入速度                    :              1rec/s
读出记录总数                    :                  19
读写失败总数                    :                   0




3.2.5 查询数据同步结果
MySQL [cqgs]> select count(*) from t1;
+----------+
| count(*) |
+----------+
|       19 |
+----------+


3.3 通过参数传递指定适合条件的数据同步
3.3.1 编辑oracle同步MySQL配置
[tidb@localhost bin]$ vi oracle2mysql4.json 


{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "oraclereader",
                    "parameter": {
                        "column": ["id","name"],
                        "connection": [
                            {
                                "jdbcUrl": ["jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))"],
                                "table": ["t1"]
                            }
                        ],
                        "password": "test",
                        "username": "test",
                        "where":"id=$id"
                    }
                },
                "writer": {
                    "name": "mysqlwriter",
                    "parameter": {
                        "column": ["id","name"],
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:mysql://192.168.40.161:4000/cqgs",
                                "table": ["t1"]
                            }
                        ],
                        "password": "SWZJ@cqgds",
                        "username": "cqgds",
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": "1"
            }
        }
    }
}


3.3.2 执行oracle同步MySQL配置
[tidb@localhost bin]$ python datax.py -p"-Did=1" ./oracle2mysql4.json


DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.




2018-02-11 15:38:39.688 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2018-02-11 15:38:39.700 [main] INFO  Engine - the machine info  => 


osInfo: Oracle Corporation 1.8 25.151-b12
jvmInfo: Linux amd64 3.10.0-693.el7.x86_64
cpu num: 4


totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1


GC Names [PS MarkSweep, PS Scavenge]


MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         




2018-02-11 15:38:39.728 [main] INFO  Engine - 
{
"content":[
{
"reader":{
"name":"oraclereader",
"parameter":{
"column":[
"id",
"name"
],
"connection":[
{
"jdbcUrl":[
"jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))"
],
"table":[
"t1"
]
}
],
"password":"****",
"username":"test",
"where":"id=1"
}
},
"writer":{
"name":"mysqlwriter",
"parameter":{
"column":[
"id",
"name"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://192.168.40.161:4000/cqgs",
"table":[
"t1"
]
}
],
"password":"**********",
"username":"cqgds"
}
}
}
],
"setting":{
"speed":{
"channel":"1"
}
}
}


2018-02-11 15:38:39.753 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2018-02-11 15:38:39.756 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2018-02-11 15:38:39.756 [main] INFO  JobContainer - DataX jobContainer starts job.
2018-02-11 15:38:39.759 [main] INFO  JobContainer - Set jobId = 0
2018-02-11 15:38:40.205 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1))).
2018-02-11 15:38:40.334 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t1] has columns:[ID,NAME].
2018-02-11 15:38:43.842 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t1] all columns:[
id,name
].
2018-02-11 15:38:46.699 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (id,name) VALUES(?,?)
], which jdbcUrl like:[jdbc:mysql://192.168.40.161:4000/cqgs?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2018-02-11 15:38:46.699 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2018-02-11 15:38:46.700 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2018-02-11 15:38:46.701 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2018-02-11 15:38:46.703 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2018-02-11 15:38:46.704 [job-0] INFO  JobContainer - Job set Channel-Number to 1 channels.
2018-02-11 15:38:46.709 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2018-02-11 15:38:46.710 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2018-02-11 15:38:46.735 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2018-02-11 15:38:46.741 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2018-02-11 15:38:46.743 [job-0] INFO  JobContainer - Running by standalone Mode.
2018-02-11 15:38:46.755 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2018-02-11 15:38:46.763 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2018-02-11 15:38:46.764 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2018-02-11 15:38:46.777 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2018-02-11 15:38:46.782 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select id,name from t1 where (id=1)
] jdbcUrl:[jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))].
2018-02-11 15:38:46.836 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select id,name from t1 where (id=1)
] jdbcUrl:[jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.118.189)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=ckts)(INSTANCE_NAME=ckts1)))].
2018-02-11 15:38:56.769 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2018-02-11 15:39:02.510 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[15736]ms
2018-02-11 15:39:02.511 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2018-02-11 15:39:06.771 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1 records, 4 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2018-02-11 15:39:06.772 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2018-02-11 15:39:06.773 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2018-02-11 15:39:06.773 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do post work.
2018-02-11 15:39:06.773 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2018-02-11 15:39:06.775 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /home/tidb/datax/hook
2018-02-11 15:39:06.777 [job-0] INFO  JobContainer - 
[total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
                        


[total gc info] => 
NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             


2018-02-11 15:39:06.777 [job-0] INFO  JobContainer - PerfTrace not enable!
2018-02-11 15:39:06.778 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1 records, 4 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2018-02-11 15:39:06.779 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2018-02-11 15:38:39
任务结束时刻                    : 2018-02-11 15:39:06
任务总计耗时                    :                 27s
任务平均流量                    :                0B/s
记录写入速度                    :              0rec/s
读出记录总数                    :                   1
读写失败总数                    :                   0


3.3.3 可以看到只抽取了一条数据
MySQL [cqgs]> select count(*) from t1;
+----------+
| count(*) |
+----------+
|      20 |
+----------+
1 row in set (0.06 sec)
 
3.4 其它测试略
 "where":"fp_dm='${fp_dm}' and fphm='${fphm}'"
python datax.py -p"-Dfp_dm=150001522060 -Dfphm=00710318" ./oracle2mysql.json
export a=150001522060
echo $a
python datax.py -p"-Dfp_dm=$a -Dfphm=00710318" ./oracle2mysql.json
python datax.py -p"-Dfp_dm=`date +%Y%m%d` -Dfphm=00710318" ./oracle2mysql.json