3、oracle相关同步-oracle通过datax同步到hdfs

DataX3.0系列文章

1、datax3.0部署与验证
2、mysql相关同步-mysql同步到mysql、mysql和hdfs相互同步
3、oracle相关同步-oracle到hdfs
4、sybase相关同步-sybase到hdfs
5、ETL工具的比较(DataPipeline，Kettle，Talend，Informatica，Datax ，Oracle Goldeng

---

---

本文介绍oracle的相关同步，oracle同步到hdfs中。
本文分为三部分，即配置文件模板、配置文件和提交任务。
本文的前提：数据库对应的表已经建好。

一、配置文件模板

[root@bd-node-05 job]# python ../bin/datax.py -r oraclereader -w hdfswriter

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


Please refer to the oraclereader document:
     https://github.com/alibaba/DataX/blob/master/oraclereader/doc/oraclereader.md 

Please refer to the hdfswriter document:
     https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md 
 
Please save the following configuration as a json file and  use
     python {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json 
to run the job.

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "oraclereader", 
                    "parameter": {
                        "column": [], 
                        "connection": [
                            {
                                "jdbcUrl": [], 
                                "table": []
                            }
                        ], 
                        "password": "", 
                        "username": ""
                    }
                }, 
                "writer": {
                    "name": "hdfswriter", 
                    "parameter": {
                        "column": [], 
                        "compress": "", 
                        "defaultFS": "", 
                        "fieldDelimiter": "", 
                        "fileName": "", 
                        "fileType": "", 
                        "path": "", 
                        "writeMode": ""
                    }
                }
            }
        ], 
        "setting": {
            "speed": {
                "channel": ""
            }
        }
    }
}

二、编写配置文件

t_N.json

{
    "job": {
        "content": [
            {
                "reader": {
                    "name": "oraclereader",
                    "parameter": {
                        "username": "test",
                        "password": "88888",
                        "where": "",
                        "connection": [
                            {
                                "querySql": ["select * from T_N_TEST"],
                                "jdbcUrl": ["jdbc:oracle:thin:@192.168.*.*:1521:sdb"]
                            }
                        ]
                    }
                },
                "writer": {
                    "name": "hdfswriter",
                    "parameter": {
                       "defaultFS": "hdfs://hdfsHA",
                        "hadoopConfig":{
                            "dfs.nameservices": "hdfsHA",
                            "dfs.ha.namenodes.hdfsHA": "namenode36,namenode38",
                            "dfs.namenode.rpc-address.hdfsHA.namenode36": "bd-node-02:8020",
                            "dfs.namenode.rpc-address.hdfsHA.namenode38": "bd-node-03:8020",
                            "dfs.client.failover.proxy.provider.hdfsHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider"
                        },

                        "column": [
                            {"name": "LSH","type": "INT"},
                            {"name": "DSHH","type": "STRING"},
                            {"name": "DKHZH","type": "STRING"},
                            {"name": "DCZRQ","type": "TIMESTAMP"},
                            {"name": "DCZJE","type": "double"},
                            {"name": "DYXQ","type": "TIMESTAMP"},
                            {"name": "DCZFS","type": "STRING"},
                            {"name": "DCZKH","type": "STRING"},
                            {"name": "DIPDH","type": "STRING"},
                            {"name": "DCZRY","type": "INT"},
                            {"name": "DTGID","type": "STRING"},
                            {"name": "HZKHZH","type": "STRING"},
                            {"name": "DXGDDDBH","type": "STRING"},
                            {"name": "SHRY","type": "INT"},
                            {"name": "SHRQ","type": "TIMESTAMP"},
                            {"name": "SHFLAG","type": "STRING"},
                            {"name": "HZFLAG","type": "STRING"},
                            {"name": "FLAG","type": "STRING"},
                            {"name": "DFXSH","type": "STRING"},
                            {"name": "DZT","type": "STRING"},
                            {"name": "DJSSH","type": "STRING"},
                            {"name": "DKHLSH","type": "INT"}
                        ],
                        "compress": "",

                        "fieldDelimiter": ",",
                        "fileName": "t_n",
                        "fileType": "text",
                        "path": "/user/hive/warehouse/test.db/t_n",
                        "writeMode": "append"
                    }
                }
            }
        ],
        "setting": {
            "speed": {
                "channel": 6
            }
        }
    }
}

三、提交任务

python ../bin/datax.py --jvm="-Xms32G -Xmx32G" t_N.json

[root@bd-node-05 job]# python ../bin/datax.py --jvm="-Xms32G -Xmx32G" t_N.json

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2023-04-04 14:55:39.768 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-04-04 14:55:39.777 [main] INFO  Engine - the machine info  => 

        osInfo: Oracle Corporation 1.8 25.181-b13
        jvmInfo:        Linux amd64 3.10.0-1160.88.1.el7.x86_64
        cpu num:        32

        totalPhysicalMemory:    -0.00G
        freePhysicalMemory:     -0.00G
        maxFileDescriptorCount: -1
        currentOpenFileDescriptorCount: -1

        GC Names        [PS MarkSweep, PS Scavenge]

        MEMORY_NAME                    | allocation_size                | init_size                      
        PS Eden Space                  | 8,192.50MB                     | 8,192.50MB                     
        Code Cache                     | 240.00MB                       | 2.44MB                         
        PS Survivor Space              | 1,365.00MB                     | 1,365.00MB                     
        PS Old Gen                     | 21,845.50MB                    | 21,845.50MB                    
        Metaspace                      | -0.00MB                        | 0.00MB                         


2023-04-04 14:55:39.806 [main] INFO  Engine - 
{
        "content":[
                {
                        "reader":{
                                "name":"oraclereader",
                                "parameter":{
                                        "connection":[
                                                {
                                                        "jdbcUrl":[
                                                                "jdbc:oracle:thin:@192.168.*.*:1521:sdb"
                                                        ],
                                                        "querySql":[
                                                                "select * from T_N_TEST"
                                                        ]
                                                }
                                        ],
                                        "password":"****",
                                        "username":"test",
                                        "where":""
                                }
                        },
                        "writer":{
                                "name":"hdfswriter",
                                "parameter":{
                                        "column":[
                                                {
                                                        "name":"LSH",
                                                        "type":"INT"
                                                },
                                                {
                                                        "name":"DSHH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DKHZH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DCZRQ",
                                                        "type":"TIMESTAMP"
                                                },
                                                {
                                                        "name":"DCZJE",
                                                        "type":"double"
                                                },
                                                {
                                                        "name":"DYXQ",
                                                        "type":"TIMESTAMP"
                                                },
                                                {
                                                        "name":"DCZFS",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DCZKH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DIPDH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DCZRY",
                                                        "type":"INT"
                                                },
                                                {
                                                        "name":"DTGID",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"HZKHZH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DXGDDDBH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"SHRY",
                                                        "type":"INT"
                                                },
                                                {
                                                        "name":"SHRQ",
                                                        "type":"TIMESTAMP"
                                                },
                                                {
                                                        "name":"SHFLAG",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"HZFLAG",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"FLAG",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DFXSH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DZT",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DJSSH",
                                                        "type":"STRING"
                                                },
                                                {
                                                        "name":"DKHLSH",
                                                        "type":"INT"
                                                }
                                        ],
                                        "compress":"",
                                        "defaultFS":"hdfs://hdfsHA",
                                        "fieldDelimiter":",",
                                        "fileName":"t_n",
                                        "fileType":"text",
                                        "hadoopConfig":{
                                                "dfs.client.failover.proxy.provider.hdfsHA":"org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
                                                "dfs.ha.namenodes.hdfsHA":"namenode36,namenode38",
                                                "dfs.namenode.rpc-address.hdfsHA.namenode36":"bd-node-02:8020",
                                                "dfs.namenode.rpc-address.hdfsHA.namenode38":"bd-node-03:8020",
                                                "dfs.nameservices":"hdfsHA"
                                        },
                                        "path":"/user/hive/warehouse/test.db/t_n",
                                        "writeMode":"append"
                                }
                        }
                }
        ],
        "setting":{
                "speed":{
                        "channel":6
                }
        }
}

2023-04-04 14:55:39.830 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-04-04 14:55:39.832 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-04-04 14:55:39.832 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-04-04 14:55:39.834 [main] INFO  JobContainer - Set jobId = 0
2023-04-04 14:55:40.247 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:oracle:thin:@192.168.10.125:1521:sit2db.
Apr 04, 2023 2:55:40 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-04-04 14:55:41.524 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-04-04 14:55:41.525 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2023-04-04 14:55:41.526 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] do prepare work .
2023-04-04 14:55:41.638 [job-0] INFO  HdfsWriter$Job - 由于您配置了writeMode append, 写入前不做清理工作, [/user/hive/warehouse/test.db/t_czmx] 目录下写入相应文件名前缀  [t_czmx] 的文件
2023-04-04 14:55:41.639 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-04-04 14:55:41.640 [job-0] INFO  JobContainer - Job set Channel-Number to 6 channels.
2023-04-04 14:55:41.643 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [1] tasks.
2023-04-04 14:55:41.643 [job-0] INFO  HdfsWriter$Job - begin do split...
2023-04-04 14:55:41.649 [job-0] INFO  HdfsWriter$Job - splited write file name:[hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx__660c1ea4_4b84_4abe_b81f_f626e51f40f9/t_czmx__48e62953_4261_4f28_8c1a_4dbaeed84845]
2023-04-04 14:55:41.650 [job-0] INFO  HdfsWriter$Job - end do split.
2023-04-04 14:55:41.650 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] splits to [1] tasks.
2023-04-04 14:55:41.673 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-04-04 14:55:41.678 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-04-04 14:55:41.682 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-04-04 14:55:41.692 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-04-04 14:55:41.698 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-04-04 14:55:41.699 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-04-04 14:55:41.709 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-04-04 14:55:41.714 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from T_CZMX_N_TEST
] jdbcUrl:[jdbc:oracle:thin:@192.168.10.125:1521:sit2db].
2023-04-04 14:55:41.749 [0-0-0-writer] INFO  HdfsWriter$Task - begin do write...
2023-04-04 14:55:41.749 [0-0-0-writer] INFO  HdfsWriter$Task - write to file : [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx__660c1ea4_4b84_4abe_b81f_f626e51f40f9/t_czmx__48e62953_4261_4f28_8c1a_4dbaeed84845]
2023-04-04 14:55:51.708 [job-0] INFO  StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-04-04 14:56:01.713 [job-0] INFO  StandAloneJobContainerCommunicator - Total 364064 records, 75158330 bytes | Speed 7.17MB/s, 36406 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 4.351s |  All Task WaitReaderTime 0.108s | Percentage 0.00%
2023-04-04 14:56:11.716 [job-0] INFO  StandAloneJobContainerCommunicator - Total 788448 records, 158817628 bytes | Speed 7.98MB/s, 42438 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 8.063s |  All Task WaitReaderTime 0.127s | Percentage 0.00%
2023-04-04 14:56:21.721 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1236640 records, 251410563 bytes | Speed 8.83MB/s, 44819 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 12.302s |  All Task WaitReaderTime 0.140s | Percentage 0.00%
2023-04-04 14:56:31.724 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1878848 records, 384090269 bytes | Speed 12.65MB/s, 64220 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 15.643s |  All Task WaitReaderTime 0.181s | Percentage 0.00%
2023-04-04 14:56:41.736 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2260288 records, 462854390 bytes | Speed 7.51MB/s, 38144 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 20.287s |  All Task WaitReaderTime 0.196s | Percentage 0.00%
2023-04-04 14:56:51.738 [job-0] INFO  StandAloneJobContainerCommunicator - Total 2692704 records, 552224653 bytes | Speed 8.52MB/s, 43241 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 24.429s |  All Task WaitReaderTime 0.212s | Percentage 0.00%
2023-04-04 14:57:01.739 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3153216 records, 647450156 bytes | Speed 9.08MB/s, 46051 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 28.152s |  All Task WaitReaderTime 0.240s | Percentage 0.00%
2023-04-04 14:57:11.741 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3628640 records, 746042872 bytes | Speed 9.40MB/s, 47542 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 32.027s |  All Task WaitReaderTime 0.258s | Percentage 0.00%
2023-04-04 14:57:21.743 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4081184 records, 839941636 bytes | Speed 8.95MB/s, 45254 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 35.559s |  All Task WaitReaderTime 0.275s | Percentage 0.00%
2023-04-04 14:57:31.744 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4518528 records, 930651510 bytes | Speed 8.65MB/s, 43734 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 39.606s |  All Task WaitReaderTime 0.290s | Percentage 0.00%
2023-04-04 14:57:41.746 [job-0] INFO  StandAloneJobContainerCommunicator - Total 5227488 records, 1077738288 bytes | Speed 14.03MB/s, 70896 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 42.516s |  All Task WaitReaderTime 0.341s | Percentage 0.00%
2023-04-04 14:57:51.748 [job-0] INFO  StandAloneJobContainerCommunicator - Total 5737440 records, 1183501317 bytes | Speed 10.09MB/s, 50995 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 46.185s |  All Task WaitReaderTime 0.361s | Percentage 0.00%
2023-04-04 14:58:01.750 [job-0] INFO  StandAloneJobContainerCommunicator - Total 6412096 records, 1323434937 bytes | Speed 13.35MB/s, 67465 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 49.287s |  All Task WaitReaderTime 0.409s | Percentage 0.00%
2023-04-04 14:58:11.753 [job-0] INFO  StandAloneJobContainerCommunicator - Total 6873600 records, 1419023771 bytes | Speed 9.12MB/s, 46150 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 52.938s |  All Task WaitReaderTime 0.426s | Percentage 0.00%
2023-04-04 14:58:21.754 [job-0] INFO  StandAloneJobContainerCommunicator - Total 7346976 records, 1517131339 bytes | Speed 9.36MB/s, 47337 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 56.703s |  All Task WaitReaderTime 0.443s | Percentage 0.00%
2023-04-04 14:58:31.756 [job-0] INFO  StandAloneJobContainerCommunicator - Total 7849408 records, 1621286606 bytes | Speed 9.93MB/s, 50243 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 60.467s |  All Task WaitReaderTime 0.471s | Percentage 0.00%
2023-04-04 14:58:41.758 [job-0] INFO  StandAloneJobContainerCommunicator - Total 8546816 records, 1758394513 bytes | Speed 13.08MB/s, 69740 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 63.179s |  All Task WaitReaderTime 0.521s | Percentage 0.00%
2023-04-04 14:58:51.760 [job-0] INFO  StandAloneJobContainerCommunicator - Total 9071584 records, 1855859070 bytes | Speed 9.29MB/s, 52476 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 66.520s |  All Task WaitReaderTime 0.541s | Percentage 0.00%
2023-04-04 14:59:01.768 [job-0] INFO  StandAloneJobContainerCommunicator - Total 9548704 records, 1944383759 bytes | Speed 8.44MB/s, 47712 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 69.981s |  All Task WaitReaderTime 0.563s | Percentage 0.00%
2023-04-04 14:59:11.769 [job-0] INFO  StandAloneJobContainerCommunicator - Total 10233632 records, 2071482993 bytes | Speed 12.12MB/s, 68492 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 72.720s |  All Task WaitReaderTime 0.631s | Percentage 0.00%
2023-04-04 14:59:21.771 [job-0] INFO  StandAloneJobContainerCommunicator - Total 10783712 records, 2173657545 bytes | Speed 9.74MB/s, 55008 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 75.918s |  All Task WaitReaderTime 0.662s | Percentage 0.00%
2023-04-04 14:59:31.773 [job-0] INFO  StandAloneJobContainerCommunicator - Total 11292896 records, 2268308285 bytes | Speed 9.03MB/s, 50918 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 79.182s |  All Task WaitReaderTime 0.689s | Percentage 0.00%
2023-04-04 14:59:41.774 [job-0] INFO  StandAloneJobContainerCommunicator - Total 11791328 records, 2361035165 bytes | Speed 8.84MB/s, 49843 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 82.508s |  All Task WaitReaderTime 0.711s | Percentage 0.00%
2023-04-04 14:59:51.776 [job-0] INFO  StandAloneJobContainerCommunicator - Total 12556384 records, 2503218056 bytes | Speed 13.56MB/s, 76505 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 84.926s |  All Task WaitReaderTime 0.787s | Percentage 0.00%
2023-04-04 15:00:01.779 [job-0] INFO  StandAloneJobContainerCommunicator - Total 13026272 records, 2590452251 bytes | Speed 8.32MB/s, 46988 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 88.515s |  All Task WaitReaderTime 0.803s | Percentage 0.00%
2023-04-04 15:00:11.781 [job-0] INFO  StandAloneJobContainerCommunicator - Total 13546048 records, 2687111286 bytes | Speed 9.22MB/s, 51977 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 91.896s |  All Task WaitReaderTime 0.821s | Percentage 0.00%
2023-04-04 15:00:21.783 [job-0] INFO  StandAloneJobContainerCommunicator - Total 14034880 records, 2777915766 bytes | Speed 8.66MB/s, 48883 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 95.701s |  All Task WaitReaderTime 0.843s | Percentage 0.00%
2023-04-04 15:00:31.786 [job-0] INFO  StandAloneJobContainerCommunicator - Total 14752736 records, 2911439294 bytes | Speed 12.73MB/s, 71785 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 98.210s |  All Task WaitReaderTime 0.912s | Percentage 0.00%
2023-04-04 15:00:41.789 [job-0] INFO  StandAloneJobContainerCommunicator - Total 15263584 records, 3006475966 bytes | Speed 9.06MB/s, 51084 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 102.098s |  All Task WaitReaderTime 0.936s | Percentage 0.00%
2023-04-04 15:00:41.794 [job-0] INFO  VMInfo - 
         [delta cpu info] => 
                curDeltaCpu                    | averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
                -1.00%                         | -1.00%                         | -1.00%                         | -1.00%
                        

         [delta memory info] => 
                 NAME                           | used_size                      | used_percent                   | max_used_size                  | max_percent                    
                 PS Eden Space                  | 4,356.46MB                     | 40.03%                         | 4,356.46MB                     | 40.03%                         
                 Code Cache                     | 12.14MB                        | 85.93%                         | 12.14MB                        | 85.93%                         
                 PS Survivor Space              | 2.09MB                         | 10.21%                         | 2.09MB                         | 10.21%                         
                 PS Old Gen                     | 29.48MB                        | 0.13%                          | 29.48MB                        | 0.13%                          
                 Metaspace                      | 30.58MB                        | 97.85%                         | 30.58MB                        | 97.85%                         

         [delta gc info] => 
                 NAME                 | curDeltaGCCount    | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | curDeltaGCTime     | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
                 PS MarkSweep         | 1                  | 1                  | 1                  | 1                  | 0.050s             | 0.050s             | 0.050s             | 0.050s             
                 PS Scavenge          | 28                 | 28                 | 28                 | 28                 | 0.312s             | 0.312s             | 0.312s             | 0.312s             

2023-04-04 15:00:51.795 [job-0] INFO  StandAloneJobContainerCommunicator - Total 15753056 records, 3097577099 bytes | Speed 8.69MB/s, 48947 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 105.317s |  All Task WaitReaderTime 0.978s | Percentage 0.00%
2023-04-04 15:01:01.797 [job-0] INFO  StandAloneJobContainerCommunicator - Total 16275776 records, 3194886855 bytes | Speed 9.28MB/s, 52272 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 108.683s |  All Task WaitReaderTime 1.000s | Percentage 0.00%
2023-04-04 15:01:11.800 [job-0] INFO  StandAloneJobContainerCommunicator - Total 16799648 records, 3292384628 bytes | Speed 9.30MB/s, 52387 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 112.121s |  All Task WaitReaderTime 1.018s | Percentage 0.00%
2023-04-04 15:01:21.802 [job-0] INFO  StandAloneJobContainerCommunicator - Total 17311744 records, 3387202766 bytes | Speed 9.04MB/s, 51209 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 115.273s |  All Task WaitReaderTime 1.040s | Percentage 0.00%
2023-04-04 15:01:31.804 [job-0] INFO  StandAloneJobContainerCommunicator - Total 17809376 records, 3478215345 bytes | Speed 8.68MB/s, 49763 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 118.389s |  All Task WaitReaderTime 1.075s | Percentage 0.00%
2023-04-04 15:01:41.806 [job-0] INFO  StandAloneJobContainerCommunicator - Total 18585248 records, 3621287915 bytes | Speed 13.64MB/s, 77587 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 120.856s |  All Task WaitReaderTime 1.135s | Percentage 0.00%
2023-04-04 15:01:51.808 [job-0] INFO  StandAloneJobContainerCommunicator - Total 19227520 records, 3740798227 bytes | Speed 11.40MB/s, 64227 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 123.855s |  All Task WaitReaderTime 1.175s | Percentage 0.00%
2023-04-04 15:02:01.810 [job-0] INFO  StandAloneJobContainerCommunicator - Total 19854016 records, 3857581823 bytes | Speed 11.14MB/s, 62649 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 126.758s |  All Task WaitReaderTime 1.203s | Percentage 0.00%
2023-04-04 15:02:11.811 [job-0] INFO  StandAloneJobContainerCommunicator - Total 20230688 records, 3927823505 bytes | Speed 6.70MB/s, 37667 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 131.807s |  All Task WaitReaderTime 1.222s | Percentage 0.00%
2023-04-04 15:02:21.813 [job-0] INFO  StandAloneJobContainerCommunicator - Total 20657056 records, 4007273953 bytes | Speed 7.58MB/s, 42636 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 135.797s |  All Task WaitReaderTime 1.251s | Percentage 0.00%
2023-04-04 15:02:31.815 [job-0] INFO  StandAloneJobContainerCommunicator - Total 20930912 records, 4058268755 bytes | Speed 4.86MB/s, 27385 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 138.955s |  All Task WaitReaderTime 1.262s | Percentage 0.00%
2023-04-04 15:02:41.817 [job-0] INFO  StandAloneJobContainerCommunicator - Total 21370464 records, 4140184722 bytes | Speed 7.81MB/s, 43955 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 145.977s |  All Task WaitReaderTime 1.293s | Percentage 0.00%
2023-04-04 15:02:51.819 [job-0] INFO  StandAloneJobContainerCommunicator - Total 21503648 records, 4165013984 bytes | Speed 2.37MB/s, 13318 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 154.199s |  All Task WaitReaderTime 1.298s | Percentage 0.00%
2023-04-04 15:03:01.820 [job-0] INFO  StandAloneJobContainerCommunicator - Total 22001632 records, 4257769146 bytes | Speed 8.85MB/s, 49798 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 157.548s |  All Task WaitReaderTime 1.325s | Percentage 0.00%
2023-04-04 15:03:11.824 [job-0] INFO  StandAloneJobContainerCommunicator - Total 22291488 records, 4311701845 bytes | Speed 5.14MB/s, 28985 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 163.472s |  All Task WaitReaderTime 1.335s | Percentage 0.00%
2023-04-04 15:03:21.825 [job-0] INFO  StandAloneJobContainerCommunicator - Total 23044064 records, 4452265650 bytes | Speed 13.41MB/s, 75257 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 165.943s |  All Task WaitReaderTime 1.428s | Percentage 0.00%
2023-04-04 15:03:31.828 [job-0] INFO  StandAloneJobContainerCommunicator - Total 23528000 records, 4547787745 bytes | Speed 9.11MB/s, 48393 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 170.004s |  All Task WaitReaderTime 1.444s | Percentage 0.00%
2023-04-04 15:03:41.830 [job-0] INFO  StandAloneJobContainerCommunicator - Total 24044384 records, 4644070979 bytes | Speed 9.18MB/s, 51638 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 173.295s |  All Task WaitReaderTime 1.463s | Percentage 0.00%
2023-04-04 15:03:51.833 [job-0] INFO  StandAloneJobContainerCommunicator - Total 24550336 records, 4738247200 bytes | Speed 8.98MB/s, 50595 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 176.563s |  All Task WaitReaderTime 1.481s | Percentage 0.00%
2023-04-04 15:04:01.835 [job-0] INFO  StandAloneJobContainerCommunicator - Total 25029184 records, 4827459815 bytes | Speed 8.51MB/s, 47884 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 179.770s |  All Task WaitReaderTime 1.503s | Percentage 0.00%
2023-04-04 15:04:11.838 [job-0] INFO  StandAloneJobContainerCommunicator - Total 25557664 records, 4925795426 bytes | Speed 9.38MB/s, 52848 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 182.818s |  All Task WaitReaderTime 1.526s | Percentage 0.00%
2023-04-04 15:04:21.840 [job-0] INFO  StandAloneJobContainerCommunicator - Total 26072608 records, 5021987171 bytes | Speed 9.17MB/s, 51494 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 186.044s |  All Task WaitReaderTime 1.570s | Percentage 0.00%
2023-04-04 15:04:31.841 [job-0] INFO  StandAloneJobContainerCommunicator - Total 26603488 records, 5121506534 bytes | Speed 9.49MB/s, 53088 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 189.248s |  All Task WaitReaderTime 1.592s | Percentage 0.00%
2023-04-04 15:04:41.844 [job-0] INFO  StandAloneJobContainerCommunicator - Total 27104608 records, 5215652596 bytes | Speed 8.98MB/s, 50112 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 192.669s |  All Task WaitReaderTime 1.611s | Percentage 0.00%
2023-04-04 15:04:51.845 [job-0] INFO  StandAloneJobContainerCommunicator - Total 27637216 records, 5315023314 bytes | Speed 9.48MB/s, 53260 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 195.883s |  All Task WaitReaderTime 1.635s | Percentage 0.00%
2023-04-04 15:05:01.847 [job-0] INFO  StandAloneJobContainerCommunicator - Total 28096640 records, 5400561102 bytes | Speed 8.16MB/s, 45942 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 199.792s |  All Task WaitReaderTime 1.651s | Percentage 0.00%
2023-04-04 15:05:05.249 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from T_CZMX_N_TEST
] jdbcUrl:[jdbc:oracle:thin:@192.168.10.125:1521:sit2db].
2023-04-04 15:05:05.289 [0-0-0-writer] INFO  HdfsWriter$Task - end do write
2023-04-04 15:05:05.380 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[563672]ms
2023-04-04 15:05:05.381 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-04-04 15:05:11.849 [job-0] INFO  StandAloneJobContainerCommunicator - Total 28913070 records, 5553346220 bytes | Speed 14.57MB/s, 81643 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 203.362s |  All Task WaitReaderTime 1.728s | Percentage 100.00%
2023-04-04 15:05:11.849 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-04-04 15:05:11.850 [job-0] INFO  JobContainer - DataX Writer.Job [hdfswriter] do post work.
2023-04-04 15:05:11.851 [job-0] INFO  HdfsWriter$Job - start rename file [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx__660c1ea4_4b84_4abe_b81f_f626e51f40f9/t_czmx__48e62953_4261_4f28_8c1a_4dbaeed84845] to file [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx/t_czmx__48e62953_4261_4f28_8c1a_4dbaeed84845].
2023-04-04 15:05:11.886 [job-0] INFO  HdfsWriter$Job - finish rename file [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx__660c1ea4_4b84_4abe_b81f_f626e51f40f9/t_czmx__48e62953_4261_4f28_8c1a_4dbaeed84845] to file [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx/t_czmx__48e62953_4261_4f28_8c1a_4dbaeed84845].
2023-04-04 15:05:11.886 [job-0] INFO  HdfsWriter$Job - start delete tmp dir [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx__660c1ea4_4b84_4abe_b81f_f626e51f40f9] .
2023-04-04 15:05:11.919 [job-0] INFO  HdfsWriter$Job - finish delete tmp dir [hdfs://hdfsHA/user/hive/warehouse/test.db/t_czmx__660c1ea4_4b84_4abe_b81f_f626e51f40f9] .
2023-04-04 15:05:11.920 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do post work.
2023-04-04 15:05:11.920 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-04-04 15:05:11.922 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook
2023-04-04 15:05:12.025 [job-0] INFO  JobContainer - 
         [total cpu info] => 
                averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
                -1.00%                         | -1.00%                         | -1.00%
                        

         [total gc info] => 
                 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
                 PS MarkSweep         | 1                  | 1                  | 0                  | 0.050s             | 0.050s             | 0.000s             
                 PS Scavenge          | 48                 | 28                 | 20                 | 0.591s             | 0.312s             | 0.279s             

2023-04-04 15:05:12.026 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-04-04 15:05:12.026 [job-0] INFO  StandAloneJobContainerCommunicator - Total 28913070 records, 5553346220 bytes | Speed 9.29MB/s, 50724 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 203.362s |  All Task WaitReaderTime 1.728s | Percentage 100.00%
2023-04-04 15:05:12.029 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-04-04 14:55:39
任务结束时刻                    : 2023-04-04 15:05:12
任务总计耗时                    :                572s
任务平均流量                    :            9.29MB/s
记录写入速度                    :          50724rec/s
读出记录总数                    :            28913070
读写失败总数                    :                   0