datax(1):安装部署并且检测

一、datax实现功能

可以实现关系型数据库、阿里云数仓、nosql数据库、无结构化数据以及时间序列数据库之间的互相导入,如下图,功能十分强大,所以不觉想要研究一下

datax(1):安装部署并且检测_第1张图片

二、环境

1.Centos6.4

2.Datax

三、下载以及部署

1.下载

https://github.com/alibaba/DataX

点击下载

datax(1):安装部署并且检测_第2张图片

2.上传至服务器解压缩

tar -zxvf datax.tar.gz -C /opt/modules/

3.检测datax安装成功与否

(1)进入datax根目录

/opt/modules/datax

(2)运行测试job

python bin/datax.py job/job.json 

结果:
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2019-05-18 08:43:12.897 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2019-05-18 08:43:12.906 [main] INFO  Engine - the machine info  => 

        osInfo: Oracle Corporation 1.8 25.151-b12
        jvmInfo:        Linux amd64 2.6.32-358.el6.x86_64
        cpu num:        2

        totalPhysicalMemory:    -0.00G
        freePhysicalMemory:     -0.00G
        maxFileDescriptorCount: -1
        currentOpenFileDescriptorCount: -1

        GC Names        [PS MarkSweep, PS Scavenge]

        MEMORY_NAME                    | allocation_size                | init_size                      
        PS Eden Space                  | 256.00MB                       | 256.00MB                       
        Code Cache                     | 240.00MB                       | 2.44MB                         
        Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
        PS Survivor Space              | 42.50MB                        | 42.50MB                        
        PS Old Gen                     | 683.00MB                       | 683.00MB                       
        Metaspace                      | -0.00MB                        | 0.00MB                         


2019-05-18 08:43:12.959 [main] INFO  Engine - 
{
        "content":[
                {
                        "reader":{
                                "name":"streamreader",
                                "parameter":{
                                        "column":[
                                                {
                                                        "type":"string",
                                                        "value":"DataX"
                                                },
                                                {
                                                        "type":"long",
                                                        "value":19890604
                                                },
                                                {
                                                        "type":"date",
                                                        "value":"1989-06-04 00:00:00"
                                                },
                                                {
                                                        "type":"bool",
                                                        "value":true
                                                },
                                                {
                                                        "type":"bytes",
                                                        "value":"test"
                                                }
                                        ],
                                        "sliceRecordCount":100000
                                }
                        },
                        "writer":{
                                "name":"streamwriter",
                                "parameter":{
                                        "encoding":"UTF-8",
                                        "print":false
                                }
                        }
                }
        ],
        "setting":{
                "errorLimit":{
                        "percentage":0.02,
                        "record":0
                },
                "speed":{
                        "byte":10485760
                }
        }
}

2019-05-18 08:43:12.988 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2019-05-18 08:43:12.991 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2019-05-18 08:43:12.991 [main] INFO  JobContainer - DataX jobContainer starts job.
2019-05-18 08:43:12.994 [main] INFO  JobContainer - Set jobId = 0
2019-05-18 08:43:13.034 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2019-05-18 08:43:13.035 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do prepare work .
2019-05-18 08:43:13.035 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2019-05-18 08:43:13.035 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2019-05-18 08:43:13.036 [job-0] INFO  JobContainer - Job set Max-Byte-Speed to 10485760 bytes.
2019-05-18 08:43:13.037 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2019-05-18 08:43:13.042 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2019-05-18 08:43:13.077 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2019-05-18 08:43:13.084 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2019-05-18 08:43:13.090 [job-0] INFO  JobContainer - Running by standalone Mode.
2019-05-18 08:43:13.127 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2019-05-18 08:43:13.133 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2019-05-18 08:43:13.133 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2019-05-18 08:43:13.160 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2019-05-18 08:43:13.564 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[418]ms
2019-05-18 08:43:13.564 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2019-05-18 08:43:23.141 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.028s |  All Task WaitReaderTime 0.120s | Percentage 100.00%
2019-05-18 08:43:23.141 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2019-05-18 08:43:23.142 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2019-05-18 08:43:23.142 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
2019-05-18 08:43:23.142 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2019-05-18 08:43:23.147 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /opt/modules/datax/hook
2019-05-18 08:43:23.149 [job-0] INFO  JobContainer - 
         [total cpu info] => 
                averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
                -1.00%                         | -1.00%                         | -1.00%
                        

         [total gc info] => 
                 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
                 PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
                 PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             

2019-05-18 08:43:23.149 [job-0] INFO  JobContainer - PerfTrace not enable!
2019-05-18 08:43:23.149 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.028s |  All Task WaitReaderTime 0.120s | Percentage 100.00%
2019-05-18 08:43:23.152 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2019-05-18 08:43:12
任务结束时刻                    : 2019-05-18 08:43:23
任务总计耗时                    :                 10s
任务平均流量                    :          253.91KB/s
记录写入速度                    :          10000rec/s
读出记录总数                    :              100000
读写失败总数                    :                   0

(部署成功~)

你可能感兴趣的:(datax)