datax安装及配置

System Requirements

• Linux
• JDK(1.8以上，推荐1.8)
• Python(推荐Python2.6.X)
• Apache Maven 3.x (Compile DataX)

1. wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
2. tar zxvf datax.tar.gz
3. 测试脚本及环境运行情况：python bin/datax.py job/job.json

打印如下说明正常：

[root@masternode datax]# python bin/datax.py job/job.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.


2018-08-31 11:24:00.048 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2018-08-31 11:24:00.059 [main] INFO  Engine - the machine info  => 

	osInfo:	Oracle Corporation 1.8 25.171-b11
	jvmInfo:	Linux amd64 3.10.0-862.el7.x86_64
	cpu num:	16

	totalPhysicalMemory:	-0.00G
	freePhysicalMemory:	-0.00G
	maxFileDescriptorCount:	-1
	currentOpenFileDescriptorCount:	-1

	GC Names	[PS MarkSweep, PS Scavenge]

	MEMORY_NAME                    | allocation_size                | init_size                      
	PS Eden Space                  | 256.00MB                       | 256.00MB                       
	Code Cache                     | 240.00MB                       | 2.44MB                         
	Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
	PS Survivor Space              | 42.50MB                        | 42.50MB                        
	PS Old Gen                     | 683.00MB                       | 683.00MB                       
	Metaspace                      | -0.00MB                        | 0.00MB                         


2018-08-31 11:24:00.084 [main] INFO  Engine - 
{
	"content":[
		{
			"reader":{
				"name":"streamreader",
				"parameter":{
					"column":[
						{
							"type":"string",
							"value":"DataX"
						},
						{
							"type":"long",
							"value":19890604
						},
						{
							"type":"date",
							"value":"1989-06-04 00:00:00"
						},
						{
							"type":"bool",
							"value":true
						},
						{
							"type":"bytes",
							"value":"test"
						}
					],
					"sliceRecordCount":100000
				}
			},
			"writer":{
				"name":"streamwriter",
				"parameter":{
					"encoding":"UTF-8",
					"print":false
				}
			}
		}
	],
	"setting":{
		"errorLimit":{
			"percentage":0.02,
			"record":0
		},
		"speed":{
			"byte":10485760
		}
	}
}

2018-08-31 11:24:00.106 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2018-08-31 11:24:00.108 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2018-08-31 11:24:00.108 [main] INFO  JobContainer - DataX jobContainer starts job.
2018-08-31 11:24:00.111 [main] INFO  JobContainer - Set jobId = 0
2018-08-31 11:24:00.131 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2018-08-31 11:24:00.131 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do prepare work .
2018-08-31 11:24:00.132 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2018-08-31 11:24:00.132 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2018-08-31 11:24:00.133 [job-0] INFO  JobContainer - Job set Max-Byte-Speed to 10485760 bytes.
2018-08-31 11:24:00.134 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2018-08-31 11:24:00.135 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2018-08-31 11:24:00.164 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2018-08-31 11:24:00.171 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2018-08-31 11:24:00.173 [job-0] INFO  JobContainer - Running by standalone Mode.
2018-08-31 11:24:00.182 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2018-08-31 11:24:00.188 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2018-08-31 11:24:00.189 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2018-08-31 11:24:00.201 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2018-08-31 11:24:00.503 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[302]ms
2018-08-31 11:24:00.504 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2018-08-31 11:24:10.193 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.019s |  All Task WaitReaderTime 0.041s | Percentage 100.00%
2018-08-31 11:24:10.193 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2018-08-31 11:24:10.193 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2018-08-31 11:24:10.194 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
2018-08-31 11:24:10.194 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2018-08-31 11:24:10.195 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /opt/datax/hook
2018-08-31 11:24:10.197 [job-0] INFO  JobContainer - 
	 [total cpu info] => 
		averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
		-1.00%                         | -1.00%                         | -1.00%
                        

	 [total gc info] => 
		 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
		 PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
		 PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             

2018-08-31 11:24:10.197 [job-0] INFO  JobContainer - PerfTrace not enable!
2018-08-31 11:24:10.198 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.019s |  All Task WaitReaderTime 0.041s | Percentage 100.00%
2018-08-31 11:24:10.200 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2018-08-31 11:24:00
任务结束时刻                    : 2018-08-31 11:24:10
任务总计耗时                    :                 10s
任务平均流量                    :          253.91KB/s
记录写入速度                    :          10000rec/s
读出记录总数                    :              100000
读写失败总数                    :                   0