【DataX-Web的搭建（Windows/Linux环境）

备注：我本地目录规划，统一放到文件夹：

1. D:\java\Projects\sync-data\datax
2. D:\java\Projects\sync-data\datax-web

一、安装DataX

(1) 安装python3（配置环境变量），但从github上下载的版本只支持python2.x版本，而现在python3.x正流行，这给我们带来不少麻烦。

datax 支持Python3需要修改替换datax/bin下面的三个python文件，替换文件地址：datax-web/doc/datax-web/datax-python3 at master · WeiYe-Jing/datax-web · GitHub

(2) 安装jdk8（配置环境变量）

(3) 安装 maven 3.x（配置环境变量）

(4) datax.tar.gz包下载并解压：http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz

(5) datax源码下载：https://github.com/alibaba/DataX。(下载源码主要是因为某些源码需要自己修改，然后重新打包覆盖导datax，如下述的mysql版本不一致例子)

• ps1：自动下载依赖，如果hitsdb-client报错，则将hitsdb-client依赖版本改为0.3.7即可
• ps2：因为我的项目用的是mysql8，所以需要将mysqlReader、mysqlWriter服务的mysql的依赖改为8.0.17，默认是5.x
• ps3：datax-all服务执行clean-install
• ps4：将mysqlReader、mysqlWriter生成的target–>datax–>plugins复制到datax的bin同级的目录下，覆盖即可
• ps5：将“datax\plugin\reader\mysqlreader\libs”、“datax\plugin\writer\mysqlwriter\libs”目录下的mysql5的包删除
   

二、使用DataX

(1)  打开cmd界面

(2) 进入datax的bin目录下

(3) 输入CHCP 65001 防止中文乱码

(4) 查看模板命令行：python datax.py -r streamreader -w streamwriter

(5) 执行脚本：python datax.py E:\datax\datax\datax\job\job.json
ps1：报错：提供的配置文件[D:\java\Projects\sync-data\datax\plugin\reader._cassandrareader\plugin.json]不存在. 请检查您的配置文件.
​   —>解决：删除datax\plugin\writer、datax\plugin\reader底下"._"开头文件

(6) 编写脚本，将脚本存放到datax\datax\job目录底下，然后执行后python datax.py datax\job\你的脚本.json运行

ps1：脚本格式信息可以去https://github.com/alibaba/DataX/查看

ps2：脚本基础参数说明（mysql、oracle）

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62

{     "job": {         "setting": {             "speed": {                 "channel": 并发数             }         },         "content": [             {                  "reader": {                     "name": "固定的名字",                     "parameter": {                         "username": "账号",                         "password": "密码",                         "column": [                             "字段1",                             "字段2",                             "字段3"                             ps:[*]表示所有列，但是不建议使用                         ],                         "splitPk": "分片字段，丢给task，所以必须是整形。一般用主键即可（非必须）",                         "connection": [                             {                                 "table": [ "表"],                                 "jdbcUrl": [  "数据库连接地址"],                                 "querySql":["这里可以写查询数据的sql语句，但是如果这里写了，则不允许再配置table，否则会报错（非必须，table和querySql只能存在一个）"]                             }                         ],                         "where":"过滤条件"                     }                 },                 "writer": {                     "name": "固定的名字",                     "parameter": {                         "writeMode": "写入策略",                         "username": "账户",                         "password": "密码",                         "column": [                             "字段1",                             "字段2",                             "字段3"                         ],                         "session": [                             "DataX在获取Mysql连接时，执行session指定的SQL语句，修改当前connection session属性（非必须）"                         ],                         "preSql": [                             "写入数据到目的表前，会先执行这里的标准语句（非必须）"                         ],                         "connection": [                             {                                 "jdbcUrl": "数据库连接地址",                                 "table": [                                     "表"                                 ]                             }                         ]                     }                 }             }         ]     } }

ps3：将mysql的datax数据过滤后迁移到oracle实例

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

{     "job": {         "setting": {             "speed": {                 "channel": 1             }         },         "content": [             {                  "reader": {                     "name": "mysqlreader",                     "parameter": {                         "username": "root",                         "password": "tiger",                         "column": [                             "id",                             "name",                             "time"                         ],                         "splitPk": "id",                         "connection": [                             {                                 "table": [                                     "datax"                                 ],                                 "jdbcUrl": [      "jdbc:mysql://127.0.0.1:3306/datax?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false&serverTimezone=GMT%2b8"                                 ]                             }                         ],                         "where":"id < 10"                     }                 },                  "writer": {                     "name": "oraclewriter",                     "parameter": {                         "username": "HYDROPOWER_JIANGXI",                         "password": "ffcsict123",                         "column": [                             "id",                             "name",                             "time"                         ],                         "preSql": [                             "delete from datax"                         ],                         "connection": [                             {                                 "jdbcUrl": "jdbc:oracle:thin:@192.168.35.9:1521:orcl",                                 "table": [                                     "datax"                                 ]                             }                         ]                     }                 }             }         ]     } }

(7) 达梦数据库：因为数据库介绍没有dm数据库，所以只能使用“通用RDBMS(支持所有关系型数据库)”这种模式进行传输。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

{     "job": {         "setting": {             "speed": {                 "channel": 1             }         },         "content": [             {                  "reader": {                     "name": "mysqlreader",                     "parameter": {                         "username": "root",                         "password": "tiger",                         "column": [                             "id",                             "name",                             "time"                         ],                         "splitPk": "id",                         "connection": [                             {                                 "table": [                                     "datax"                                 ],                                 "jdbcUrl": [      "jdbc:mysql://127.0.0.1:3306/datax?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false&serverTimezone=GMT%2b8"                                 ]                             }                         ],                         "where":"id < 10"                     }                 },                  "writer": {                     "name": "rdbmswriter",                     "parameter": {                         "username": "AES",                         "password": "ffcsict123",                         "column": [                             "id",                             "name",                             "time"                         ],                         "preSql": [                             "delete from datax"                         ],                         "connection": [                             {                                 "jdbcUrl": "jdbc:dm://127.0.0.1:5236/AES?zeroDateTimeBehavior=converToNull&useUnicode=true&characterEncoding=utf-8",                                 "table": [                                     "datax"                                 ]                             }                         ]                     }                 }             }         ]     } }

(8) 文件流：只能读取规则的二维表结构文件，并且只能读取本地文件及远程服务器的文件，只能调用api接口获取流

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72

{     "setting": {},     "job": {         "setting": {             "speed": {                 "channel": 2             }         },         "content": [             {                 "reader": {                     "name": "ftpreader",                     "parameter": {                         "protocol": "sftp",                         "host": "192.168.248.11",                         "port": 22,                         "username": "root",                         "password": "ffcsict123",                         "path": [                             "/root/datax/datax.txt","/root/datax/datax.csv"                         ],                         "column": [                             {                                 "index": 0,                                 "type": "string"                             },                             {                                 "index": 1,                                 "type": "string"                             },                             {                                 "index": 2,                                 "type": "string",                                                           }                         ],                         "encoding": "GBK",                         "fieldDelimiter": ",",                         "skipHeader": "true"                     }                 },                 "writer": {                     "name": "mysqlwriter",                     "parameter": {                         "writeMode": "insert",                         "username": "root",                         "password": "tiger",                         "column": [                             "id",                             "name",                             "time"                         ],                         "session": [                             "set session sql_mode='ANSI'"                         ],                         "preSql": [                             "delete from datax1"                         ],                         "connection": [                             {                                 "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/datax?useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&useSSL=false&serverTimezone=GMT%2b8",                                 "table": [                                     "datax1"                                 ]                             }                         ]                     }                 }             }         ]     } }

三、安装datax-web

3.1 下载 data-web源码：https://github.com/WeiYe-Jing/datax-web

3.2 创建数据库：脚本文件路径–>datax-web\bin\db

3.3 修改datax-admin下配置文件resources/application.yml

修改数据源配置，目前仅支持mysql

1 2 3 4 5 6

#数据源   datasource:     username: root     password: root     url: jdbc:mysql://localhost:3306/datax_web?serverTimezone=Asia/Shanghai&useLegacyDatetimeCode=false&useSSL=false&nullNamePatternMatchesAll=true&useUnicode=true&characterEncoding=UTF-8     driver-class-name: com.mysql.jdbc.Driver

修改日志路径path

1 2 3 4 5

# 配置mybatis-plus打印slq日志 logging:   level:     com.wugui.datax.admin.mapper: error   path: ./data/applogs/admin

修改邮件发送配置(不需要可以不修改)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

# datax-web email mail:   host: smtp.qq.com   port: 25   username: [email protected]   password: xxx   properties:     mail:       smtp:         auth: true         starttls:           enable: true           required: true       socketFactory:         class: javax.net.ssl.SSLSocketFactory

3.4 修改datax-executor下配置文件resources/application.yml

修改日志路径path

1 2 3 4

# log config logging:   config: classpath:logback.xml   path: ./data/applogs/executor/jobhandler

修改datax.job配置

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

datax:   job:     admin:       ### datax-web admin address       addresses: http://127.0.0.1:8080     executor:       appname: datax-executor       ip:       port: 9999       ### job log path       logpath: ./data/applogs/executor/jobhandler       ### job log retention days       logretentiondays: 30   executor:     jsonpath: D:\java\Projects\sync-data\json      pypath: D:\java\Projects\sync-data\datax\bin\datax.py

• admin.addresses datax_admin部署地址，如调度中心集群部署存在多个地址则用逗号分隔，执行器将会使用该地址进行"执行器心跳注册"和"任务结果回调"；
• executor.appname 执行器AppName，每个执行器机器集群的唯一标示,执行器心跳注册分组依据；
• executor.ip 默认为空表示自动获取IP，多网卡时可手动设置指定IP，该IP不会绑定Host仅作为通讯实用；地址信息用于 "执行器注册" 和 "调度中心请求并触发任务"；
• executor.port 执行器Server端口号，默认端口为9999，单机部署多个执行器时，注意要配置不同执行器端口；
• executor.logpath 执行器运行日志文件存储磁盘路径,需要对该路径拥有读写权限;
• executor.logretentiondays 执行器日志文件保存天数,过期日志自动清理, 限制值大于等于3时生效; 否则, 如-1, 关闭自动清理功能；
• executor.jsonpath datax json临时文件保存路径
• pypath DataX启动脚本地址，例如：xxx/datax/bin/datax.py

如果系统配置DataX环境变量（DATAX_HOME）logpath、jsonpath、pypath可不配，log文件和临时json存放在环境变量路径下。

四、启动项目

(1) 本地idea开发环境

1. 运行datax-admin下 DataXAdminApplication
2. 运行datax-executor下 DataXExecutorApplication

 

 

admin启动成功后日志会输出三个地址，两个接口文档地址，一个前端页面地址

(2) Linux环境部署

• 1.本地安装好maven环境，安装此处细节忽略
• 2.执行mvn package -Dmaven.test.skip=true
• 3.打包成功后分别将datax-admin、datax-executor模块target下datax-admin-2.1.1.jar、datax-executor-2.1.1.jar放到指定目录
• 4.分别启动datax-admin-1.0.0.jar、datax-executor-1.0.0.jar
• 5.启动命令demo：

  1 2	nohup java -Xmx1024M -Xms1024M -Xmn448M -XX:MaxMetaspaceSize=192M -XX:MetaspaceSize=192M -jar datax-admin-2.1.1.jar& nohup java -Xmx1024M -Xms1024M -Xmn448M -XX:MaxMetaspaceSize=192M -XX:MetaspaceSize=192M -jar datax-executor-2.1.1.jar&

五、启动成功

启动成功后打开页面（默认管理员用户名：admin 密码：123456），http://localhost:8080/index.html#/dashboard

六、集群部署

调度中心、执行器支持集群部署，提升调度系统容灾和可用性。

• 1.调度中心集群：

  DB配置保持一致；
  集群机器时钟保持一致（单机集群忽视）；

• 2.执行器集群:

  执行器回调地址(admin.addresses）需要保持一致；执行器根据该配置进行执行器自动注册等操作。

  同一个执行器集群内AppName（executor.appname）需要保持一致；调度中心根据该配置动态发现不同集群的在线执行器列表。