使用阿里云DataX完成数据同步
DataX
DataX 是阿里云 DataWorks 数据集成的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS, databend 等各种异构数据源之间高效的数据同步功能。
官网：https://github.com/alibaba/DataX
环境准备
参考官网：https://github.com/alibaba/DataX/blob/master/userGuid.md
下载解压后
MongoDB同步数据至达梦数据库
新建xxx.json，写入如下内容
{
"job": {
"setting": {
"speed": {
"channel": 10
}
},
"content": [
{
"reader": {
"name": "mongodbreader",
"parameter": {
"address": ["192.168.2.39:27017"],
"userName": "",
"userPassword": "",
"dbName": "ztcx_test",
"collectionName": "gps_driver_location_20230718",
"column": [
{
"name": "longitude",
"type": "Double"
},
{
"name": "latitude",
"type": "Double"
},
{
"name": "location_type",
"type": "int"
},
{
"name": "type",
"type": "int"
},
{
"name": "driver_id",
"type": "Long"
},
{
"name": "plate_no",
"type": "string"
},
{
"name": "create_date",
"type": "Date",
}
]
}
},
"writer": {
"name": "rdbmswriter",
"parameter": {
"connection": [
{
"jdbcUrl": "jdbc:dm://192.168.2.27:5236/DATAX",
"table": [
"DRIVER_LOCATION"
]
}
],
"username": "xxx",
"password": "****",
"table": "DRIVER_LOCATION",
"column": [
"longitude",
"latitude",
"location_type",
"type",
"driver_id",
"plate_no",
"create_date"
],
"preSql": [
"delete from DRIVER_LOCATION;"
]
}
}
}
]
}
}
配置文件属性，可参考
切换至bin目录，执行以下命令即可
python datax.py ./xxx.json
