软件版本：elasticsearch版本（6.3.2）
数据迁移方案
datax导入数据到elasticsearch步骤：
- 安装elasticsearchwriter插件，安装教程：
- 从datax的github地址（https://github.com/alibaba/DataX
）下载源码工程（Datax-master） - 修改pom.xml，配置modules模块,按需保留elasticsearchwriter模块
- 从datax的github地址（https://github.com/alibaba/DataX
common
core
transformer
mysqlreader
elasticsearchwriter
plugin-rdbms-util
plugin-unstructured-storage-util
hbase20xsqlreader
hbase20xsqlwriter
打包，mvn clean install -Dmaven.test.skip=true，复制当前目录下的/elasticsearchwriter/target/datax/plugin/writer/elasticsearchwriter目录到datax的plugin目录下
配置datax脚本：
参数含义参照：
https://help.aliyun.com/knowledge_detail/61990.html
{
"job": {
"setting": {
"speed": {
"channel": 1
},
"errorLimit": {
"percentage": 0
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "xxx",
"password": "xxx",
"connection": [{
"querySql": ["数据查询SQL "],
"jdbcUrl": ["数据库地址"]
}]
}
},
"writer": {
"name": "elasticsearchwriter",
"parameter": {
"endpoint":"es的地址：xxx:9200",
"accessId": "连接权限配置，如果不需要权限认证，可随意配置，但不能为空，或者不配",
"accessKey": "连接权限配置，如果不需要权限认证，可随意配置",
"index": "xxx索引",
"type": "_doc",
"cleanup": false,
"discovery":false,
"column": [
{
"name": "id",
"type": "long"
},
{
"name": "userId",
"type": "long"
},
{
"name": "memberId",
"type": "long"
},
{
"name": "receiveTime",
"type": "date"
},
{
"name": "couponTemplateId",
"type": "long"
},
{
"name": "couponStatus",
"type": "keyword"
},
{
"name": "consumeTime",
"type": "date"
},
{
"name": "platformId",
"type": "long"
},
{
"name": "orderId",
"type": "keyword"
},
{
"name": "startDate",
"type": "date"
},
{
"name": "endDate",
"type": "date"
},
{
"name": "lockDate",
"type": "date"
},
{
"name": "reduceAmount",
"type": "double"
},
{
"name": "prodId",
"type": "long"
},
{
"name": "orderAmount",
"type": "double"
},
{
"name": "activityId",
"type": "long"
},
{
"name": "activityType",
"type": "keyword"
},
{
"name": "createUser",
"type": "keyword"
},
{
"name": "updateUser",
"type": "keyword"
},
{
"name": "createDatetime",
"type": "date"
},
{
"name": "updateDatetime",
"type": "date"
}
]
}
}
}
]
}
}
问题：
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException:
Code:[Framework-02], Description:
[DataX引擎运行过程出错，具体原因请参看DataX运行结束时的错误诊断信息 .].
- java.lang.IllegalArgumentException:
Preemptive authentication set without credentials provider
排查原因：
- datax脚本，elasticsearchwriter未配置accessId、accessKey，或者accessId、accessKey的值为空(""),如果es不需要权限认证，可以设置accessId、accessKey为任意非空值。
ERROR RetryUtil - Exception when calling callable,
异常Msg:No Server is assigned to client to connect‘
io.searchbox.client.config.exception.NoServerConfiguredException:
No Server is assigned to client to connect
排查原因：elasticsearchwriter的配置参数discovery:true导致，应修改为默认值false，该属性描述含义：启用节点发现将(轮询)并定期更新客户机中的服务器列表，官方es插件在发文前声明只在5.x的es版本号上测试，在6.x及以上的未给出声明。