通过datax导入数据到elasticsearch

软件版本：elasticsearch版本（6.3.2）

数据迁移方案

datax导入数据到elasticsearch步骤：

• 安装elasticsearchwriter插件，安装教程：
  • 从datax的github地址（https://github.com/alibaba/DataX
    ）下载源码工程（Datax-master）
  • 修改pom.xml，配置modules模块,按需保留elasticsearchwriter模块

        common
        core
        transformer

        
        mysqlreader
        
        
        elasticsearchwriter

        
        plugin-rdbms-util
        plugin-unstructured-storage-util
        hbase20xsqlreader
        hbase20xsqlwriter

• 打包，mvn clean install -Dmaven.test.skip=true，复制当前目录下的/elasticsearchwriter/target/datax/plugin/writer/elasticsearchwriter目录到datax的plugin目录下

• 配置datax脚本：
  参数含义参照：
  https://help.aliyun.com/knowledge_detail/61990.html

{
    "job": {
      "setting": {
          "speed": {
              "channel": 1 
          },
          "errorLimit": {
              "percentage": 0
          }
      },
      "content": [
          {
            "reader": {
                "name": "mysqlreader",
                "parameter": {
                    "username": "xxx",
                    "password": "xxx",
                    "connection": [{
                        "querySql": ["数据查询SQL "],
                        "jdbcUrl": ["数据库地址"]
                    }]
                }
            },
              "writer": {
                  "name": "elasticsearchwriter",
                  "parameter": {
                      "endpoint":"es的地址：xxx:9200",
                      "accessId": "连接权限配置，如果不需要权限认证，可随意配置，但不能为空，或者不配",
                      "accessKey": "连接权限配置，如果不需要权限认证，可随意配置",
                      "index": "xxx索引", 
                      "type": "_doc",  
                      "cleanup": false,  
                      "discovery":false, 
                      "column": [  
                          {
                            "name": "id",  
                            "type": "long"  
                          },
                          {
                            "name": "userId",  
                            "type": "long"  
                          },
                          {
                            "name": "memberId",
                            "type": "long" 
                          },
                          {
                            "name": "receiveTime",
                            "type": "date"  
                        },
                        {
                            "name": "couponTemplateId",
                            "type": "long"  
                        },
                        {
                            "name": "couponStatus",  
                            "type": "keyword"  
                        },
                        {
                            "name": "consumeTime",
                            "type": "date" 
                        },
                        {
                            "name": "platformId",  
                            "type": "long"  
                        },
                        {
                            "name": "orderId",
                            "type": "keyword" 
                        },
                        {
                            "name": "startDate",
                            "type": "date"  
                        },
                        {
                            "name": "endDate",
                            "type": "date" 
                        },
                        {
                            "name": "lockDate",
                            "type": "date"  
                        },
                        {
                            "name": "reduceAmount",
                            "type": "double" 
                        },
                        {
                            "name": "prodId",  
                            "type": "long"  
                        },
                        {
                            "name": "orderAmount",
                            "type": "double"  
                        },

                          {
                              "name": "activityId",  
                              "type": "long"  
                          },
                          {
                              "name": "activityType",
                              "type": "keyword"  
                          },
                          {
                            "name": "createUser",  
                            "type": "keyword"  
                        },
                        {
                            "name": "updateUser",
                            "type": "keyword" 
                        },
                          
                        
                        
                        
                        {
                            "name": "createDatetime",
                            "type": "date" 
                        },
                        
                        {
                            "name": "updateDatetime",
                            "type": "date" 
                        }
                        
                      ]
                  }
              }
            }
        ]
        }
    }

问题：

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: 
Code:[Framework-02], Description:
[DataX引擎运行过程出错，具体原因请参看DataX运行结束时的错误诊断信息  .].
- java.lang.IllegalArgumentException: 
Preemptive authentication set without credentials provider

排查原因：

• datax脚本，elasticsearchwriter未配置accessId、accessKey，或者accessId、accessKey的值为空(""),如果es不需要权限认证，可以设置accessId、accessKey为任意非空值。

ERROR RetryUtil - Exception when calling callable, 
异常Msg:No Server is assigned to client to connect‘
io.searchbox.client.config.exception.NoServerConfiguredException: 
No Server is assigned to client to connect

排查原因：elasticsearchwriter的配置参数discovery:true导致，应修改为默认值false，该属性描述含义：启用节点发现将(轮询)并定期更新客户机中的服务器列表，官方es插件在发文前声明只在5.x的es版本号上测试，在6.x及以上的未给出声明。