hive2elasticsearch

申明：这个是自己在用到这个插件时碰到的一点坑坑，有不同意见的，敬请告知，欢迎讨论。

一，Hive向ElasticSearch中写数据（基于elasticsearch-Hadoop的6.4.0.jar）

• 1.引入的jar包

      将elasticsearch-Hadoop的6.4.0.jar导入hive的jar包目录下或者hive的auxlib下

     add jar /opt/apache-hive-3.1.0-bin/auxlib/elasticsearch-hadoop-6.4.0.jar;

• 2.创建ES索引

//建立索引
curl -X PUT "localhost:9200/hive2es" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "number_of_shards" : 3,
		"number_of_replicas": 2
    },
    "mappings" : {
        "detail" : {
            "properties" : {
				"name":{"type":"keyword"},
				"value":{"type":"keyword"},
				"time":{"type":"long"}
            }
        }
    }
}'
//查看索引
curl -X GET "localhost:9200/_cat/indices?v"

• 2.创建hive表

CREATE external TABLE es_hive2es(
name string,
value string,                 
time bigint
)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'hive2es/detail','es.nodes'='192.168.111.115:9200','es.mapping.id' = 'name','es.index.auto.create' = 'false');

• 3.将数据插入hive

//hive2es是需要插入到es的表的原始数据
insert into es_hive2es select * from hive2es;

• 4.查询数据

curl -X GET "localhost:9200/hive2es/_search?pretty"
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "hive2es",
        "_type" : "detail",
        "_id" : "lotusyu",                
        "_score" : 1.0,
        "_source" : {
          "address" : "lotusyu",             
          "value" : 500,
          "time" : 1232882841
        }
      }
 
    ]
  }
}

二，运用Java spring模板读取es中的数据

坑点：字段名是下划线时用的查询会报错，字段时大写，在上课中会自动生成相对应的小写的字段名字。

• 1.创建对象

在创建DOMIN时，指定索引名字和类型

import org.springframework.data.elasticsearch.annotations.Document;

@Document(indexName = "hive2es", type = "detail",createIndex = false)
public class Test {


    private String address;

    private String value;

    private Long time;
}

• 2.调用找到方法

调用找到方法，根据字段查询并返回数据。

import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;

public interface TestRepository extends ElasticsearchRepository {

    Page findByAddress(String address, Pageable pageable);

    List findByAddress(String address);

    List findByReleaseTimeBetween(Long startTime,Long endTime);

}

• 3.字段信息涉及到下划线

当字段信息涉及到下划线的时候，可以用queryForPage

NativeSearchQueryBuilder nbq = new NativeSearchQueryBuilder().withIndices(Test.INDEX).withTypes(Test
        .ORDER_TYPE).withSearchType(SearchType.DEFAULT).withPageable(PageRequest.of(0, 1000000000, Sort.Direction.ASC,"time"));
BoolQueryBuilder bqb = boolQuery();
//过滤地址
if (address != null && !"".equals(address)) {
    bqb.must(termQuery("test_address", address));
}
 Page result = elasticsearchTemplate.queryForPage(nbq.withQuery(bqb).build(), Test.class);

备注：

也可以根据ES提供的对外查询接口查询数据。