使用hive访问elasticsearch的数据

使用hive访问elasticsearch的数据

1.配置
将elasticsearch-hadoop-2.1.1.jar拷贝到hive/lib

hive -hiveconf hive.aux.jars.path=/usr/local/hive-1.2.1/lib/elasticsearch-hadoop-2.1.1.jar

或者配置：
hive-site.xml

<property>
  <name>hive.aux.jars.path</name>
  <value>file:///usr/local/hive-1.2.1/lib/elasticsearch-hadoop-2.1.1.jar</value>
  <description>A comma separated list (with no spaces) of the jar files</description>
</property>

2.创建外表

CREATE EXTERNAL TABLE achi_ex(
vid string,
dtime timestamp,
platid bigint)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘\t’
LINES TERMINATED BY ‘\n’
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(‘es.resource’ = ‘db_1/achi’,
‘es.index.auto.create’ = ‘true’);

*hive 中的timestamp和ES中的date相互映射

3.导入数据
insert overwrite table achi_ex select * from achi;

4.测试

select * from achi_ex limit 10;

错误：

Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.io.IntWritable

hive表中field类型为int时，映射到es中变成long，所以会报此错误。将hive表中int改为bigint即可。

CREATE EXTERNAL TABLE achi_ex(
vid string,
dtime timestamp,
platid bigint)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ‘\t’
LINES TERMINATED BY ‘\n’
STORED BY ‘org.elasticsearch.hadoop.hive.EsStorageHandler’
TBLPROPERTIES(‘es.resource’ = ‘db_1/achi’,
‘es.mapping.names’ = ‘vid:vID,dtime:dTime,platid:PlatID
‘);