Caused by: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.hive.EsHiveInputFormat

Caused by: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.hive.EsHiveInputFormat

Hive和Tez整合之后往ES中插入数据报错的问题

在我吧hive的引擎替换为tez以后,运行SQL报错,后来发现是往ES映射表的时候报错

Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1585893599059_0645_1_00, diagnostics=[Vertex vertex_1585893599059_0645_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
 

Caused by: java.lang.RuntimeException: Failed to load plan: 
        ... 30 more
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: org.elasticsearch.hadoop.hive.EsHiveInputFormat
Serialization trace:
inputFileFormatClass (org.apache.hadoop.hive.ql.plan.TableDesc)
tableInfo (org.apache.hadoop.hive.ql.plan.FileSinkDesc)
conf (org.apache.hadoop.hive.ql.exec.FileSinkOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)

Caused by: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.hive.EsHiveInputFormat
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
        ... 79 more
]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

原因:

这是因为我们在整合tez之后引入了本地tez的jar包

在hive的conf目录下的hive-env.sh添加了如下配置

export HIVE_AUX_JARS_PATH=/application/hive-2.3.6/lib/hadoop-lzo-0.4.20.jar$TEZ_JARS

引入了hive下的hadoop-lzo-0.4.20.jar 和 tez的jar包

而我在配置hive-site.xml中的配置失效了

<property>

        <name>hive.aux.jars.pathname>

        <value>file:///application/hive-2.3.6/lib/elasticsearch-hadoop-7.5.1.jarvalue>

        <description>A comma separated list (with no spaces) of the jar filesdescription>

property>

解决:

1.在插入es映射表的SQL前面加上

add jar /application/hive-2.3.6/lib/elasticsearch-hadoop-7.5.1.jar;

这个路径是你整合es之后hive下的lib下的es的jar包的地址

2.或者你可以在hive-site.xml里

<property>
    <name>hive.aux.jars.pathname>

    <value>file:///application/hive-2.3.6/lib/elasticsearch-hadoop-7.5.1.jarvalue>

    <description>A comma separated list (with no spaces) of the jar filesdescription>
    property>

A comma separated list (with no spaces) of the jar files

多添加几个jar包地址,以逗号分隔