spark sql读取映射hbase数据的hive外部表报错

集群环境CDH5.8.0 / spark2.1.0

我们用执行以下命令报错:

spark2-submit --master yarn --class com.test.hive.SparkReadHbaseTest ./dacproject.jar 'SELECT count(*) FROM test' 'hdfs:///user/test'

其中test表是从HBASE映射过来的表

报错信息如下:
Exception in thread “main” java.lang.RuntimeException:
java.lang.ClassNotFoundException:org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat
spark sql读取映射hbase数据的hive外部表报错_第1张图片
查找网上方法,缺少包:
hbase-site.xml
hbase-protocol-1.2.0-cdh5.8.0.jar
hbase-client-1.2.0-cdh5.8.0.jar
hbase-common-1.2.0-cdh5.8.0.jar
hbase-server-1.2.0-cdh5.8.0.jar
hive-hbase-handler-1.1.0-cdh5.8.0.jar
metrics-core-2.2.0.jar
于是添加后报错:
spark sql读取映射hbase数据的hive外部表报错_第2张图片
查找后发现少了
htrace-core-3.2.0-incubating.jar

spark2-submit \
--master local[2] \
--driver-class-path /etc/hbase/conf/hbase-site.xml:/opt/cloudera/parcels/CDH/jars/hbase-protocol-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hbase-client-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hbase-common-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hbase-server-1.2.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/hive-hbase-handler-1.1.0-cdh5.8.0.jar:/opt/cloudera/parcels/CDH/jars/metrics-core-2.2.0.jar:/opt/cloudera/parcels/CDH/jars/htrace-core-3.2.0-incubating.jar \
--class com.lhx.hive.SparkReadHbaseTest \
./dacproject.jar 'SELECT * FROM test' 'hdfs:///user/test'

最终执行成功!
注:以上是local模式运行,如果要在yarn模式运行,需要每台集群都执行命令:

cp /etc/hbase/conf/hbase-site.xml /opt/cloudera/parcels/SPARK2/lib/spark2/conf
cp /opt/cloudera/parcels/CDH/jars/hbase-protocol-1.2.0-cdh5.8.0.jar cp /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hbase-client-1.2.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hbase-common-1.2.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hbase-server-1.2.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/hive-hbase-handler-1.1.0-cdh5.8.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/metrics-core-2.2.0.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars
cp /opt/cloudera/parcels/CDH/jars/htrace-core-3.2.0-incubating.jar /opt/cloudera/parcels/SPARK2/lib/spark2/jars

最终才算解决问题。

你可能感兴趣的:(CDH)