spark jar包方式运行hive步骤

1,用maven install 打包
2,配置好下面的脚本,其中--driver-class-path要选择mysql的jar包所在的目录,另外有博客说可以在spark-env指定
export SPARK_CLASSPATH=$SPARK_HOME/lib/mysql-connector-java-5.0.8-bin.jar
-----注意配置了--driver-class-path 就不能再配spark-env不然,启动会报错:
Exception in thread "main" org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.

----但是用spark-env方式配置,运行还是报错找不断mysql驱动:

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

run-sql.sh内容:
#!/bin/sh
FWDIR="$SPARK_HOME"
$FWDIR/bin/spark-submit \
--class zfh.spark.study.hive.HiveDataSource \
--num-executors 3 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 3 \
--files /home/zfh/apache/hive-2.0.0/conf/hive-site.xml \
--driver-class-path /home/zfh/apache/hive-2.0.0/lib/mysql-connector-java-5.0.8-bin.jar \
/home/zfh/runjar/spark-study-java-0.0.1.jar \

你可能感兴趣的:(Spark,Hive)