Spark SQL读取hive数据时报找不到mysql驱动

Exception：

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "BoneCP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.

Solution:

1、$HIVE_HOME/conf/hive-site.xml中增加关于 hive.metastore.uris 的配置信息，如下：
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://namenode1:9083</value>
  <description>IP address (or fully-qualified domain name) and port of the metastore host</description>
</property>

2、执行:$HIVE_HOME/bin/hive --service metastore,启动元数据存储服务；

3、将$HIVE_HOME/conf/hive-site.xml拷贝至$SPARK_HOME/conf/目录下；

4、启动spark-shell进行验证:$SPARK_HOME/bin/spark-shell --master namenode1:7077或spark-sql -> show databases.

Note:
1. 当在Intellij IDE中编写Spark SQL程序时（val hiveContext = new HiveContext(sc); import hiveContext.sql; sql("show databases")），打包成相应的.jar文件，并利用如下脚本将任务提交到Spark集群运行时，Spark默认采用derby进行metastore，即元数据的存储；当再次在不同目录下执行该任务时，之前创建的数据库或表数据无法获取，有点即用即删的感觉。故要想访问Hive下的元数据，首先需要将Hive目录下的配置文件中的hive-site.xml文件放到Spark目录下的配置文件中，让Spark集群执行程序时能识别进入Hive元数据的路径，然后启动上述服务（hive --service metastore）即可访问Hive相应数据。

2.

/**

* An instance of the Spark SQL execution engine that integrates with data stored in Hive.

* Configuration for Hive is read from hive-site.xml on the classpath.

*/

class HiveContext(sc: SparkContext) extends SQLContext(sc) {



 ....................................



}

3. 

Use HiveContext instead.  It will still create a local metastore if one is not specified. However, note that the default directory is ./metastore_db, not ./metastore

 

测试程序如下：

package com.husor.Hive



import org.apache.spark.{SparkContext, SparkConf}

import org.apache.spark.sql.hive.HiveContext



/* Spark SQL执行时的sql是临时的，即用即删 **/



/**

 * Created by kelvin on 2015/1/27.

 */

object Recommendation {

  def main(args: Array[String]) {



    println("Test is starting......")



    if (args.length < 1) {

      System.err.println("Usage:HDFS_OutputDir <Directory>")

      System.exit(1)

    }



    //System.setProperty("hadoop.home.dir", "d:\\winutil\\")



    val conf = new SparkConf().setAppName("Recommendation")

    val spark = new SparkContext(conf)



    val hiveContext = new HiveContext(spark)



    import hiveContext.sql



    /*sql("create database if not exists baby")

    val databases = sql("show databases")

    databases.collect.foreach(println)*/



    sql("use baby")

    /*sql("CREATE EXTERNAL TABLE if not exists origin_orders (oid string, uid INT, gmt_create INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/beibei/order'")

    sql("CREATE EXTERNAL TABLE if not exists items (iid INT, pid INT, title string, cid INT, brand INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/beibei/item'")

    sql("CREATE EXTERNAL TABLE if not exists order_item (oid string, iid INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/beibei/order_item'")

    sql("create table if not exists test_orders(oid string, uid INT, gmt_create INT)")

    sql("create table if not exists verify_orders(oid string, uid INT, gmt_create INT)")

    sql("insert OVERWRITE table test_orders select * from origin_orders where gmt_create <= 1415635200")

    sql("insert OVERWRITE table verify_orders select * from origin_orders where gmt_create > 1415635200")



    val tables = sql("show tables")

    tables.collect.foreach(println)*/



    sql("SET spark.sql.shuffle.partitions = 5")



    val olderTime = System.currentTimeMillis()



    val userOrderData = sql("select i.pid, o.uid, o.gmt_create from items i " +

                                         "join order_item oi " +

                                         "on i.iid = oi.iid     " +

                                         "join test_orders o " +

                                         "on oi.oid = o.oid")



    userOrderData.take(10).foreach(println)



    val newTime = System.currentTimeMillis()



    println("Consume Time: " + (newTime - olderTime))



    userOrderData.saveAsTextFile(args(0))

    spark.stop()



    println("Test is Succeed!!!")



  }



}