spark读取hive

spark2.0+ 使用Sparksession替代HiveContext

1.添加MAVEN依赖

       
           mysql
           mysql-connector-java
           5.1.35
       

       
           org.apache.spark
           spark-hive_2.11
           2.1.1
       

       
       
           org.apache.spark
           spark-sql_2.11
           2.1.1
       


       
       
           org.apache.spark
           spark-core_2.11
           2.1.1
       

2.编写SparkSession

package com.hualala.bi

import java.io.File

import org.apache.spark.sql.SparkSession

/**
  *
  * @author jiaquanyu 
  *
  */
object SparkSqlApp {

  def main(args: Array[String]): Unit = {

    case class Record(key: Int, value: String)

    val warehouseLocation = new File("spark-warehouse").getAbsolutePath

    val spark = SparkSession
      .builder()
      .appName("Spark SQL on hive")
      .master("spark://192.168.4.4:7077")
      .config("spark.sql.warehouse.dir", warehouseLocation)
      .enableHiveSupport()
      .getOrCreate()

    spark.sql("show databases").collect().foreach(println)
  }
}

3.将hive-site.xml保存到项目中的resources下 没有需创建

注：如需debug 建议spark-shell下进行