spark sql 本地调试_scala spark sql 本地调试

对于习惯了sql的开发同学来说，写sql肯定比较用map，filter内在算法因子要顺手的多。

一，sbt项目

1，build.sbt配置

name := "scalatest"

version := "0.1"

scalaVersion := "2.11.8"

libraryDependencies += "com.alibaba" % "fastjson" % "1.2.49"

libraryDependencies ++= Seq(

"org.apache.spark" % "spark-core_2.11" % "2.3.0",

"org.apache.spark" % "spark-hive_2.11" % "2.3.0",

"org.apache.spark" % "spark-sql_2.11" % "2.3.0"

)

spark-core，spark-hive，spark-sql的版本，根据自己的实际情况来定。

2，测试代码

package ex

import org.apache.spark.sql.SparkSession

object tank {

var data = ""

def main(args: Array[String]): Unit = {

val spark = SparkSession.builder().

master("local")

// .config("spark.sql.hive.thriftServer.singleSession", true)

.enableHiveSupport()

.appName("tanktest").getOrCreate()

import spark.implicits._

val tanktest:String = "create table `tank_test` ("+

"`creative_id` string,"+

"`category_name` string,"+

"`ad_keywords` string,"+

"`creative_type` string,"+

"`inventory_type` string,"+

"`gender` string,"+

" `source` string,"+

" `advanced_creative_title` string,"+

" `first_industry_name` string,"+

" `second_industry_name` string)"+

" ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED AS TEXTFILE";

//获取参数

for(i

args(i) match {

case "--data" => data=args(i+1);

case _ => "error";

}

}

spark.sql(tanktest)

spark.sql(s"LOAD DATA LOCAL INPATH '$data/creat_partd' INTO TABLE tank_test")

spark.sql("select count(*) as total from tank_test").show()

}

}

如果报:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Hive support is required to CREATE Hive TABLE (AS SELECT);

解决办法有二种：

//以下二步，二选一

// .config("spark.sql.hive.thriftServer.singleSession", true)

.enableHiveSupport()

3，idea debug配置

spark-sql 本地debug配置

4，调式结果

spark-sql 本debug测试

spark-sql本地调度结果

注意：本地调试，并没有连接远程的hive。也没有设置hive.metastore.warehouse.dir，所有元数据目录，以及数据目录，都在当前项目目录下了。

二，mvn项目

1，pom.xml添加以下内容

org.apache.spark

spark-core_2.11

2.3.0

org.apache.spark

spark-hive_2.11

2.3.0

org.apache.spark

spark-sql_2.11

2.3.0

其他的，根上面一样。这种本地开发，只能让代码中的sql运行起来，没有数据。数据只能从线上copy，下一篇，会说一说，本地spark-sql怎么连接线上的hive。