整合spark和hive
参考1
参考2
事先启动了hive-metastore服务
启动了hadoop
启动了spark集群(若不启动集群，下面不要指定master也是可以的)
接着启动spark-shell
bin/spark-shell --master spark://moon:7077 --driver-class-path /usr/local/hive/lib/mysql-connector-java-5.1.18-bin.jar使用HiveContext:
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") sqlContext.sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src") // Queries are expressed in HiveQL sqlContext.sql("FROM src SELECT key, value").collect().foreach(println)输出：
15/08/29 14:56:34 INFO scheduler.DAGScheduler: Job 0 finished: collect at <console>:24, took 2.802633 s
[238,val_238]
[86,val_86]
[311,val_311]
[27,val_27]
[165,val_165]
[409,val_409]
[255,val_255]
[278,val_278]
[98,val_98]
[484,val_484]
[265,val_265]
[193,val_193]
[401,val_401]
[150,val_150]
[273,val_273]
[224,val_224]
[369,val_369]
[66,val_66]
[128,val_128]
[213,val_213]
[146,val_146]
[406,val_406]
[429,val_429]
[374,val_374]
[152,val_152]
[469,val_469]
[145,val_145]
[495,val_495]
。。。。。。查看表
scala> sqlContext.sql("show tables").collect()
15/08/29 15:26:06 INFO metastore.HiveMetaStore: 0: get_tables: db=default pat=.*
15/08/29 15:26:06 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_tables: db=default pat=.*
res10: Array[org.apache.spark.sql.Row] = Array([sogouq1,false], [sogouq2,false], [src,false], [t_hadoop,false], [t_hive2,false])
此时在hive中出现了一个表src:
hive> show tables;
OK
sogouq1
sogouq2
src
t_hadoop
t_hive2
hive> select * from src limit 5
> ;
OK
238 val_238
86 val_86
311 val_311
27 val_27
165 val_165
Time taken: 0.744 seconds, Fetched: 5 row(s)
同时hdfs的/user/hive/warehouse/src/kv1.txt出现