Spark基础

一。Spark基础架构

二：Spark内置模块

三 Spark Shell
1.yarn application -list
def flatMap[U](f: String => TraversableOnce[U])(implicit evidence$4: scala.reflect.ClassTag[U]): org.apache.spark.rdd.RDD[U]
sc.textFile(“in”).flatMap(x=>x.split(" ")).map((x)=>{(x,1)}).groupByKey.map(x=>{(x._1,x._2.size)}).collect
reduceByKey= groupBy+reduce 通过key聚合 先分组 再聚合 必须先转换成map

sc.textFile(“in”).flatMap(x=>x.split(" ")).map(x=>{(x,1)}).reduceByKey((x,y)=>(x+y)).collect

http://hadoop203:4040/ spark端口
http://hadoop203:8080 yarn端口

1.client