理解RDD的reduceByKey与groupByKey

数据准备

val words = Array("a","a","b","c","c")
val conf = new SparkConf().setAppName("word-count").setMaster("local");
val sc = new SparkContext(conf)
val rdd = sc.parallelize(words)

reduceByKey方法

rdd.map((_,1)).reduceByKey(_+_).collect().foreach(println)

groupByKey方法

rdd.map((_,1)).groupByKey().map(word => (word._1, word._2.sum)).collect().foreach(println)

输出结果是一致的，我们查看API文档发现有如下描述，

reduceByKey
Merge the values for each key using an associative and commutative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce. Output will be hash-partitioned with the existing partitioner/ parallelism level.

groupByKey()
Group the values for each key in the RDD into a single sequence. Hash-partitions the resulting RDD with the existing partitioner/parallelism level. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting RDD is evaluated.

根据对比，我们发现reduceByKey方法在向reducer发送数据之前会先将数据按key进行合并，而groupByKey方法是直接对计算的RDD结果进行分区。

假设我们的数据文件分布在两个节点上，那么

reduceByKey工作图解

groupByKey工作图解