Spark 的键值对（pair RDD）操作，Scala实现
一：什么是Pair RDD？
Spark为包含键值对对类型的RDD提供了一些专有操作，这些操作就被称为Pair RDD，Pair RDD是很多程序的构成要素，因为它们提供了并行操作对各个键或跨节点重新进行数据分组的操作接口。
二：Pair RDD的操作实例
1：创建Pair RDD
在saprk中有很多种创建pairRDD的方式，很多存储键值对的数据格式会在读取时直接返回由其键值对数据组成的pair RDD，此外需要把一个普通的RDD转化为pair RDD时，可以调用map函数来实现，传递的函数需要返回键值对。
scala> var lines = sc.parallelize(List("i love you"))
lines: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[8] at parallelize at :27
scala> val pairs = lines.map(x=>(x,1))
pairs: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[9] at map at :29
scala> pairs.foreach(println)
(i love you,1)
2：Pai RDDr的转化操作
由于pair RDD中包含二元组，所以需要传递函数应当操作二元组而不是独立的元素，假设键值对集合为{(1,2),(3,4),(3,6)}
rdd.reduceByKey(func)：合并具有相同key的value值
scala> val rdd = sc.parallelize(List((1,2),(3,4),(3,6)))
rdd: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[12] at parallelize at :27
scala> val result = rdd.reduceByKey((x,y)=>x+y)
result: org.apache.spark.rdd.RDD[(Int, Int)] = ShuffledRDD[14] at reduceByKey at :29
scala> result.foreach(println)
(1,2)
(3,10)
scala> val result = rdd.groupByKey()
result: org.apache.spark.rdd.RDD[(Int, Iterable[Int])] = ShuffledRDD[1] at groupByKey at :29
scala> result.foreach(println)
(3,CompactBuffer(4, 6))
(1,CompactBuffer(2))
scala> val result = rdd.mapValues(x=>x+1)
result: org.apache.spark.rdd.RDD[(Int, Int)] = MapPartitionsRDD[2] at mapValues at :29
scala> result.foreach(println)
(1,3)
(3,5)
(3,7)
scala> val result = rdd.flatMapValues(x=>(x to 5))
result: org.apache.spark.rdd.RDD[(Int, Int)] = MapPartitionsRDD[3] at flatMapValues at :29
scala> result.foreach(println)
(3,4)
(3,5)
(1,2)
(1,3)
(1,4)
(1,5)
scala> val result = rdd.keys
result: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[4] at keys at :29
scala> result.foreach(println)
3
1
3
scala> val result = rdd.values
result: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[6] at values at :29
scala> result.foreach(println)
2
4
6
rdd.sortByKey()：返回一个根据键排序的RDD
数据排序，可以通过接受ascending的参数表示我们是否想要结果按升序排序（默认是true）
scala> val result = rdd.sortByKey().collect()
result: Array[(Int, Int)] = Array((1,2), (3,4), (3,6))
scala> result
res8: Array[(Int, Int)] = Array((1,2), (3,4), (3,6))
scala> val result = rdd.sortByKey(ascending=false).collect()
result: Array[(Int, Int)] = Array((3,4), (3,6), (1,2))
3：针对两个pair RDD 的转化操作
| 函数名 | 目的 | 示例 | 结果 |
| substractByKey | 删掉RDD中键与other RDD中的键相同的元素 | rdd.subtractByKey(other) | {（1,2）} |
| join | 对两个RDD进行内连接 | rdd.join(other) | {(3,(4,9)),(3,(6,9))} |
| rightOuterJoin | 对两个RDD进行连接操作，右外连接 | rdd.rightOuterJoin(other) | {(3,(4,9)),(3,(6,9))} |
| leftOuterJoin | 对两个RDD进行连接操作，左外连接 | rdd.rightOuterJoin(other) | {(1,(2,None)),(3,(4,9)),(3,(6,9))} |
| cogroup | 将两个RDD中拥有相同键的数据分组 | rdd.cogroup(other) | {1,([2],[]),(3,[4,6],[9])} |
假设：rdd={(1,2),(3,4),(3,6)} other={(3,9)}
rdd.subtractByKey( other )：删除掉RDD中与other RDD中键相同的元素
scala> val rdd = sc.parallelize(List((1,2),(3,4),(3,6)))
rdd: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[4] at parallelize at :27
scala> val other = sc.parallelize(List((3,9)))
other: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[5] at parallelize at :27
scala> val result = rdd.subtractByKey(other)
result: org.apache.spark.rdd.RDD[(Int, Int)] = SubtractedRDD[6] at subtractByKey at :31
scala> result.foreach(println)
(1,2)
scala> val result = rdd.join(other)
result: org.apache.spark.rdd.RDD[(Int, (Int, Int))] = MapPartitionsRDD[12] at join at :31
scala> result.foreach(println)
(3,(4,9))
(3,(6,9))
scala> val result = rdd.rightOuterJoin(other)
result: org.apache.spark.rdd.RDD[(Int, (Option[Int], Int))] = MapPartitionsRDD[15] at rightOuterJoin at :31
scala> result.foreach(println)
(3,(Some(4),9))
(3,(Some(6),9))
scala> val result = rdd.leftOuterJoin(other)
result: org.apache.spark.rdd.RDD[(Int, (Int, Option[Int]))] = MapPartitionsRDD[18] at leftOuterJoin at :31
scala> result.foreach(println)
(3,(4,Some(9)))
(3,(6,Some(9)))
(1,(2,None))
scala> val result = rdd.cogroup(other)
result: org.apache.spark.rdd.RDD[(Int, (Iterable[Int], Iterable[Int]))] = MapPartitionsRDD[20] at cogroup at :31
scala> result.foreach(println)
(1,(CompactBuffer(2),CompactBuffer()))
(3,(CompactBuffer(4, 6),CompactBuffer(9)))
4：过滤操作
这里假设rdd={(1,2),(3,4),(3,6)}
对value做控制，key不加限制条件
scala> val result = rdd.filter{case(x,y)=>y%3==0}
result: org.apache.spark.rdd.RDD[(Int, Int)] = MapPartitionsRDD[22] at filter at :29
scala> result.foreach(println)
(3,6)
scala> val result = rdd.filter{case(x,y)=>y<=4}
result: org.apache.spark.rdd.RDD[(Int, Int)] = MapPartitionsRDD[23] at filter at :29
scala> result.foreach(println)
(1,2)
(3,4)
scala> 对key做控制，value不控制
scala> val result = rdd.filter{case(x,y)=>x<3}
result: org.apache.spark.rdd.RDD[(Int, Int)] = MapPartitionsRDD[24] at filter at :29
scala> result.foreach(println)
(1,2)
5：聚合操作
使用reduceByKey()和mapValues()计算每个键对应的平均值
scala> val rdd = sc.parallelize(List(Tuple2("panda",0),Tuple2("pink",3),Tuple2("pirate",3),Tuple2("panda",1),Tuple2("pink",4)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[45] at parallelize at :27
scala> val result = rdd.mapValues(x=>(x,1)).reduceByKey((x,y)=>(x._1+y._1,x._2+y._2))
result: org.apache.spark.rdd.RDD[(String, (Int, Int))] = ShuffledRDD[47] at reduceByKey at :29
scala> result.foreach(println)
(pirate,(3,1))
(panda,(1,2))
(pink,(7,2))
scala> val rdd = sc.parallelize(List("i am thinkgamer, i love cyan"))
rdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[49] at parallelize at :27
scala> val words = rdd.flatMap(line => line.split(" "))
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[50] at flatMap at :29
scala> val result = words.map(x=>(x,1)).reduceByKey((x,y) => x+y)
result: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[52] at reduceByKey at :31
scala> result.foreach(println)
(cyan,1)
(love,1)
(thinkgamer,,1)
(am,1)
(i,2)
scala> val rdd = sc.parallelize(List("i am thinkgamer, i love cyan"))
rdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at parallelize at :27
scala> val result = rdd.flatMap(x=>x.split(" ")).countByValue()
result: scala.collection.Map[String,Long] = Map(am -> 1, thinkgamer, -> 1, i -> 2, love -> 1, cyan -> 1)
scala> result.foreach(println)
(am,1)
(thinkgamer,,1)
(i,2)
(love,1)
(cyan,1)
scala> 并行度调优
每个RDD都有固定数目的分区，分区数决定了在RDD 上执行操作时的并行度，在执行聚合或分组函数时，可以要求Spark使用给定的分区，Spark始终尝试根据集群的大小，推断出一个有意义的默认值，但是有时候你可能要对并行度进行调优来获取更好的性能发展。
scala中自定义reduceByKey()的并行度
val data = Seq(("a",3),("b",4),("c",5))
sc.parallelize(data).reduceByKey((x,y)=>x+y) //默认并行度
sc.parallelize(data).reduceByKey((x,y)=>x+y,10) //自定义并行度6：pair RDD的行动操作
和转化操作一样，所有基础RDD支持的传统行动操作也都在pair RDD上可用，pair RDD提供了一些额外的行动操作，可以让我们充分利用数据的键值对特性，如下
以键值对集合{（1，2），（3，4），（3，6）}为例
| 函数名 | 描述 | 示例 | 结果 |
| countByKey | 对每个键对应的元素分别计数 | rdd.countByKey(other) | {(1,1),(3,2)} |
| collectAsMap() | 将结果以映射表的形式返回，以便查询 | rdd.collectAsMap() | Map{(1,2),(3,4),(3,6)} |
| lookup(key) | 返回给定键对应的所有值 | rdd.lookup(3) | [4,6] |
7：获取RDD的分区方式
scala> val pairs = sc.parallelize(List((1,1),(2,2),(3,3)))
pairs: org.apache.spark.rdd.RDD[(Int, Int)] = ParallelCollectionRDD[9] at parallelize at :27
scala> pairs.partitioner
res4: Option[org.apache.spark.Partitioner] = None
scala> val partitioned = pairs.partitionBy(new org.apache.spark.HashPartitioner(2))
partitioned: org.apache.spark.rdd.RDD[(Int, Int)] = ShuffledRDD[10] at partitionBy at :29
scala> partitioned.partitioner
res5: Option[org.apache.spark.Partitioner] = Some(org.apache.spark.HashPartitioner@2)