Spark_Union 算子与 Shuffle 流程

 

今天有被问到 UNION 算子会不会产生 Shuffle 。说实话，这个没有研究过，我们写个demo

 

代码 ：

package com.spark.test.offline

import org.apache.spark.{SparkConf, SparkContext}

/**
  * Created by szh on 2020/5/28.
  */
object TestUnion {

  def main(args: Array[String]): Unit = {

    var sparkConf = new SparkConf
    sparkConf
      .setAppName("Union data test")
      .setMaster("local[2]")

    val sparkContext = new SparkContext(sparkConf)
    sparkContext.setLogLevel("WARN")

    val arrayA = Array((1, "mm"), (2, "cs"), (3, "cc"))
    val arrayB = Array((4, "px"), (5, "kk"), (2, "cs"))

    val rddA = sparkContext.parallelize(arrayA).repartition(1)
    val rddB = sparkContext.parallelize(arrayB).repartition(3)

    val combRdd = rddA.union(rddB)

    println("rddA partition num :" + rddA.partitions.length)
    println("rddB partition num :" + rddB.partitions.length)
    println("combRdd partition num :" + combRdd.partitions.length)

    combRdd.collect().foreach(println(_))
    combRdd.count()

    Thread.sleep(120 * 1000)

    sparkContext.stop()

  }

}

代码输出

rddA partition num :1
rddB partition num :3
combRdd partition num :4
(1,mm)
(2,cs)
(3,cc)
(4,px)
(5,kk)
(2,cs)

 

可以看到，

1）Union 的前后并没有 Shuffle 的发生，

 

2）Union 后 partition 的数量为 Union的 RDD_A 与 RDD_B 数量总和。

 

3）Union ALL 算子并不进行去重，

    val arrayA = Array((1, "mm"), (2, "cs"), (3, "cc"))
    val arrayB = Array((4, "px"), (5, "kk"), (2, "cs"))

输出

(1,mm)
(2,cs)
(3,cc)
(4,px)
(5,kk)
(2,cs)