SparkStreaming之窗口函数




WindowOperations(窗口操作)

         Spark还提供了窗口的计算,它允许你使用一个滑动窗口应用在数据变换中。下图说明了该滑动窗口。


如图所示,每个时间窗口在一个个DStream中划过,每个DSteam中的RDD进入Window中进行合并,操作时生成为

窗口化DSteamRDD。在上图中,该操作被应用在过去的3个时间单位的数据,和划过了2个时间单位。这说明任

何窗口操作都需要指定2个参数。

  1. window length(窗口长度):窗口的持续时间(上图为3个时间单位)
  2. sliding interval (滑动间隔)- 窗口操作的时间间隔(上图为2个时间单位)。

上面的2个参数的大小,必须是接受产生一个DStream时间的倍数

让我们用一个例子来说明窗口操作。比如说,你想用以前的WordCount的例子,来计算最近30s的数据的中的单词

数,10S接受为一个DStream。为此,我们要用reduceByKey操作来计算最近30s数据中每一个DSteam中关于

(word,1)的pair操作。它可以用reduceByKeyAndWindow操作来实现。一些常见的窗口操作如下。所有这些操作

都需要两个参数— window length(窗口长度)和sliding interval(滑动间隔)。


————————-实验数据———————————————————————-

spark
Streaming
better
than
storm
you
need
it
yes
do
it

(每秒在其中随机抽取一个,作为Socket端的输入),socket端的数据模拟和实验函数等程序见附录百度云链接

———————————————–window操作————————————————————————-


   
   
   
   
  1. //输入:窗口长度(隐:输入的滑动窗口长度为形成Dstream的时间)
  2. //输出:返回一个DStream,這个DStream包含這个滑动窗口下的全部元素
  3. def window(windowDuration: Duration): DStream[T] = window(windowDuration, this.slideDuration)
  4. //输入:窗口长度和滑动窗口长度
  5. //输出:返回一个DStream,這个DStream包含這个滑动窗口下的全部元素
  6. def window(windowDuration: Duration, slideDuration: Duration): DStream[T] = ssc.withScope {
  7. new WindowedDStream( this, windowDuration, slideDuration)
  8. }


   
   
   
   
  1. import org.apache.log4j.{Level, Logger}
  2. import org.apache.spark.streaming.{Seconds, StreamingContext}
  3. import org.apache.spark.{SparkConf, SparkContext}
  4. object windowOnStreaming {
  5. def main(args: Array[String]) {
  6. /**
  7. * this is test of Streaming operations-----window
  8. */
  9. Logger.getLogger( "org.apache.spark").setLevel(Level.ERROR)
  10. Logger.getLogger( "org.eclipse.jetty.Server").setLevel(Level.OFF)
  11. val conf = new SparkConf().setAppName( "the Window operation of SparK Streaming").setMaster( "local[2]")
  12. val sc = new SparkContext(conf)
  13. val ssc = new StreamingContext(sc,Seconds( 2))
  14. //set the Checkpoint directory
  15. ssc.checkpoint( "/Res")
  16. //get the socket Streaming data
  17. val socketStreaming = ssc.socketTextStream( "master", 9999)
  18. val data = socketStreaming.map(x =>(x, 1))
  19. //def window(windowDuration: Duration): DStream[T]
  20. val getedData1 = data.window(Seconds( 6))
  21. println( "windowDuration only : ")
  22. getedData1.print()
  23. //same as
  24. // def window(windowDuration: Duration, slideDuration: Duration): DStream[T]
  25. //val getedData2 = data.window(Seconds(9),Seconds(3))
  26. //println("Duration and SlideDuration : ")
  27. //getedData2.print()
  28. ssc.start()
  29. ssc.awaitTermination()
  30. }
  31. }



——————–reduceByKeyAndWindow操作——————————–


   
   
   
   
  1. /**通过对每个滑动过来的窗口应用一个reduceByKey的操作,返回一个DSream,有点像
  2. * `DStream.reduceByKey(),但是只是這个函数只是应用在滑动过来的窗口,hash分区是采用spark集群
  3. * 默认的分区树
  4. * @param reduceFunc 从左到右的reduce 函数
  5. * @param windowDuration 窗口时间
  6. * 滑动窗口默认是1个batch interval
  7. * 分区数是是RDD默认(depend on spark集群core)
  8. */
  9. def reduceByKeyAndWindow(
  10. reduceFunc: (V, V) => V,
  11. windowDuration: Duration
  12. ): DStream[(K, V)] = ssc.withScope {
  13. reduceByKeyAndWindow(reduceFunc, windowDuration, self.slideDuration, defaultPartitioner())
  14. }
  15. /**通过对每个滑动过来的窗口应用一个reduceByKey的操作,返回一个DSream,有点像
  16. * `DStream.reduceByKey(),但是只是這个函数只是应用在滑动过来的窗口,hash分区是采用spark集群
  17. * 默认的分区树
  18. * @param reduceFunc 从左到右的reduce 函数
  19. * @param windowDuration 窗口时间
  20. * @param slideDuration 滑动时间
  21. */
  22. def reduceByKeyAndWindow(
  23. reduceFunc: (V, V) => V,
  24. windowDuration: Duration,
  25. slideDuration: Duration
  26. ): DStream[(K, V)] = ssc.withScope {
  27. reduceByKeyAndWindow(reduceFunc, windowDuration, slideDuration, defaultPartitioner())
  28. }
  29. /**通过对每个滑动过来的窗口应用一个reduceByKey的操作,返回一个DSream,有点像
  30. * `DStream.reduceByKey(),但是只是這个函数只是应用在滑动过来的窗口,hash分区是采用spark集群
  31. * 默认的分区树
  32. * @param reduceFunc 从左到右的reduce 函数
  33. * @param windowDuration 窗口时间
  34. * @param slideDuration 滑动时间
  35. * @param numPartitions 每个RDD的分区数.
  36. */
  37. def reduceByKeyAndWindow(
  38. reduceFunc: (V, V) => V,
  39. windowDuration: Duration,
  40. slideDuration: Duration,
  41. numPartitions: Int
  42. ): DStream[(K, V)] = ssc.withScope {
  43. reduceByKeyAndWindow(reduceFunc, windowDuration, slideDuration,
  44. defaultPartitioner(numPartitions))
  45. }
  46. /**
  47. /**通过对每个滑动过来的窗口应用一个reduceByKey的操作,返回一个DSream,有点像
  48. * `DStream.reduceByKey(),但是只是這个函数只是应用在滑动过来的窗口,hash分区是采用spark集群
  49. * 默认的分区树
  50. * @param reduceFunc 从左到右的reduce 函数
  51. * @param windowDuration 窗口时间
  52. * @param slideDuration 滑动时间
  53. * @param numPartitions 每个RDD的分区数.
  54. * @param partitioner 设置每个partition的分区数
  55. */
  56. def reduceByKeyAndWindow(
  57. reduceFunc: (V, V) => V,
  58. windowDuration: Duration,
  59. slideDuration: Duration,
  60. partitioner: Partitioner
  61. ): DStream[(K, V)] = ssc.withScope {
  62. self.reduceByKey(reduceFunc, partitioner)
  63. .window(windowDuration, slideDuration)
  64. .reduceByKey(reduceFunc, partitioner)
  65. }
  66. /**
  67. *通过对每个滑动过来的窗口应用一个reduceByKey的操作.同时对old RDDs进行了invReduceFunc操作
  68. * hash分区是采用spark集群,默认的分区树
  69. * @param reduceFunc从左到右的reduce 函数
  70. * @param invReduceFunc inverse reduce function; such that for all y, invertible x:
  71. * `invReduceFunc(reduceFunc(x, y), x) = y`
  72. * @param windowDuration窗口时间
  73. * @param slideDuration 滑动时间
  74. * @param filterFunc 来赛选一定条件的 key-value 对的
  75. */
  76. def reduceByKeyAndWindow(
  77. reduceFunc: (V, V) => V,
  78. invReduceFunc: (V, V) => V,
  79. windowDuration: Duration,
  80. slideDuration: Duration = self.slideDuration,
  81. numPartitions: Int = ssc.sc.defaultParallelism,
  82. filterFunc: ((K, V)) => Boolean =
  83. ): DStream[(K, V)] = ssc.withScope {
  84. reduceByKeyAndWindow(
  85. reduceFunc, invReduceFunc, windowDuration,
  86. slideDuration, defaultPartitioner(numPartitions), filterFunc
  87. )
  88. }
  89. /**
  90. *通过对每个滑动过来的窗口应用一个reduceByKey的操作.同时对old RDDs进行了invReduceFunc操作
  91. * hash分区是采用spark集群,默认的分区树
  92. * @param reduceFunc从左到右的reduce 函数
  93. * @param invReduceFunc inverse reduce function; such that for all y, invertible x:
  94. * `invReduceFunc(reduceFunc(x, y), x) = y`
  95. * @param windowDuration窗口时间
  96. * @param slideDuration 滑动时间
  97. * @param partitioner 每个RDD的分区数.
  98. * @param filterFunc 来赛选一定条件的 key-value 对的
  99. */
  100. def reduceByKeyAndWindow(
  101. reduceFunc: (V, V) => V,
  102. invReduceFunc: (V, V) => V,
  103. windowDuration: Duration,
  104. slideDuration: Duration,
  105. partitioner: Partitioner,
  106. filterFunc: ((K, V)) => Boolean
  107. ): DStream[(K, V)] = ssc.withScope {
  108. val cleanedReduceFunc = ssc.sc.clean(reduceFunc)
  109. val cleanedInvReduceFunc = ssc.sc.clean(invReduceFunc)
  110. val cleanedFilterFunc = if (filterFunc != ) Some(ssc.sc.clean(filterFunc)) else None
  111. new ReducedWindowedDStream[K, V](
  112. self, cleanedReduceFunc, cleanedInvReduceFunc, cleanedFilterFunc,
  113. windowDuration, slideDuration, partitioner
  114. )
  115. }



   
   
   
   
  1. import org.apache.log4j.{Level, Logger}
  2. import org.apache.spark.streaming.{Seconds, StreamingContext}
  3. import org.apache.spark.{SparkConf, SparkContext}
  4. object reduceByWindowOnStreaming {
  5. def main(args: Array[String]) {
  6. /**
  7. * this is test of Streaming operations-----reduceByKeyAndWindow
  8. */
  9. Logger.getLogger( "org.apache.spark").setLevel(Level.ERROR)
  10. Logger.getLogger( "org.eclipse.jetty.Server").setLevel(Level.OFF)
  11. val conf = new SparkConf().setAppName( "the reduceByWindow operation of SparK Streaming").setMaster( "local[2]")
  12. val sc = new SparkContext(conf)
  13. val ssc = new StreamingContext(sc,Seconds( 2))
  14. //set the Checkpoint directory
  15. ssc.checkpoint( "/Res")
  16. //get the socket Streaming data
  17. val socketStreaming = ssc.socketTextStream( "master", 9999)
  18. val data = socketStreaming.map(x =>(x, 1))
  19. //def reduceByKeyAndWindow(reduceFunc: (V, V) => V, windowDuration: Duration ): DStream[(K, V)]
  20. //val getedData1 = data.reduceByKeyAndWindow(_+_,Seconds(6))
  21. val getedData2 = data.reduceByKeyAndWindow(_+_,
  22. (a,b) => a+b* 0
  23. ,Seconds( 6),Seconds( 2))
  24. val getedData1 = data.reduceByKeyAndWindow(_+_,_-_,Seconds( 9),Seconds( 6))
  25. println( "reduceByKeyAndWindow : ")
  26. getedData1.print()
  27. ssc.start()
  28. ssc.awaitTermination()
  29. }
  30. }


這里出现了invReduceFunc函数這个函数有点特别,一不注意就会出错,现在通过分析源码中的

ReducedWindowedDStream這个类内部来进行说明:


——————reduceByWindow操作—————————


   
   
   
   
  1. /输入:reduceFunc、窗口长度、滑动长度
  2. //输出:(a,b)为从几个从左到右一次取得两个元素
  3. //(,a,b)进入reduceFunc,
  4. def reduceByWindow(
  5. reduceFunc: (T, T) => T,
  6. windowDuration: Duration,
  7. slideDuration: Duration
  8. ): DStream[T] = ssc.withScope {
  9. this.reduce(reduceFunc).window(windowDuration, slideDuration).reduce(reduceFunc)
  10. }
  11. /**
  12. *输入reduceFunc,invReduceFunc,窗口长度、滑动长度
  13. */
  14. def reduceByWindow(
  15. reduceFunc: (T, T) => T,
  16. invReduceFunc: (T, T) => T,
  17. windowDuration: Duration,
  18. slideDuration: Duration
  19. ): DStream[T] = ssc.withScope {
  20. this.map(( 1, _))
  21. .reduceByKeyAndWindow(reduceFunc, invReduceFunc, windowDuration, slideDuration, 1)
  22. .map(_._2)
  23. }



   
   
   
   
  1. import org.apache.log4j.{Level, Logger}
  2. import org.apache.spark.streaming.{Seconds, StreamingContext}
  3. import org.apache.spark.{SparkConf, SparkContext}
  4. /**
  5. * Created by root on 6/23/16.
  6. */
  7. object reduceByWindow {
  8. def main(args: Array[String]) {
  9. /**
  10. * this is test of Streaming operations-----reduceByWindow
  11. */
  12. Logger.getLogger( "org.apache.spark").setLevel(Level.ERROR)
  13. Logger.getLogger( "org.eclipse.jetty.Server").setLevel(Level.OFF)
  14. val conf = new SparkConf().setAppName( "the reduceByWindow operation of SparK Streaming").setMaster( "local[2]")
  15. val sc = new SparkContext(conf)
  16. val ssc = new StreamingContext(sc,Seconds( 2))
  17. //set the Checkpoint directory
  18. ssc.checkpoint( "/Res")
  19. //get the socket Streaming data
  20. val socketStreaming = ssc.socketTextStream( "master", 9999)
  21. //val data = socketStreaming.reduceByWindow(_+_,Seconds(6),Seconds(2))
  22. val data = socketStreaming.reduceByWindow(_+_,_+_,Seconds( 6),Seconds( 2))
  23. println( "reduceByWindow: count the number of elements")
  24. data.print()
  25. ssc.start()
  26. ssc.awaitTermination()
  27. }
  28. }




———————————————–countByWindow操作———————————


   
   
   
   
  1. /**
  2. * 输入 窗口长度和滑动长度,返回窗口内的元素数量
  3. * @param windowDuration 窗口长度
  4. * @param slideDuration 滑动长度
  5. */
  6. def countByWindow(
  7. windowDuration: Duration,
  8. slideDuration: Duration): DStream[Long] = ssc.withScope {
  9. this.map(_ => 1L).reduceByWindow(_ + _, _ - _, windowDuration, slideDuration)
  10. //窗口下的DStream进行map操作,把每个元素变为1之后进行reduceByWindow操作
  11. }



   
   
   
   
  1. import org.apache.log4j.{Level, Logger}
  2. import org.apache.spark.streaming.{Seconds, StreamingContext}
  3. import org.apache.spark.{SparkConf, SparkContext}
  4. /
  5. * Created by root on 6/23/16.
  6. /
  7. object countByWindow {
  8. def main(args: Array[String]) {
  9. /
  10. this is test of Streaming operations-----countByWindow
  11. */
  12. Logger.getLogger( "org.apache.spark").setLevel(Level.ERROR)
  13. Logger.getLogger( "org.eclipse.jetty.Server").setLevel(Level.OFF)
  14. val conf = new SparkConf().setAppName( "the reduceByWindow operation of SparK Streaming").setMaster( "local[2]")
  15. val sc = new SparkContext(conf)
  16. val ssc = new StreamingContext(sc,Seconds( 2))
  17. //set the Checkpoint directory
  18. ssc.checkpoint( "/Res")
  19. //get the socket Streaming data
  20. val socketStreaming = ssc.socketTextStream( "master", 9999)
  21. val data = socketStreaming.countByWindow(Seconds( 6),Seconds( 2))
  22. println( "countByWindow: count the number of elements")
  23. data.print()
  24. ssc.start()
  25. ssc.awaitTermination()
  26. }
  27. }

——————————– countByValueAndWindow————-


   
   
   
   
  1. /
  2. 输入 窗口长度、滑动时间、RDD分区数(默认分区是等于并行度)
  3. @param windowDuration width of the window; must be a multiple of this DStream's
  4. * batching interval
  5. * @param slideDuration sliding interval of the window (i.e., the interval after which
  6. * the new DStream will generate RDDs); must be a multiple of this
  7. * DStream's batching interval
  8. * @param numPartitions number of partitions of each RDD in the new DStream.
  9. /
  10. def countByValueAndWindow(
  11. windowDuration: Duration,
  12. slideDuration: Duration,
  13. numPartitions: Int = ssc.sc.defaultParallelism)
  14. (implicit ord: Ordering[T] = )
  15. : DStream[ (T, Long)] = ssc.withScope {
  16. this.map((_, 1L)).reduceByKeyAndWindow(
  17. (x: Long, y: Long) => x + y,
  18. (x: Long, y: Long) => x - y,
  19. windowDuration,
  20. slideDuration,
  21. numPartitions,
  22. (x: (T, Long)) => x._2 != 0L
  23. )
  24. }


   
   
   
   
  1. import org.apache.log4j.{Level, Logger}
  2. import org.apache.spark.streaming.{Seconds, StreamingContext}
  3. import org.apache.spark.{SparkConf, SparkContext}
  4. /
  5. Created by root on 6/23/16.
  6. /
  7. object countByValueAndWindow {
  8. def main(args: Array[String]) {
  9. /*
  10. * this is test of Streaming operations-----countByValueAndWindow
  11. */
  12. Logger.getLogger( "org.apache.spark").setLevel(Level.ERROR)
  13. Logger.getLogger( "org.eclipse.jetty.Server").setLevel(Level.OFF)
  14. val conf = new SparkConf().setAppName( "the reduceByWindow operation of SparK Streaming").setMaster( "local[2]")
  15. val sc = new SparkContext(conf)
  16. val ssc = new StreamingContext(sc,Seconds( 2))
  17. //set the Checkpoint directory
  18. ssc.checkpoint( "/Res")
  19. //get the socket Streaming data
  20. val socketStreaming = ssc.socketTextStream( "master", 9999)
  21. val data = socketStreaming.countByValueAndWindow(Seconds( 6),Seconds( 2))
  22. println( "countByWindow: count the number of elements")
  23. data.print()
  24. ssc.start()
  25. ssc.awaitTermination()
  26. }
  27. }





附录

链接:http://pan.baidu.com/s/1slkqwBb 密码:d92r
















你可能感兴趣的:(Scala)