SparkStreaming 实现入门WordCount

@羲凡——只为了更好的活着

SparkStreaming 实现入门WordCount

这是SparkStreaming的基本入门，官网：http://spark.apache.org/docs/2.3.2/streaming-programming-guide.html#a-quick-example

代码如下，在执行前在 deptest22上输入命令 nc -lk 9999。

import org.apache.spark.SparkContext
import org.apache.spark.sql.SparkSession
import org.apache.spark.streaming.dstream.ReceiverInputDStream
import org.apache.spark.streaming.{Seconds, StreamingContext}

object TestStreamingWC {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("TestStreamingWC")
      .master("local[*]")
      .getOrCreate()
    val sc: SparkContext = spark.sparkContext
    val ssc: StreamingContext = new StreamingContext(sc, Seconds(3))

    val lines = ssc.socketTextStream("deptest22", 9999)
    val wordCounts = lines.flatMap(_.split(" "))
    .map((_, 1)).reduceByKey(_ + _)
    wordCounts.print()
    ssc.start()
    ssc.awaitTermination()
  }
}

假如在deptest22输入 hadoop hadoop hadoop hadoop，得到如下结果，说明成功了

-------------------------------------------
Time: 1551329700000 ms
-------------------------------------------
(hadoop,4)

====================================================================

@羲凡——只为了更好的活着

若对博客中有任何问题，欢迎留言交流