Scala Spark WordCount

Scala所需依赖

    org.scala-lang
    scala-library
    2.11.8

Scala WordCount代码

val source: List[String] = Source.fromFile("./src/main/data/wordCount.txt").getLines().toList
source.flatMap(elem => elem.split(" "))
  .filter(_.nonEmpty)
  .groupBy(elem => elem.toLowerCase)
  .mapValues(elem => elem.size)
  .foreach(println)

Spark所需依赖

    org.apache.spark
    spark-core_2.11
    2.3.4

Spark WordCount代码

val sparkContext = new SparkContext((new SparkConf).setAppName("SparkWordCount").setMaster("local[2]"))
sparkContext.setLogLevel("WARN")
val source: RDD[String] = sparkContext.textFile("./src/main/data/wordCount.txt")
source.flatMap(_.split(" "))
  .filter(_.nonEmpty)
  .map(elem => (elem.toLowerCase, 1))
  .reduceByKey(_+_)
  .foreach(println)
sparkContext.stop