Spark之WordCount,提交到集群中运行

在idea中写好程序，提交到集群中运行

object WordCount {
  def main(args: Array[String]): Unit = {

    if (args.length != 2) {
      println(
        """
          |cn.uhome.spark01.WordCount  <input> <output>
        """.stripMargin)
    }
    sys.exit(1)

    //接收参数
    val Array(input, output) = args

    val conf = new SparkConf()
    //初始化SparkContext实例
    val sc: SparkContext = new SparkContext(conf)

    //sc.textFile(input).flatMap(_.split(" ")).map((_, 1)).reduceByKey(_ + _).sortBy(_._2, false).saveAsTextFile(output)

    //读取文件
    val rdd1: RDD[String] = sc.textFile(input)

    //将文件按行切分并压平
    val wordsRdd: RDD[String] = rdd1.flatMap(_.split(" "))

    //将单词和1组装
    val wordAndOne: RDD[(String, Int)] = wordsRdd.map((_, 1))

    //将组装的元组按key分组聚合
    val result: RDD[(String, Int)] = wordAndOne.reduceByKey((_ + _))

    //按key值排序:两种方式
    result.sortBy(_._2, false)
    //    result.sortBy(-_._2)

    //存储
    result.saveAsTextFile(output)

    //释放资源
    sc.stop()


  }

}

提交程序到集群中运行:

spark-submit --master spark://hdp-01:7077 --name "scalawordcount" --class cn.uhome.spark01.WordCount.WordCount /root/myspark-1.0-SNAPSHOT.jar hdfs://hdp-01:9000/wordcount/input hdfs://hdp-01:9000/wordcount/output2