spark之RDD

• 启动spark-shell
  bin/spark-shell --master spark://bigdata.eclipse.com:7077
• reduceBykey、groupBykey、sortByKey、join的使用

1、reduceBykey
var rdd = sc.textFile("/data/wc.input") 
val reduceBykey = rdd.flatMap(line => line.split(" ")).map(x => (x, 1)).reduceByKey((a,b) => (a + b)).collect 2、groupBykey var rdd = sc.textFile("/data/wc.input") val groupBykey= rdd.flatMap(line => line.split(" ")).map(x => (x, 1)).groupByKey().map(x => (x._1,x._2.sum)).collect 3、sortByKey var rdd = sc.textFile("/data/wc.input") val groupBykey= rdd.flatMap(line => line.split(" ")).map(x => (x, 1)).groupByKey().map(x => (x._1,x._2.sum)) 1) key升序 roupBykey.sortByKey().collect 2）key降序 groupBykey.sortByKey(false).collect 3）value降序 groupBykey.map(x => (x._2,x._1)).sortByKey(false).map(x => (x._2,x._1)).collect 4、join val rdd1 = sc.parallelize(List("1,spark","2,sqoop")).map(_.split(",")).map(x => (x(0),x(1))) val rdd2 = sc.parallelize(List("1,hadoop","2,yarn")).map(_.split(",")).map(x => (x(0),x(1))) rdd1.join(rdd2).collect

• 分组排序

1）方式一
rdd.map(line => line.split(" ")).map(x => (x(0),x(1).toInt)).groupByKey().map(x => (x._1,x._2.toList.sorted.reverse)).collect

2）方式二
使用:paste模式
rdd.map(_.split(" ")).map(x => (x(0),x(1).toInt)).groupByKey()
.map(x =>{
    var xx = x._1
    var xy = x._2.toList.sorted.reverse 
   （xx,yy）
}).collect