spark系列二：开窗函数实例

Spark 1.4.x版本以后，为Spark SQL和DataFrame引入了开窗函数，比如最经典，最常用的，row_number()，可以让我们实现分组取topn的逻辑。

package cn.spark.study.core;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.hive.HiveContext;

public class windowfunction {
 public static void main(String[] args) {
  SparkConf conf = new SparkConf()
    .setAppName("hivesource");
    
  JavaSparkContext sc = new JavaSparkContext(conf);
  HiveContext hc = new HiveContext(sc.sc());
  hc.sql("drop table if exists sales");
  hc.sql("create table sales(product STRING,category STRING,revenue BIGINT)");
  hc.sql("load data local inpath '/usr/local/src/spark-java/sales.txt' into table sales");
  String sql = "select product,category,revenue from ( select product,category,revenue,row_number() over(partition by category order by revenue DESC ) rank from sales) t1 where t1.rank <= 3 ";
     DataFrame df = hc.sql(sql);
     hc.sql("drop table if exists topsales");
     df.saveAsTable("topsales");
 }

}

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/30541278/viewspace-2154849/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/30541278/viewspace-2154849/