Hive分析函数之LAG、LEAD、FIRST_VALUE、LAST_VALUE学习

数据准备：
创建表：

drop table if exists xxx_src_table;
create table xxx_src_table
(polno string,
eff_date string,
userno string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
stored as textfile;

数据准备：

[hadoop@emr-worker-10 hiveAnalyticFunction]$ cat xxx_src_table.txt 
P066666666666,2016-04-02 09:00:02,user01
P066666666666,2016-04-02 09:00:00,user02
P066666666666,2016-04-02 09:03:04,user11
P066666666666,2016-04-02 09:50:05,user03
P066666666666,2016-04-02 10:00:00,user51
P066666666666,2016-04-02 09:10:00,user09
P066666666666,2016-04-02 09:50:01,user32
P088888888888,2016-04-02 09:00:02,user41
P088888888888,2016-04-02 09:00:00,user55
P088888888888,2016-04-02 09:03:04,user23
P088888888888,2016-04-02 09:50:05,user80
P088888888888,2016-04-02 10:00:00,user08
P088888888888,2016-04-02 09:10:00,user22
P088888888888,2016-04-02 09:50:01,user31

数据装载：

LOAD DATA LOCAL INPATH  '/home/hadoop/nisj/hiveAnalyticFunction/xxx_src_table.txt'  OVERWRITE INTO TABLE xxx_src_table;

1. LAG
LAG(col,n,DEFAULT) 用于统计窗口内往上第n行值
第一个参数为列名，第二个参数为往上第n行（可选，默认为1），第三个参数为默认值（当往上第n行为NULL时候，取默认值，如不指定，则为NULL）

SELECT
polno,
eff_date,
userno,
ROW_NUMBER() OVER(PARTITION BY polno ORDER BY eff_date) AS rn,
LAG(eff_date,1,'1970-01-01 00:00:00') OVER(PARTITION BY polno ORDER BY eff_date) AS last_1_time,
LAG(eff_date,2) OVER(PARTITION BY polno ORDER BY eff_date) AS last_2_time
FROM xxx_src_table;

2. LEAD
与LAG相反
LEAD(col,n,DEFAULT) 用于统计窗口内往下第n行值
第一个参数为列名，第二个参数为往下第n行（可选，默认为1），第三个参数为默认值（当往下第n行为NULL时候，取默认值，如不指定，则为NULL）

SELECT
polno,
eff_date,
userno,
ROW_NUMBER() OVER(PARTITION BY polno ORDER BY eff_date) AS rn,
LEAD(eff_date,1,'1970-01-01 00:00:00') OVER(PARTITION BY polno ORDER BY eff_date) AS next_1_time,
LEAD(eff_date,2) OVER(PARTITION BY polno ORDER BY eff_date) AS next_2_time
FROM xxx_src_table;

3. FIRST_VALUE
取分组内排序后，截止到当前行，第一个值

SELECT
polno,
eff_date,
userno,
ROW_NUMBER() OVER(PARTITION BY polno ORDER BY eff_date) AS rn,
FIRST_VALUE(userno) OVER(PARTITION BY polno ORDER BY eff_date) AS first1
FROM xxx_src_table;

4. LAST_VALUE
取分组内排序后，截止到当前行，最后一个值

SELECT
polno,
eff_date,
userno,
ROW_NUMBER() OVER(PARTITION BY polno ORDER BY eff_date) AS rn,
LAST_VALUE(userno) OVER(PARTITION BY polno ORDER BY eff_date) AS last1
FROM xxx_src_table;

上面的结果看起来跟没做处理一样的结果；如果想要取分组内排序后最后一个值，则需要变通一下：

SELECT
polno,
eff_date,
userno,
ROW_NUMBER() OVER(PARTITION BY polno ORDER BY eff_date) AS rn,
LAST_VALUE(userno) OVER(PARTITION BY polno ORDER BY eff_date) AS last1,
FIRST_VALUE(userno) OVER(PARTITION BY polno ORDER BY eff_date DESC) AS last2
FROM xxx_src_table ORDER BY polno,eff_date;

如果不指定ORDER BY，则默认按照记录在文件中的偏移量进行排序，会出现错误的结果：

FIRST_VALUE没有排序：
SELECT
 polno,
 eff_date,
 userno,
 FIRST_VALUE(userno) OVER(PARTITION BY polno) AS first2  
FROM xxx_src_table;
LAST_VALUE没有排序：
SELECT
polno,
eff_date,
userno,
LAST_VALUE(userno) OVER(PARTITION BY polno) AS last2  
FROM xxx_src_table;

所以，一定要和分区排序结合起来。