司有一个表记录客户的信息,其中包括有客户的id,name和age(为了演示只列出这几个字段)。
创建Hive的表:
create table customer
(
id int,
age tinyint,
name string
)
partitioned by(dt string)
row format delimited
fields terminated by '|'
stored as textfile;
导入初始化数据:
load data local inpath '/home/hadoop/hivetestdata/customer.txt' into table customer partition(dt = '201506');
hive> select * from customer order by id;
customer.id customer.age customer.name customer.dt
1 25 jiangshouzhuang 201506
2 23 zhangyun 201506
3 24 yiyi 201506
4 32 mengmeng 201506
对于保险公司来说,客户每天都会发生变化,我们使用临时数据表customer_temp来记录每天客户信息,字段和属性与customer表一致,
create table customer_temp like customer;
load data local inpath '/home/Hadoop/hivetestdata/customer_temp.txt' into table customer_temp partition(dt = '201506');
包含的数据示例如下所示:
hive> select * from customer_temp;
customer_temp.id customer_temp.age customer_temp.name customer_temp.dt
1 26 jiangshouzhuang 201506
5 45 xiaosan 201506
如果需要实现客户表的增量更新,我们需要将两个表进行full outer join,将customer_temp表中发生修改的数据更新到customer表中。
hive (hive)> select * from customer_temp
> union all
> select a.* from customer a
> left outer join customer_temp b
> on a.id = b.id where b.id is null;
_u1.id _u1.age _u1.name _u1.dt
2 23 zhangyun 201506
3 24 yiyi 201506
4 32 mengmeng 201506
1 26 jiangshouzhuang 201506
5 45 xiaosan 201506
之前看到网上有使用类似如下的方法,感觉是存在问题的:
hive> select customer.id,
coalesce(customer_temp.age,customer.age),
customer.name,
coalesce(customer_temp.dt,customer.dt)
from customer_temp
full outer join customer on customer_temp.id = customer.id;
执行后的结果为:
customer.id _c1 customer.name _c3
1 26 jiangshouzhuang 201506
2 23 zhangyun 201506
3 24 yiyi 201506
4 32 mengmeng 201506
NULL 45 NULL 201506
可以看出的确是有问题的。
如果朋友们有更好的优化方法请赐教,谢谢。