hive 动态分区表

1. 环境版本
2. 使用方式
3. 问题解读
4. 参考文章

环境版本

hive:1.2

cdh:1.5.6

spark:1.6.1

动态分区表使用方式

1、设置属性

SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions=2048;
SET hive.exec.max.dynamic.partitions.pernode=256;

2、创建动态分区表

drop table dw_fact_fp_partition;
CREATE TABLE dw_fact_fp_partition 
(fp_id string ,fpdm string ,fphm string ) PARTITIONED BY (date_key bigint);

3、插入数据

INSERT OVERWRITE TABLE dw_fact_fp_partition PARTITION(date_key)
SELECT fp_id,fpdm,fphm,date_key FROM dw_fact_fp;  

报错问题解读

1、Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.
metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

Set the values for hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode to higher values.  For example:

SET hive.exec.dynamic.partition=true;
SET hive.exec.max.dynamic.partitions=2048;
SET hive.exec.max.dynamic.partitions.pernode=256;

参考文章

https://blog.csdn.net/oDaiLiDong/article/details/49884571

https://www.ibm.com/support/knowledgecenter/en/SSERCR_1.0.0/com.ibm.swg.im.infosphere.biginsights.bigsql.doc/doc/bsql_create_table_as.html

http://www.cnblogs.com/yongjian/p/6640951.html