Hive动态分区（Dynamic Partition）导致的内存问题
## Hive动态分区（Dynamic Partition）导致的内存问题
默认hive.optimize.sort.dynamic.partition是false，hive进行动态分区的时候，可以看见一直在 FileSinkOperator: New Final Path导致内存中保存太多句柄和数据，导致OOM或者内存超被Yarn Kill。开启下面的参数可以缓解内存压力。
hive.optimize.sort.dynamic.partition
- Default Value: true in Hive 0.13.0 and 0.13.1; false in Hive 0.14.0 and later (HIVE-8151)
- Added In: Hive 0.13.0 with HIVE-6455
- When enabled, dynamic partitioning column will be globally sorted. This way we can keep only one record writer open for each partition value in the reducer thereby reducing the memory pressure on reducers.
Container killed on request. Exit code is 137 Container exited with a non-zero exit code 137 Killed by external signal
2019-04-17 16:16:58,358 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20181009/own_library_flag=0/000000_0
2019-04-17 16:16:58,374 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20180909/own_library_flag=0/000000_0
2019-04-17 16:16:58,374 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_task_tmp.-ext-10002/dayno=20180909/own_library_flag=0/_tmp.000000_0
2019-04-17 16:16:58,374 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20180909/own_library_flag=0/000000_0
2019-04-17 16:16:58,388 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20181001/own_library_flag=0/000000_0
2019-04-17 16:16:58,388 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_task_tmp.-ext-10002/dayno=20181001/own_library_flag=0/_tmp.000000_0
2019-04-17 16:16:58,388 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20181001/own_library_flag=0/000000_0
2019-04-17 16:16:58,396 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20180906/own_library_flag=0/000000_0
2019-04-17 16:16:58,396 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_task_tmp.-ext-10002/dayno=20180906/own_library_flag=0/_tmp.000000_0
2019-04-17 16:16:58,396 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20180906/own_library_flag=0/000000_0
2019-04-17 16:16:58,405 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20181127/own_library_flag=0/000000_0
2019-04-17 16:16:58,405 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_task_tmp.-ext-10002/dayno=20181127/own_library_flag=0/_tmp.000000_0
2019-04-17 16:16:58,405 INFO [main] org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS hdfs://alg-hdfs/warehouse/browser/browser.db/t_xqh_temp_20190417_document/.hive-staging_hive_2019-04-17_16-16-01_858_4125803238649808238-15659/_tmp.-ext-10002/dayno=20181127/own_library_flag=0/000000_0
2019-04-17 16:16:58,436 FATAL [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“doc_id”:“V_01KGpTaP”,“id”:-1,“outid”:“V_01KGpTaP”,“source”:“yidian”,“gid”:null,“title”:“福克斯RS到底跑多快”,“style”:null,“cover1”:"",“cover2”:"",“cover3”:"",“doctype”:-1,“article_type”:“视频”,“summary”:"",“author”:“绝对的汽车控”,“publishtime”:-1,“expiretime”:-1,“content”:null,“url”:null,“imgcount”:-1,“videoduration”:-1,“originlevel”:100,“computerlevel”:-1,“artificiallevel”:-1,“topcategory”:“汽车”,“secondcategory”:null,“origintags”:"",“computertags”:"",“artificialtags”:"",“secondresource”:null,“origindata”:"",“status”:-1,“mark”:"",“clickcount”:-1,“upscount”:-1,“commentcount”:-1,“sharecount”:-1,“threadcount”:-1,“clickusercount”:-1,“createtime”:1550938435,“createtime_st”:“20190224 00:13:55”,“updatetime”:-1,“updatetime_st”:"-1",“openid”:-1,“sourceen”:"",“base62id”:"",“docattr”:"",“docflag”:"",“similarid”:"",“author_name”:“绝对的汽车控”,“author_level”:-1,“author_type”:-1,“own_library_flag”:0,“import_time”:null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild 2. r u n ( Y a r n C h i l d . j a v a : 164 ) a t j a v a . s e c u r i t y . A c c e s s C o n t r o l l e r . d o P r i v i l e g e d ( N a t i v e M e t h o d ) a t j a v a x . s e c u r i t y . a u t h . S u b j e c t . d o A s ( S u b j e c t . j a v a : 422 ) a t o r g . a p a c h e . h a d o o p . s e c u r i t y . U s e r G r o u p I n f o r m a t i o n . d o A s ( U s e r G r o u p I n f o r m a t i o n . j a v a : 1920 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . Y a r n C h i l d . m a i n ( Y a r n C h i l d . j a v a : 158 ) C a u s e d b y : o r g . a p a c h e . h a d o o p . h i v e . q l . m e t a d a t a . H i v e F a t a l E x c e p t i o n : [ E r r o r 20004 ] : F a t a l e r r o r o c c u r r e d w h e n n o d e t r i e d t o c r e a t e t o o m a n y d y n a m i c p a r t i t i o n s . T h e m a x i m u m n u m b e r o f d y n a m i c p a r t i t i o n s i s c o n t r o l l e d b y h i v e . e x e c . m a x . d y n a m i c . p a r t i t i o n s a n d h i v e . e x e c . m a x . d y n a m i c . p a r t i t i o n s . p e r n o d e . M a x i m u m w a s s e t t o : 100 a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . F i l e S i n k O p e r a t o r . g e t D y n O u t P a t h s ( F i l e S i n k O p e r a t o r . j a v a : 896 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . F i l e S i n k O p e r a t o r . p r o c e s s O p ( F i l e S i n k O p e r a t o r . j a v a : 676 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . O p e r a t o r . f o r w a r d ( O p e r a t o r . j a v a : 815 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . S e l e c t O p e r a t o r . p r o c e s s O p ( S e l e c t O p e r a t o r . j a v a : 84 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . O p e r a t o r . f o r w a r d ( O p e r a t o r . j a v a : 815 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . T a b l e S c a n O p e r a t o r . p r o c e s s O p ( T a b l e S c a n O p e r a t o r . j a v a : 97 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . M a p O p e r a t o r 2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100 at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:896) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:676) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:97) at org.apache.hadoop.hive.ql.exec.MapOperator 2.run(YarnChild.java:164)atjava.security.AccessController.doPrivileged(NativeMethod)atjavax.security.auth.Subject.doAs(Subject.java:422)atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)atorg.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)Causedby:org.apache.hadoop.hive.ql.metadata.HiveFatalException:[Error20004]:Fatalerroroccurredwhennodetriedtocreatetoomanydynamicpartitions.Themaximumnumberofdynamicpartitionsiscontrolledbyhive.exec.max.dynamic.partitionsandhive.exec.max.dynamic.partitions.pernode.Maximumwassetto:100atorg.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:896)atorg.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:676)atorg.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)atorg.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)atorg.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)atorg.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:97)atorg.apache.hadoop.hive.ql.exec.MapOperatorMapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497)
… 9 more