FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
执行Hive sql时报错：
执行语句和报错详情如下：
hive (thnews_new)>
> insert into table thnews_new.thnews_all
> select date_id.id,date_id.source,date_id.sendTime,date_id.descripe,date_id.subject,date_id.type,context_by_id.context
> from thnews_new.context_by_id as context_by_id join thnews_new.date_id as date_id
> on date_id.id = context_by_id.id;
Automatically selecting local only mode for query
Query ID = root_20190710125148_fd34f996-1eb8-40c3-a3a5-7e04c6edad76
Total jobs = 1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/soft/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/soft/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
19/07/10 12:51:57 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
19/07/10 12:51:57 WARN conf.Configuration: file:/tmp/root/e96efa4b-a506-4956-8bda-ca74b326b128/hive_2019-07-10_12-51-48_070_3371426343275360779-1/-local-10005/jobconf.xml:an attempt to override final parameter: mapreduce.framework.name; Ignoring.
Execution log at: /tmp/root/root_20190710125148_fd34f996-1eb8-40c3-a3a5-7e04c6edad76.log
2019-07-10 12:51:59 Starting to launch local task to process map join; maximum memory = 518979584
2019-07-10 12:52:03 Dump the side-table for tag: 0 with group count: 39 into file: file:/tmp/root/e96efa4b-a506-4956-8bda-ca74b326b128/hive_2019-07-10_12-51-48_070_3371426343275360779-1/-local-10002/HashTable-Stage-4/MapJoin-mapfile60–.hashtable
2019-07-10 12:52:03 Uploaded 1 File to: file:/tmp/root/e96efa4b-a506-4956-8bda-ca74b326b128/hive_2019-07-10_12-51-48_070_3371426343275360779-1/-local-10002/HashTable-Stage-4/MapJoin-mapfile60–.hashtable (105284 bytes)
2019-07-10 12:52:03 End of local task; Time Taken: 4.079 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there’s no reduce operator
19/07/10 12:52:04 [main]: ERROR mr.ExecDriver: local
Job running in-process (local Hadoop)
19/07/10 12:52:04 [LocalJobRunner Map Task Executor #0]: ERROR exec.MapJoinOperator: Unexpected exception: null
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.getRefKey(MapJoinOperator.java:327)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:339)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator M a p O p C t x . f o r w a r d ( M a p O p e r a t o r . j a v a : 162 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . M a p O p e r a t o r . p r o c e s s ( M a p O p e r a t o r . j a v a : 508 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . m r . E x e c M a p p e r . m a p ( E x e c M a p p e r . j a v a : 163 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p R u n n e r . r u n ( M a p R u n n e r . j a v a : 54 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k . r u n O l d M a p p e r ( M a p T a s k . j a v a : 453 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . M a p T a s k . r u n ( M a p T a s k . j a v a : 343 ) a t o r g . a p a c h e . h a d o o p . m a p r e d . L o c a l J o b R u n n e r MapOpCtx.forward(MapOperator.java:162) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.LocalJobRunner MapOpCtx.forward(MapOperator.java:162)atorg.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)atorg.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)atorg.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)atorg.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)atorg.apache.hadoop.mapred.MapTask.run(MapTask.java:343)atorg.apache.hadoop.mapred.LocalJobRunnerJob M a p T a s k R u n n a b l e . r u n ( L o c a l J o b R u n n e r . j a v a : 243 ) a t j a v a . u t i l . c o n c u r r e n t . E x e c u t o r s MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors MapTaskRunnable.run(LocalJobRunner.java:243)atjava.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutorKaTeX parse error: Expected 'EOF', got '#' at position 135: … Task Executor #̲0]: FATAL mr.Ex…Job M a p T a s k R u n n a b l e . r u n ( L o c a l J o b R u n n e r . j a v a : 243 ) a t j a v a . u t i l . c o n c u r r e n t . E x e c u t o r s MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors MapTaskRunnable.run(LocalJobRunner.java:243)atjava.util.concurrent.ExecutorsRunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor W o r k e r . r u n ( T h r e a d P o o l E x e c u t o r . j a v a : 617 ) a t j a v a . l a n g . T h r e a d . r u n ( T h r e a d . j a v a : 745 ) C a u s e d b y : o r g . a p a c h e . h a d o o p . h i v e . q l . m e t a d a t a . H i v e E x c e p t i o n : U n e x p e c t e d e x c e p t i o n : n u l l a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . M a p J o i n O p e r a t o r . p r o c e s s ( M a p J o i n O p e r a t o r . j a v a : 426 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . O p e r a t o r . f o r w a r d ( O p e r a t o r . j a v a : 837 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . F i l t e r O p e r a t o r . p r o c e s s ( F i l t e r O p e r a t o r . j a v a : 122 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . O p e r a t o r . f o r w a r d ( O p e r a t o r . j a v a : 837 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . T a b l e S c a n O p e r a t o r . p r o c e s s ( T a b l e S c a n O p e r a t o r . j a v a : 97 ) a t o r g . a p a c h e . h a d o o p . h i v e . q l . e x e c . M a p O p e r a t o r Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: null at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:426) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) at org.apache.hadoop.hive.ql.exec.MapOperator Worker.run(ThreadPoolExecutor.java:617)atjava.lang.Thread.run(Thread.java:745)Causedby:org.apache.hadoop.hive.ql.metadata.HiveException:Unexpectedexception:nullatorg.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:426)atorg.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)atorg.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:122)atorg.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)atorg.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)atorg.apache.hadoop.hive.ql.exec.MapOperatorMapOpCtx.forward(MapOperator.java:162)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
… 10 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.getRefKey(MapJoinOperator.java:327)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:339)
… 16 more
2019-07-10 12:52:05,879 Stage-4 map = 0%, reduce = 0%
Ended Job = job_local1546372873_0008 with errors
19/07/10 12:52:05 [main]: ERROR exec.Task: Ended Job = job_local1546372873_0008 with errors
Error during job, obtaining debugging information…
19/07/10 12:52:05 [Thread-296]: ERROR exec.Task: Error during job, obtaining debugging information…
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
19/07/10 12:52:05 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-4: HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
★解决方法：
set hive.auto.convert.join=false;
set hive.vectorized.execution.enabled=false;
set hive.vectorized.execution.reduce.enabled=false;
●set hive.vectorized.execution.enabled=false;
默认关闭，开启后，每次处理数据时会将1024行数据组成一个batch进行处理，而不是一行一行进行处理，这样能够显著提高执行速度。
★利用矢量化（ Vectorization）
矢量化（Vectorization） 通过在一次操作中提取t 1024 行数据提升性能，而不是一次只取一条。它提升了像过滤, 联合, 聚合等等操作的性能。
Vectorization 可以通过在环境中执行如下命令而得到启用。
set hive.vectorized.execution.enabled=true;
set hive.vectorized.execution.reduce.enabled=true;