hive 0.11使用bug记

1、多表join group by count(distinct)出现NPE异常

 2013-09-17 11:57:56,060 ERROR ql.Driver (SessionState.java:printError(386)) - FAILED: NullPointerException null
java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.optimizer.physical.MetadataOnlyOptimizer$MetadataOnlyTaskDispatcher.dispatch(MetadataOnlyOptimizer.java:308)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:87)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:124)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:101)
    at org.apache.hadoop.hive.ql.optimizer.physical.MetadataOnlyOptimizer.resolve(MetadataOnlyOptimizer.java:175)
    at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:79)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:8399)

参见：HIVE-4963,Potential NPE in MetadataOnlyOptimizer, https://issues.apache.org/jira/browse/HIVE-4935

2、别名不能以数字开头，否则报bug

3、子查询join失败

URL:
  http://YZSJHL19-43.opi.com:50030/taskdetails.jsp?jobid=job_201309051836_0310&tipid=task_201309051836_0310_m_000001
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1393)
    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1397)
    at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1397)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
    ... 8 more
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)

参见：NPE - subquery smb join fails, https://issues.apache.org/jira/browse/HIVE-4968

4、hive参数set时候不能以数学函数（朋友告知）

[dp@YZSJHL19-87 ~]$ hive -S -e "set hive.exec.reducers.max=100*9;"
Query returned non-zero code: 1, cause: 'SET hive.exec.reducers.max=100*9' FAILED because hive.exec.reducers.max expects an INT value.
[dp@YZSJHL19-87 ~]$ hive -S -e "set hive.exec.reducers.max=900;"
[dp@YZSJHL19-87 ~]$ 

5、insert overwrite partitioned table时partition字段描述的名字和顺序必须和table schema中定义的保持一致（朋友告知）

6、group by语句产生的数据有错误，demo sql

explain from (select key, value from src group by key, value) s select s.key group by s.key;

第二次 相同的group by s.key会产生多行记录， ReduceSinkOperator的key是x,y导致相同的数据分配到不同的reduce上，partition设置错误，可以参考https://issues.apache.org/jira/browse/HIVE-5237，https://issues.apache.org/jira/browse/HIVE-5149