Hive – Group By 的实现 explain分析

准备数据

SELECT uid, SUM(COUNT) FROM logs GROUP BY uid;

hive> SELECT * FROM logs;
a	苹果	5
a	橙子	3
a      苹果   2
b	烧鸡	1
 
hive> SELECT uid, SUM(COUNT) FROM logs GROUP BY uid;
a	10
b	1

计算过程

默认设置了hive.map.aggr=true，所以会在mapper端先group by一次，最后再把结果merge起来，为了减少reducer处理的数据量。注意看explain的mode是不一样的。mapper是hash，reducer是mergepartial。如果把hive.map.aggr=false，那将groupby放到reducer才做，他的mode是complete.

Operator

Explain

hive> explain SELECT uid, sum(count) FROM logs group by uid;
OK
ABSTRACT SYNTAX TREE:
  (TOK_QUERY (TOK_FROM (TOK_TABREF (TOK_TABNAME logs))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL uid)) (TOK_SELEXPR (TOK_FUNCTION sum (TOK_TABLE_OR_COL count)))) (TOK_GROUPBY (TOK_TABLE_OR_COL uid))))
 
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage
 
STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        logs 
          TableScan // 扫描表
            alias: logs
            Select Operator //选择字段
              expressions:
                    expr: uid
                    type: string
                    expr: count
                    type: int
              outputColumnNames: uid, count
              Group By Operator //这里是因为默认设置了hive.map.aggr=true，会在mapper先做一次聚合，减少reduce需要处理的数据
                aggregations:
                      expr: sum(count) //聚集函数
                bucketGroup: false
                keys: //键
                      expr: uid
                      type: string
                mode: hash //hash方式，processHashAggr()
                outputColumnNames: _col0, _col1
                Reduce Output Operator //输出key，value给reducer
                  key expressions:
                        expr: _col0
                        type: string
                  sort order: +
                  Map-reduce partition columns:
                        expr: _col0
                        type: string
                  tag: -1
                  value expressions:
                        expr: _col1
                        type: bigint
      Reduce Operator Tree:
        Group By Operator
 
          aggregations:
                expr: sum(VALUE._col0)
//聚合
          bucketGroup: false
          keys:
                expr: KEY._col0
                type: string
          mode: mergepartial //合并值
          outputColumnNames: _col0, _col1
          Select Operator //选择字段
            expressions:
                  expr: _col0
                  type: string
                  expr: _col1
                  type: bigint
            outputColumnNames: _col0, _col1
            File Output Operator //输出到文件
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
 
  Stage: Stage-0
    Fetch Operator
      limit: -1

hive> select distinct value from src; 
hive> select max(key) from src; 

因为没有grouping keys，所以只有一个reducer。



  

      2.2 如果有聚合函数或者groupby，做如下处理： 

            插入一个select operator，选取所有的字段，用于优化阶段ColumnPruner的优化 

            2.2.1  hive.map.aggr为true，默认是true，开启的，在map端做部分聚合 

                  2.2.1.1 hive.groupby.skewindata为false，默认是关闭的，groupby的数据没有倾斜。      

                  生成的operator是: GroupByOperator+ReduceSinkOperator+GroupByOperator。 

      GroupByOperator+ReduceSinkOperator用于在map端做操作，第一个GroupByOperator在map端先做部分聚合。第二个用于在reduce端做GroupBy操作 

                  2.2.1.2 hive.groupby.skewindata为true 

                  生成的operator是: GroupbyOperator+ReduceSinkOperator+GroupbyOperator+ReduceSinkOperator +GroupByOperator 

               GroupbyOperator+ReduceSinkOperator（第一个MapredTask的map阶段） 

               GroupbyOperator(第一个MapredTask的reduce阶段) 

               ReduceSinkOperator （第二个MapredTask的map阶段） 

               GroupByOperator（第二个MapredTask的reduce阶段） 

            2.2.2  hive.map.aggr为false 

                   2.2.2.1 hive.groupby.skewindata为true 

                    生成的operator是: ReduceSinkOperator+GroupbyOperator+ReduceSinkOperator +GroupByOperator                    

               ReduceSinkOperator(第一个MapredTask的map阶段) 

               GroupbyOperator(第一个MapredTask的reduce阶段) 

               ReduceSinkOperator （第二个MapredTask的map阶段） 

               GroupByOperator（第二个MapredTask的reduce阶段） 

                   2.2.2.2 hive.groupby.skewindata为false 

                    生成的operator是: ReduceSinkOperator(map阶段运行)+GroupbyOperator（reduce阶段运行） 







第一种情况：

set hive.map.aggr=false;

set hive.groupby.skewindata=false;

SemanticAnalyzer.genGroupByPlan1MR(){

  （1）ReduceSinkOperator： It will put all Group By keys and the 
distinct field (if any) in the map-reduce sort key, and all other fields
 in the map-reduce value.

  （2）GroupbyOperator：GroupByDesc.Mode.COMPLETE，Reducer: iterate/merge (mode = COMPLETE)

}



第二种情况：

set hive.map.aggr=true; 

set hive.groupby.skewindata=false; 

SemanticAnalyzer.genGroupByPlanMapAggr1MR(){

   （1）GroupByOperator：GroupByDesc.Mode.HASH，The agggregation 
evaluation functions are as follows: Mapper: iterate/terminatePartial 
(mode = HASH)

   （2）ReduceSinkOperator：Partitioning Key: grouping key。Sorting Key:
 grouping key if no DISTINCT grouping + distinct key if DISTINCT

   （3）GroupByOperator：GroupByDesc.Mode.MERGEPARTIAL，Reducer: 
iterate/terminate if DISTINCT merge/terminate if NO DISTINCT (mode = 
MERGEPARTIAL)

}



第三种情况：

set hive.map.aggr=false; 

set hive.groupby.skewindata=true; 

SemanticAnalyzer.genGroupByPlan2MR(){

    （1）ReduceSinkOperator：Partitioning Key: random() if no DISTINCT 
grouping + distinct key if DISTINCT。Sorting Key: grouping key if no 
DISTINCT grouping + distinct key if DISTINCT

    （2）GroupbyOperator：GroupByDesc.Mode.PARTIAL1，Reducer: iterate/terminatePartial (mode = PARTIAL1)

    （3）ReduceSinkOperator：Partitioning Key: grouping key。Sorting 
Key: grouping key if no DISTINCT grouping + distinct key if DISTINCT

    （4）GroupByOperator：GroupByDesc.Mode.FINAL，Reducer: merge/terminate (mode = FINAL)

}



第四种情况：

set hive.map.aggr=true; 

set hive.groupby.skewindata=true; 

SemanticAnalyzer.genGroupByPlanMapAggr2MR(){

     （1）GroupbyOperator：GroupByDesc.Mode.HASH，Mapper: iterate/terminatePartial (mode = HASH)

     （2）ReduceSinkOperator： Partitioning Key: random() if no 
DISTINCT grouping + distinct key if DISTINCT。 Sorting Key: grouping key 
if no DISTINCT grouping + distinct key if DISTINCT。

     （3）GroupbyOperator：GroupByDesc.Mode.PARTIALS， Reducer: 
iterate/terminatePartial if DISTINCT merge/terminatePartial if NO 
DISTINCT (mode = MERGEPARTIAL)

     （4）ReduceSinkOperator：Partitioining Key: grouping key。Sorting 
Key: grouping key if no DISTINCT grouping + distinct key if DISTINCT

     （5）GroupByOperator：GroupByDesc.Mode.FINAL，Reducer: merge/terminate (mode = FINAL)

}





ReduceSinkOperator的processOp(Object row, int 
tag)会根据相应的条件设置Key的hash值，如第四种情况的第一个ReduceSinkOperator：Partitioning Key: 
random() if no DISTINCT grouping + distinct key if 
DISTINCT，如果没有DISTINCT字段，那么在OutputCollector.collect前会设置当前Key的hash值为一个随机
数，random = new Random(12345);。如果有DISTINCT字段，那么key的hash值跟grouping + 
distinct key有关。







GroupByOperator：

initializeOp(Configuration hconf)

processOp(Object row, int tag) 

closeOp(boolean abort)

forward(ArrayList