[2] Hive3.x 查询流程源码-Cli端-01

Hive架构简图

Hive架构简图 hive原理与源码分析-hive源码架构与理论

Hive3.x安装准备工作

详细参见：Hive3.x 安装与debug

1 Hive命令行提交查询

SELECT deptno, count(deptname) as deptno_cnt from hive3_test.depts group by deptno;

执行栈

下面简要分析关键路径代码

2 CliDriver接收查询

public int processCmd(String cmd) {
  ...
  processLocalCmd(String cmd, CommandProcessor proc, CliSessionState ss) {
      ret = (ReExecDriver)qp.run(cmd).getResponseCode();
      这里run调用Driver::run
      //public CommandProcessorResponse run(String command) //见下文
      ret = processLocalCmd(cmd, proc, ss);//待单独分析
    }
  ...
  
}

3 ReExecDriver compileAndRespond查询

public CommandProcessorResponse run(String command) {
    CommandProcessorResponse r0 = compileAndRespond(command);
    private void runInternal(String command, boolean alreadyCompiled){
    }
}

compileAndRespond中调用Driver compile查询

public CommandProcessorResponse compileAndRespond(String command, boolean cleanupTxnList){
   private void compileInternal(String command, boolean deferClose){
	     Driver::compile(command, true, deferClose);
    }
}

4 Driver compile请求

private void compile(String command, boolean resetTaskIds, boolean deferClose) throws CommandProcessorResponse {
   ......
   //ParseUtils.parse 见command  sql解析为AST
   try {
        tree = ParseUtils.parse(command, ctx);
      } catch (ParseException e) {
        parseError = true;
        throw e;
      } finally {
        hookRunner.runAfterParseHook(command, parseError);
      }
  ......

  //语法分析 Do semantic analysis and plan generation
  BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(queryState, tree);
  sem.analyze(tree, ctx);

  //检测语法合法性 validate the plan
  sem.validate();
  .....
  // 逻辑执行计划
  plan = new QueryPlan(queryStr, sem, perfLogger.getStartTime(PerfLogger.DRIVER_RUN), queryId,
        queryState.getHiveOperation(), schema);
}

4.1 ParseUitls SQL AST解析

ParseUitls
/** Parses the Hive query. */
public static ASTNode parse(
    String command, Context ctx, String viewFullyQualifiedName) throws ParseException {
  ParseDriver pd = new ParseDriver();
  ASTNode tree = pd.parse(command, ctx, viewFullyQualifiedName);
  tree = findRootNonNullToken(tree);
  handleSetColRefs(tree);
  return tree;
}

ParseUitls 调起HiveParser

ParseDriver
public ASTNode parse(String command, Context ctx, String viewFullyQualifiedName){

HiveParser parser = new HiveParser(tokens);
...
ASTNode tree = (ASTNode) r.getTree();
}

ASTNode的结构如下

4.2 BaseSemanticAnalyzer语法分析

接上文sem.analyze(tree, ctx);语法分析

会先进入calciteplanner进行查询优化
CalcitePlanner待续
public void analyzeInternal(ASTNode ast) throws SemanticException {
    if (runCBO) {
      super.analyzeInternal(ast, new PlannerContextFactory() {
        @Override
        public PlannerContext create() {
          return new PreCboCtx();
        }
      });
    } else {
      super.analyzeInternal(ast);
    }
}