Spark源码解析之org.apache.spark.deploy.SparkSubmit源码解析

前面解读launch.main的时候已经了解了spark-submit的提交流程，这里大概看下流程。

当打jar提交到集群运行的时候，一般会设置一些参数，例如本地提交examples的SparkPi：

spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://192.168.2.1:7077 \
D:\spark\spark-2.4.3\examples\target\original-spark-examples_2.11-2.4.3.jar

所以首先会调用spark-submit脚本，主要是调用spark-class脚本，把SparkSubmit 类和输入的参数都作为spark-class的参数。

exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"

spark-class中会执行下面的shell，进入launcher.main，参数仍然是SparkSubmit类和参数

# java -Xmx128m -cp ...jars org.apache.spark.launcher.Main "$@"
"$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"

 launcher.main中会解析过滤参数，构建执行命令，返回给spark-class脚本，最后通过 exec "${CMD[@]}" 真正调用SparkSubmit类。这里详细解读下SparknSubmit类。

先看看最前的说明。

/**
 * Whether to submit, kill, or request the status of an application.
 * The latter two operations are currently supported only for standalone and Mesos cluster modes.
 * 这个类主要是提交app,终止和请求状态,但目前终止和请求只能在standalone和mesos模式下
 */

// 继承了枚举类,定义了4个属性,多了一个打印spark版本
private[deploy] object SparkSubmitAction extends Enumeration {
  type SparkSubmitAction = Value
  val SUBMIT, KILL, REQUEST_STATUS, PRINT_VERSION = Value
}

 惯例首先进入object SparkSubmit

object SparkSubmit extends CommandLineUtils with Logging {

  // Cluster managers
  // spark支持的调度方式,yarn,standalne,mesos,local,kubernetes
  private val YARN = 1
  private val STANDALONE = 2
  private val MESOS = 4
  private val LOCAL = 8
  private val KUBERNETES = 16
  private val ALL_CLUSTER_MGRS = YARN | STANDALONE | MESOS | LOCAL | KUBERNETES

  // Deploy modes
  // yarn的两种部署模式
  private val CLIENT = 1
  private val CLUSTER = 2
  private val ALL_DEPLOY_MODES = CLIENT | CLUSTER

  // Special primary resource names that represent shells rather than application jars.
  // shell命令相关的常量
  private val SPARK_SHELL = "spark-shell"
  private val PYSPARK_SHELL = "pyspark-shell"
  private val SPARKR_SHELL = "sparkr-shell"
  private val SPARKR_PACKAGE_ARCHIVE = "sparkr.zip"
  private val R_PACKAGE_ARCHIVE = "rpkg.zip"

  // 找不到类的错误码
  private val CLASS_NOT_FOUND_EXIT_STATUS = 101

  // Following constants are visible for testing.
  // 测试用的常量
  private[deploy] val YARN_CLUSTER_SUBMIT_CLASS =
    "org.apache.spark.deploy.yarn.YarnClusterApplication"
  private[deploy] val REST_CLUSTER_SUBMIT_CLASS = classOf[RestSubmissionClientApp].getName()
  private[deploy] val STANDALONE_CLUSTER_SUBMIT_CLASS = classOf[ClientApp].getName()
  private[deploy] val KUBERNETES_CLUSTER_SUBMIT_CLASS =
    "org.apache.spark.deploy.k8s.submit.KubernetesClientApplication"

  override def main(args: Array[String]): Unit = {
    // 这里先创建了SparkSubmit实例
    val submit = new SparkSubmit() {
      self =>
      
      // 重写了class SparkSubmit的解析加载参数方法
      override protected def parseArguments(args: Array[String]): SparkSubmitArguments = {
        new SparkSubmitArguments(args) {
          override protected def logInfo(msg: => String): Unit = self.logInfo(msg)

          override protected def logWarning(msg: => String): Unit = self.logWarning(msg)
        }
      }

      // 日志输出方法
      override protected def logInfo(msg: => String): Unit = printMessage(msg)

      // warning输出方法
      override protected def logWarning(msg: => String): Unit = printMessage(s"Warning: $msg")

      // 重写任务提交方法,捕获异常
      override def doSubmit(args: Array[String]): Unit = {
        try {
          // 这里会进入class SparkSubmit的doSubmit()
          super.doSubmit(args)
        } catch {
          case e: SparkUserAppException =>
            exitFn(e.exitCode)
        }
      }

    }

    // 调用上面SparkSubmit实例的doSubmit()
    submit.doSubmit(args)
  }

  /**
   * Return whether the given primary resource represents a user jar.
   */
  private[deploy] def isUserJar(res: String): Boolean = {
    !isShell(res) && !isPython(res) && !isInternal(res) && !isR(res)
  }

  /**
   * Return whether the given primary resource represents a shell.
   */
  private[deploy] def isShell(res: String): Boolean = {
    (res == SPARK_SHELL || res == PYSPARK_SHELL || res == SPARKR_SHELL)
  }

  /**
   * Return whether the given main class represents a sql shell.
   */
  private[deploy] def isSqlShell(mainClass: String): Boolean = {
    mainClass == "org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver"
  }

  /**
   * Return whether the given main class represents a thrift server.
   */
  private def isThriftServer(mainClass: String): Boolean = {
    mainClass == "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2"
  }

  /**
   * Return whether the given primary resource requires running python.
   */
  private[deploy] def isPython(res: String): Boolean = {
    res != null && res.endsWith(".py") || res == PYSPARK_SHELL
  }

  /**
   * Return whether the given primary resource requires running R.
   */
  private[deploy] def isR(res: String): Boolean = {
    res != null && res.endsWith(".R") || res == SPARKR_SHELL
  }

  private[deploy] def isInternal(res: String): Boolean = {
    res == SparkLauncher.NO_RESOURCE
  }

}

class SparkSubmit

Object SparkSubmit中，创建了SparkSubmit实例

private[spark] class SparkSubmit extends Logging {

  import DependencyUtils._
  import SparkSubmit._

  // 执行Submit的方法
  def doSubmit(args: Array[String]): Unit = {
    // Initialize logging if it hasn't been done yet. Keep track of whether logging needs to
    // be reset before the application starts.
    // 初始化logging系统,并跟日志判断是否需要在app启动时重启
    val uninitLog = initializeLogIfNecessary(true, silent = true)

    // 调用parseArguments()解析参数,解析了提交的参数及spark配置文件
    val appArgs = parseArguments(args)

    // 参数不重复则输出配置
    if (appArgs.verbose) {
      logInfo(appArgs.toString)
    }

    // 匹配输入的执行请求,也就是提交,终止,请求状态和打印版本
    // 在解析的时候将执行状态封装到了SparkSubmitAction中,这里进行匹配
    // 如果没有执行状态,则SparkSubmitArguments默认设置为SparkSubmitAction.SUBMIT
    // 这里提交会进入submit()
    appArgs.action match {
      case SparkSubmitAction.SUBMIT => submit(appArgs, uninitLog)
      case SparkSubmitAction.KILL => kill(appArgs)
      case SparkSubmitAction.REQUEST_STATUS => requestStatus(appArgs)
      case SparkSubmitAction.PRINT_VERSION => printVersion()
    }
  }

 /**
   * 解析参数的方法
   * 这里首先进入了Object SparkSubmit重写的parseArguments()中
   * parseArguments其实就是SparkSubmitArguments类的实例，先创建了SparkSubmitArguments(args)实例
   * 而SparkSubmitArguments继承了SparkSubmitArgumentsParser抽象类
   * SparkSubmitArgumentsParser继承了SparkSubmitOptionParser
   * SparkSubmitOptionParser其实也是launcher.main中解析参数的OptionParser.parser()继承的父类
   * SparkSubmitArguments类中，定义了一堆参数，其实就是各种运行模式需要的参数。
   * 这里解析了submit所有模式需要的参数和spark默认配置
   */
  protected def parseArguments(args: Array[String]): SparkSubmitArguments = {
    new SparkSubmitArguments(args)
  }

  /**
   * Kill an existing submission using the REST protocol. Standalone and Mesos cluster mode only.
   */
  // 终止Submit,只有Standalone和Mesos模式有用
  private def kill(args: SparkSubmitArguments): Unit = {
    new RestSubmissionClient(args.master)
      .killSubmission(args.submissionToKill)
  }

  /**
   * Request the status of an existing submission using the REST protocol.
   * Standalone and Mesos cluster mode only.
   */
  // 请求状态,只有Standalone和Mesos模式有用
  private def requestStatus(args: SparkSubmitArguments): Unit = {
    new RestSubmissionClient(args.master)
      .requestSubmissionStatus(args.submissionToRequestStatusFor)
  }

  /** Print version information to the log. */
  // 打印版本信息
  private def printVersion(): Unit = {
    logInfo("""Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version %s
      /_/
                        """.format(SPARK_VERSION))
    logInfo("Using Scala %s, %s, %s".format(
      Properties.versionString, Properties.javaVmName, Properties.javaVersion))
    logInfo(s"Branch $SPARK_BRANCH")
    logInfo(s"Compiled by user $SPARK_BUILD_USER on $SPARK_BUILD_DATE")
    logInfo(s"Revision $SPARK_REVISION")
    logInfo(s"Url $SPARK_REPO_URL")
    logInfo("Type --help for more information.")
  }
}

 submit()

通过匹配会进入submit()，先准备运行环境，然后调用doRunMain()，再调用runMain()。

  /**
   * Submit the application using the provided parameters.
   *
   * This runs in two steps. First, we prepare the launch environment by setting up
   * the appropriate classpath, system properties, and application arguments for
   * running the child main class based on the cluster manager and the deploy mode.
   * Second, we use this launch environment to invoke the main method of the child
   * main class.
   */
 /** 
   * 通过匹配SUBMIT执行的submit()
   * 如上所说,分成两部
   * 首先是根据不同调度模式和yarn不同模式,导入调用类的路径,默认配置及输入参数,准备相应的启动环境
   * 然后通过对应的环境来调用相应子类的main方法
   * 这里因为涉及到重复调用,所以采用了@tailrec尾递归,即重复调用方法的最后一句并返回结果
   * 即:runMain(childArgs, childClasspath, sparkConf, childMainClass, args.verbose)
   */
  @tailrec
  private def submit(args: SparkSubmitArguments, uninitLog: Boolean): Unit = {

 /** 先准备运行环境,传入解析的各种参数
   * 这里会先进入
   * lazy val secMgr = new SecurityManager(sparkConf)
   * 先初始化SecurityManager后,再进入prepareSubmitEnvironment()
   * prepareSubmitEnvironment()代码比较长,放到最下面去解析
   */
    val (childArgs, childClasspath, sparkConf, childMainClass) = prepareSubmitEnvironment(args)

    // 主要是调用runMain()启动相应环境的main()的方法
    // 环境准备好以后,会先往下运行判断,这里是在等着调用
    def doRunM