Spark源码学习（9）——Spark On Yarn

本文要解决的问题：

了解Spark在Yarn平台上的运行过程。

修改配置文件

首先需要修改配置文件spark-env.sh。在这个文件中需要添加两个属性：

Export HADOOP_HOME=/../hadoop..

ExportHADOOP_CONF_DIR=/../hadoop/etc/hadoop

这里，一个是要hadoop的home目录。一个是配置文件目录。

还需要配置一个就是spark-defaults.conf这个文件：

需要修改红色框内的文件。下面看下这个文件里面的内容：

在spark的源文件中给出了一些配置参数的示例。另外它还下面一个可以配置的属性

属性名	说明	默认值
spark.yarn.applicationMaster.waitTries	RM等待Spark AppMaster启动次数，也就是SparkContext初始化次数。超过这个数值，启动失败。	10
spark.yarn.submit.file.replication	应用程序上载到HDFS的文件的复制因子	3
spark.yarn.preserve.staging.files	设置为true，在job结束后，将stage相关的文件保留而不是删除。	false
spark.yarn.scheduler.heartbeat.interval-ms	Spark AppMaster发送心跳信息给YARN RM的时间间隔	5000
spark.yarn.max.executor.failures	导致应用程序宣告失败的最大executor失败数	2倍于executor数
spark.yarn.historyServer.address	Spark history server的地址（不要加http://）。这个地址会在应用程序完成后提交给YARN RM，使得将信息从RM UI连接到history server UI上。	无

运行流程

下面是Spark On Yarn的流程图：

下面再来看看具体的源码：

Client

在Client类中的main方法实例话Client：new Client(args, sparkConf).run()。在run方法中，又调用了val appId = runApp()方法。runApp()源码如下

def runApp() = {

    validateArgs() 

    init(yarnConf)

    start()

    logClusterResourceDetails()

    val newApp = super.getNewApplication()

    val appId = newApp.getApplicationId()

    verifyClusterResources(newApp)

    val appContext = createApplicationSubmissionContext(appId)

    val appStagingDir = getAppStagingDir(appId)

    val localResources = prepareLocalResources(appStagingDir)

    val env = setupLaunchEnv(localResources, appStagingDir)

    val amContainer = createContainerLaunchContext(newApp, localResources, env)

    val capability = Records.newRecord(classOf[Resource]).asInstanceOf[Resource]

    // Memory for the ApplicationMaster.

    capability.setMemory(args.amMemory + memoryOverhead)

    amContainer.setResource(capability) 

    appContext.setQueue(args.amQueue)

    appContext.setAMContainerSpec(amContainer)

    appContext.setUser(UserGroupInformation.getCurrentUser().getShortUserName())

    submitApp(appContext)

    appId

  }

1）这里首先对一些参数配置的校验，然后初始化、启动Client

2）提交请求到ResouceManager，检查集群的内存情况。

3）设置一些参数，请求队列

4）正式提交APP

ApplicationManager

AM负责运行Spark Application的Driver程序，并分配执行需要的Executors。里面也有个main方法实例化AM并调用run，源码如下：

final def run(): Int = {
    try {
      val appAttemptId = client.getAttemptId()

      if (isClusterMode) {
        // Set the web ui port to be ephemeral for yarn so we don't conflict with
        // other spark processes running on the same box
        System.setProperty("spark.ui.port", "0")

        // Set the master and deploy mode property to match the requested mode.
        System.setProperty("spark.master", "yarn")
        System.setProperty("spark.submit.deployMode", "cluster")

        // Set this internal configuration if it is running on cluster mode, this
        // configuration will be checked in SparkContext to avoid misuse of yarn cluster mode.
        System.setProperty("spark.yarn.app.id", appAttemptId.getApplicationId().toString())
      }

      logInfo("ApplicationAttemptId: " + appAttemptId)

      val fs = FileSystem.get(yarnConf)

      // This shutdown hook should run *after* the SparkContext is shut down.
      val priority = ShutdownHookManager.SPARK_CONTEXT_SHUTDOWN_PRIORITY - 1
      ShutdownHookManager.addShutdownHook(priority) { () =>
        val maxAppAttempts = client.getMaxRegAttempts(sparkConf, yarnConf)
        val isLastAttempt = client.getAttemptId().getAttemptId() >= maxAppAttempts

        if (!finished) {
          // The default state of ApplicationMaster is failed if it is invoked by shut down hook.
          // This behavior is different compared to 1.x version.
          // If user application is exited ahead of time by calling System.exit(N), here mark
          // this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
          // System.exit(0) to terminate the application.
          finish(finalStatus,
            ApplicationMaster.EXIT_EARLY,
            "Shutdown hook called before final status was reported.")
        }

        if (!unregistered) {
          // we only want to unregister if we don't want the RM to retry
          if (finalStatus == FinalApplicationStatus.SUCCEEDED || isLastAttempt) {
            unregister(finalStatus, finalMsg)
            cleanupStagingDir(fs)
          }
        }
      }