本文以Spark 1.6 Standalone模式为例，介绍用户提交Spark Job后的Job的执行流程。大体流程如下图所示
1729304-0fb8aae3eaafa180.png
当action触发执行后，Driver会创建SC对象，SC向Cluster Manager(在standalone 模式下是Spark Master) 申请Executor资源，并将job分解为stage(taskSet), stage由并行化处理的task组成，然后将task分发到不同的Executor执行，Executor在task执行完后吧结果返回给SC.
下面首先介绍SparkContext申请Executor资源的过程，整个过程如下图所示。
1729304-8b8b7416ee6e9c71.png
整个过程分为8步：
1: SparkContext创建TaskSchedulerImpl，SparkDeploySchedulerBackend和DAGSchedule
// Create and start the scheduler
val (sched, ts) = SparkContext.createTaskScheduler(this, master)
_schedulerBackend = sched
_taskScheduler = ts
_dagScheduler = new DAGScheduler(this)
DAGScheduler负责将Job划分为不同的Stage，并在每个Stage内化为出一系列可并行处理的task，然后将task递交给TaskSchedulerImpl调度。此过程之后详谈。
TaskSchedulerImpl负责通过SparkDeploySchedulerBackend来调度任务（task），目前实现了FIFO调度和Fair调度。注意如果是Yarn模式，则是通过YarnSchedulerBackend来进行调度。
- SparkDeploySchedulerBackend创建AppClient，并通过一些回调函数来得到Executor信息
client = new AppClient(sc.env.actorSystem, masters, appDesc, this, conf)
client.start()
- AppClient向Master注册Application
try {
registerWithMaster()
} catch {
case e: Exception =>
logWarning("Failed to connect to master", e)
markDisconnected()
context.stop(self)
}
- Master向Woker发送LaunchExecutor消息，同时向AppClient发送ExecutorAdded消息
Master收到RegisterApplication信息后，开始分配Executor资源。目前有两种策略（摘抄原话）：There are two modes of launching executors. The first attempts to spread out an application's executors on as many workers as possible, while the second does the opposite (i.e. launch them on as few workers as possible). The former is usually better for data locality purposes and is the default.
case RegisterApplication(description) => {
if (state == RecoveryState.STANDBY) {
// ignore, don't send response
} else {
logInfo("Registering app " + description.name)
val app = createApplication(description, sender)
registerApplication(app)
logInfo("Registered app " + description.name + " with ID " + app.id)
persistenceEngine.addApplication(app)
sender ! RegisteredApplication(app.id, masterUrl)
schedule() //分配Executor资源
}
}
Worker创建ExecutorRunner，并向Master发送ExecutorStateChanged的消息
ExecutorRunner创建CoarseGrainedSchedulerBackend
在函数fetchAndRunExecutor中，
val builder = CommandUtils.buildProcessBuilder(appDesc.command, memory,
sparkHome.getAbsolutePath, substituteVariables)
- CoarseGrainedExecutorBackend向SparkDeploySchedulerBackend发送RegisterExecutor消息
override def onStart() {
logInfo("Connecting to driver: " + driverUrl)
rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref =>
// This is a very fast action so we can use "ThreadUtils.sameThread"
driver = Some(ref)
ref.ask[RegisteredExecutor.type](
RegisterExecutor(executorId, self, hostPort, cores, extractLogUrls))
}(ThreadUtils.sameThread).onComplete {
// This is a very fast action so we can use "ThreadUtils.sameThread"
case Success(msg) => Utils.tryLogNonFatalError {
Option(self).foreach(_.send(msg)) // msg must be RegisteredExecutor
}
case Failure(e) => logError(s"Cannot register with driver: $driverUrl", e)
}(ThreadUtils.sameThread)
}
- CoarseGrainedExecutorBackend在接收到SparkDeploySchedulerBackend发送的RegisteredExecutor消息后，创建Executor
override def receive: PartialFunction[Any, Unit] = {
case RegisteredExecutor =>
logInfo("Successfully registered with driver")
val (hostname, _) = Utils.parseHostPort(hostPort)
executor = new Executor(executorId, hostname, env, userClassPath, isLocal = false)
end： executor创建成功