接下来的几篇文章我们会结合源码来分析注册Application以及启动Executor并向Driver注册的具体流程。
上文我们跟踪源码到了SparkDeploySchedulerBackend的start()方法的实例化AppClient部分，同时SparkDeploySchedulerBackend的start()方法中首先执行的就是其父类也就是CoarseGrainedSchedulerBackend的start方法，上面的两部分源码如下：
SparkDeploySchedulerBackend中的start()方法：
override def start() {
// 首先调用父类也就是CoarseGrainedSchedulerBackend的start方法，最重要的就是创建并注册DriverEndpoint
super.start()
...
client = new AppClient(sc.env.rpcEnv, masters, appDesc, this, conf)
client.start()
launcherBackend.setState(SparkAppHandle.State.SUBMITTED)
waitForRegistration()
launcherBackend.setState(SparkAppHandle.State.RUNNING)
}
CoarseGrainedSchedulerBackend中的start()方法：
override def start() {
val properties = new ArrayBuffer[(String, String)]
for ((key, value) <- scheduler.sc.conf.getAll) {
if (key.startsWith("spark.")) {
properties += ((key, value))
}
}
// TODO (prashant) send conf instead of properties
// 这里的ENDPOINT_NAME="CoarseGrainedScheduler"
driverEndpoint = rpcEnv.setupEndpoint(ENDPOINT_NAME, createDriverEndpoint(properties))
}
DriverEndpoint
下面我们继续追踪源码，首先来看CoarseGrainedSchedulerBackend的start()方法(见上面的源码)，从上面的源码可以看出主要的操作就是创建了DriverEndpoint并向RpcEnv进行注册，所以我们进入createDriverEndpoint方法：
protected def createDriverEndpoint(properties: Seq[(String, String)]): DriverEndpoint = {
// 实例化DriverEndpoint
new DriverEndpoint(rpcEnv, properties)
}
内部实例化了DriverEndpoint：
class DriverEndpoint(override val rpcEnv: RpcEnv, sparkProperties: Seq[(String, String)])
extends ThreadSafeRpcEndpoint with Logging {
// If this DriverEndpoint is changed to support multiple threads,
// then this may need to be changed so that we don't share the serializer
// instance across threads
private val ser = SparkEnv.get.closureSerializer.newInstance()
override protected def log = CoarseGrainedSchedulerBackend.this.log
protected val addressToExecutorId = new HashMap[RpcAddress, String]
private val reviveThread =
ThreadUtils.newDaemonSingleThreadScheduledExecutor("driver-revive-thread")
可以看出DriverEndpoint继承自ThreadSafeRpcEndpoint，所以DriverEndpoint是一个消息循环体，那么他到底负责与谁进行通信呢？我们继续追踪源码，实例化完成后回到CoarseGrainedSchedulerBackend中的start方法：向RpcEnv进行注册，注册的名字是ENDPOINT_NAME，即"CoarseGrainedScheduler"，注册的时候会调用DriverEndpoint的onStart方法(Rpc内部的机制决定的，具体可以参考Spark RPC 到底是个什么鬼？)：
override def onStart() {
// Periodically revive offers to allow delay scheduling to work
val reviveIntervalMs = conf.getTimeAsMs("spark.scheduler.revive.interval", "1s")
reviveThread.scheduleAtFixedRate(new Runnable {
override def run(): Unit = Utils.tryLogNonFatalError {
Option(self).foreach(_.send(ReviveOffers))
}
}, 0, reviveIntervalMs, TimeUnit.MILLISECONDS)
}
这里使用的是延迟调度的机制，关于延迟调度我们会单独用文章进行阐述，现在我们继续追踪源码，可以看到内部实际上是向自己发送了一条消息ReviveOffers，所以我们看DriverEndpoint接收到这条消息后都做了什么：
case ReviveOffers =>
makeOffers()
可以看到是执行了makeOffers()方法：
// Make fake resource offers on all executors
private def makeOffers() {
// Filter out executors under killing
val activeExecutors = executorDataMap.filterKeys(executorIsAlive)
val workOffers = activeExecutors.map { case (id, executorData) =>
new WorkerOffer(id, executorData.executorHost, executorData.freeCores)
}.toSeq
launchTasks(scheduler.resourceOffers(workOffers))
}
首先就是提供计算资源，注释说的很清楚了“Make fake resource offers on all executors”，因为此时Executors还没有启动和注册，即还没有资源，所以是“fake”(假)的，等到Executors启动并注册后，这里就会获得空闲的计算资源，然后去执行launchTasks的操作，即向具体的Executor提交和运行tasks。所以CoarseGrainedSchedulerBackend端的start方法我们就追踪到这里，最后用一张图总结一下：
AppClient和ClientEndpoint
下面来看SparkDeploySchedulerBackend的start方法中剩余的那部分，即上篇文章中没有说完的那部分：
// Start executors with a few necessary configs for registering with the scheduler
val sparkJavaOpts = Utils.sparkJavaOpts(conf, SparkConf.isExecutorStartupConf)
val javaOpts = sparkJavaOpts ++ extraJavaOpts
val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpt
val appUIAddress = sc.ui.map(_.appUIAddress).getOrElse("")
val coresPerExecutor = conf.getOption("spark.executor.cores").map(_.toInt)
val appDesc = new ApplicationDescription(sc.appName, maxCores, sc.executorMemory,
command, appUIAddress, sc.eventLogDir, sc.eventLogCodec, coresPerExecutor)
client = new AppClient(sc.env.rpcEnv, masters, appDesc, this, conf)
client.start()
launcherBackend.setState(SparkAppHandle.State.SUBMITTED)
waitForRegistration()
launcherBackend.setState(SparkAppHandle.State.RUNNING)
可以看见首先创建AppClient，这里需要注意一下传进来的参数中appDesc就是上面实例化的ApplicationDescription，他封装了应用程序的一些配置参数，最重要的是command，通过他可以看出启动Executor时执行main方法的入口类就是"org.apache.spark.executor.CoarseGrainedExecutorBackend"，我们后面会提到，所以这里先提一下。我们继续看实例化AppClient的部分：
private[spark] class AppClient(
rpcEnv: RpcEnv,
masterUrls: Array[String],
appDescription: ApplicationDescription,
listener: AppClientListener,
conf: SparkConf)
extends Logging {
// 获得masters的RpcAddress
private val masterRpcAddresses = masterUrls.map(RpcAddress.fromSparkURL(_))
private val REGISTRATION_TIMEOUT_SECONDS = 20
private val REGISTRATION_RETRIES = 3
// 用来保存ClientEndpoint注册完成后返回的RpcEndpointRef，实际上是NettyRpcEndpointRef
private val endpoint = new AtomicReference[RpcEndpointRef]
// 应用程序的ID
private val appId = new AtomicReference[String]
// 是否已经完成注册
private val registered = new AtomicBoolean(false)
AppClient实例化完成后调用了他的start()方法：
def start() {
// Just launch an rpcEndpoint; it will call back into the listener.
endpoint.set(rpcEnv.setupEndpoint("AppClient", new ClientEndpoint(rpcEnv)))
}
可以看到内部创建了一个ClientEndpoint实例并向RpcEnv注册，注册的名称为“AppClient”，注册完成后将返回的NettyRpcEndpointRef赋值给AppClient的endpoint，而且注册后调用了ClientEndpoint的onStart()方法，实例化ClientEndpoint的时候执行了一些初始化操作，基本上都是和线程池和消息相关，这里就不列出来了，我们现在关注的是这个onStart()方法：
override def onStart(): Unit = {
try {
registerWithMaster(1)
} catch {
case e: Exception =>
logWarning("Failed to connect to master", e)
markDisconnected()
stop()
}
}
进入registerWithMaster，即向Master进行注册：
private def registerWithMaster(nthRetry: Int) {
registerMasterFutures.set(tryRegisterAllMasters())
registrationRetryTimer.set(registrationRetryThread.scheduleAtFixedRate(new Runnable {
override def run(): Unit = {
if (registered.get) {
registerMasterFutures.get.foreach(_.cancel(true))
registerMasterThreadPool.shutdownNow()
} else if (nthRetry >= REGISTRATION_RETRIES) {
markDead("All masters are unresponsive! Giving up.")
} else {
registerMasterFutures.get.foreach(_.cancel(true))
registerWithMaster(nthRetry + 1)
}
}
}, REGISTRATION_TIMEOUT_SECONDS, REGISTRATION_TIMEOUT_SECONDS, TimeUnit.SECONDS))
}
上面的源码说明：ClientEndpoint需要向所有的masters进行异步的注册，因为ClientEndpoint并不知道哪个Master是Active级别的，如果注册失败会按照一定的时间间隔进行一定次数的重试，可以看到具体的注册是执行的tryRegisterAllMasters方法：
private def tryRegisterAllMasters(): Array[JFuture[_]] = {
for (masterAddress <- masterRpcAddresses) yield {
registerMasterThreadPool.submit(new Runnable {
override def run(): Unit = try {
if (registered.get) {
return
}
// 打印日志，我们在使用spark-submit的client模式提交程序的时候可以看到这条日志
logInfo("Connecting to master " + masterAddress.toSparkURL + "...")
// 获取Master的RpcEndpointRef，实际上是NettyRpcEndpointRef，用来向Master发送消息
val masterRef =
rpcEnv.setupEndpointRef(Master.SYSTEM_NAME, masterAddress, Master.ENDPOINT_NAME)
// 向Master发送消息，注意这里的self是指ClientEndpoint
masterRef.send(RegisterApplication(appDescription, self))
} catch {
case ie: InterruptedException => // Cancelled
case NonFatal(e) => logWarning(s"Failed to connect to master $masterAddress", e)
}
})
}
}
可以看到是使用了线程池进行异步注册，线程池的大小就是Master的个数，这样可以保证同一时间可以向多个Masters发送注册请求，因为ClientEndpoint并不知道哪个Master是Active级别的，如果一个一个注册会产生阻塞。内部首先获得Master的RpcEndpoint，实际上是NettyRpcEndpointRef，然后通过这个masterRef向Master发送消息：RegisterApplication(appDescription, self)，这里的self指的就是ClientEndpoint。
下面我们看Master接收到消息是如何进行处理的：
case RegisterApplication(description, driver) => {
// TODO Prevent repeated registrations from some driver
if (state == RecoveryState.STANDBY) {
// ignore, don't send response
// 如果该Master是STANDBY的状态就直接忽略，什么也不做
} else {
logInfo("Registering app " + description.name)
// 创建Application，注意这里传入的driver就是ClientEndpoint
val app = createApplication(description, driver)
// 注册上一步创建的Application
registerApplication(app)
logInfo("Registered app " + description.name + " with ID " + app.id)
// 向持久化引擎中加入该Application的信息，用于HA，底层可以依赖Zookeeper实现
persistenceEngine.addApplication(app)
// 向driver即ClientEndpoint发送消息RegisteredApplication(app.id, self)，这里的self就是Master
driver.send(RegisteredApplication(app.id, self))
// 最后执行Master的schedule()方法
schedule()
}
}
主要的流程已经在源码中进行注释，现在我们一一进行分析，为了方便大家的阅读，这里我使用如下标题加以区分(持久化不是本文的重点，所以此处省略)：
- 创建Application
- 注册Application
- 向Driver发送消息
- 执行schedule()
创建Application
首先来看Application的创建过程：
private def createApplication(desc: ApplicationDescription, driver: RpcEndpointRef):
ApplicationInfo = {
val now = System.currentTimeMillis()
val date = new Date(now)
val appId = newApplicationId(date) // 根据当前的时间构建Application的Id
// 实例化ApplicationInfo
new ApplicationInfo(now, appId, desc, date, driver, defaultCores)
}
而具体实例化的时候是执行了init()方法，限于篇幅这里就不一一说明了：
private def init() {
state = ApplicationState.WAITING
executors = new mutable.HashMap[Int, ExecutorDesc]
coresGranted = 0
endTime = -1L
appSource = new ApplicationSource(this)
nextExecutorId = 0
removedExecutors = new ArrayBuffer[ExecutorDesc]
executorLimit = Integer.MAX_VALUE
appUIUrlAtHistoryServer = None
}
注册Application
下面就是注册Application：
private def registerApplication(app: ApplicationInfo): Unit = {
val appAddress = app.driver.address
// 首先要判断是否发过来注册请求的driver的RpcAddress是否已经存在
if (addressToApp.contains(appAddress)) {
logInfo("Attempted to re-register application at same address: " + appAddress)
return
}
// 关于统计系统的部分，我们这里不深入分析
applicationMetricsSystem.registerSource(app.appSource)
// 下面就是向Master的各种数据结构中存入该Application的信息的操作
apps += app
idToApp(app.id) = app
endpointToApp(app.driver) = app
addressToApp(appAddress) = app
waitingApps += app
}
向Driver发送消息
然后就是向driver即ClientEndpoint发送消息：RegisteredApplication(app.id, self)，ClientEndpoint在接收到消息后的处理如下：
case RegisteredApplication(appId_, masterRef) =>
// FIXME How to handle the following cases?
// 一些意外的情况，感兴趣的可以研究下怎么解决。
// 1. A master receives multiple registrations and sends back multiple
// RegisteredApplications due to an unstable network.
// 2. Receive multiple RegisteredApplication from different masters because the master is
// changing.
// 将获得的appId_存入到appId中，注意这里的appId是属于AppClient的
// 下面的registered、master也是如此
appId.set(appId_)
registered.set(true)
master = Some(masterRef)
listener.connected(appId.get)
至此，我们使用一张图总结一下：
执行schedule()
我们在Spark-Core源码精读(2)、Master中的schedule详解已经分析了一部分源码，我们简单的回顾一下(我们这里都是假设集群是Standalone模式的，所以直接看Executor启动的部分)：
schedule中的最后一句就是：startExecutorsOnWorkers()，我们从这里开始：
private def startExecutorsOnWorkers(): Unit = {
// Right now this is a very simple FIFO scheduler. We keep trying to fit in the first app
// in the queue, then the second app, etc.
for (app <- waitingApps if app.coresLeft > 0) {
val coresPerExecutor: Option[Int] = app.desc.coresPerExecutor
// Filter out workers that don't have enough resources to launch an executor
val usableWorkers = workers.toArray.filter(_.state == WorkerState.ALIVE)
.filter(worker => worker.memoryFree >= app.desc.memoryPerExecutorMB &&
worker.coresFree >= coresPerExecutor.getOrElse(1))
.sortBy(_.coresFree).reverse
val assignedCores = scheduleExecutorsOnWorkers(app, usableWorkers, spreadOutApps)
// Now that we've decided how many cores to allocate on each worker, let's allocate them
for (pos <- 0 until usableWorkers.length if assignedCores(pos) > 0) {
allocateWorkerResourceToExecutors(
app, assignedCores(pos), coresPerExecutor, usableWorkers(pos))
}
}
}
上面源码首先决定在每个worker上分配多少个cores，即scheduleExecutorsOnWorkers方法，默认是将exeutors分配到尽可能多的workers上，然后就是使用allocateWorkerResourceToExecutors方法在对应的Woker上启动Executor并为其分配计算资源：
private def allocateWorkerResourceToExecutors(
app: ApplicationInfo,
assignedCores: Int,
coresPerExecutor: Option[Int],
worker: WorkerInfo): Unit = {
// If the number of cores per executor is specified, we divide the cores assigned
// to this worker evenly among the executors with no remainder.
// Otherwise, we launch a single executor that grabs all the assignedCores on this worker.
val numExecutors = coresPerExecutor.map { assignedCores / _ }.getOrElse(1)
val coresToAssign = coresPerExecutor.getOrElse(assignedCores)
for (i <- 1 to numExecutors) {
val exec = app.addExecutor(worker, coresToAssign)
launchExecutor(worker, exec)
app.state = ApplicationState.RUNNING
}
}
内部执行的是launchExecutor(worker, exec)，注意exec就是指ExecutorDesc，即Executor的描述信息，我们继续追踪launchExecutor：
private def launchExecutor(worker: WorkerInfo, exec: ExecutorDesc): Unit = {
logInfo("Launching executor " + exec.fullId + " on worker " + worker.id)
worker.addExecutor(exec)
worker.endpoint.send(LaunchExecutor(masterUrl,
exec.application.id, exec.id, exec.application.desc, exec.cores, exec.memory))
exec.application.driver.send(
ExecutorAdded(exec.id, worker.id, worker.hostPort, exec.cores, exec.memory))
}
可以看到具体分成了两步：
- 向worker发送启动Executor的消息
- 启动完成后向driver发送ExecutorAdded的消息，这里的driver就是ClientEndpoint
考虑到大家的阅读体验，我们将这两部分放到下一篇文章进行阐述。
本文参照的是Spark 1.6.3版本的源码，同时给出Spark 2.1.0版本的连接：
Spark 1.6.3 源码
Spark 2.1.0 源码
本文为原创，欢迎转载，转载请注明出处、作者，谢谢！