Spark源码分析之Master主备切换机制

Master作为Spark standalone模式的核心，如果Master出现异常，那么集群就不能正常工作。所以Spark会从Standby中选择一个节点作为Master.

 

Spark支持以下几种策略，这种策略可以通过配置文件spark-env.sh配置spark.deploy.recoveryMode

# ZOOKEEPER: 集群元数据持久化到zookeeper,当master出现异常的时候，zookeeper会通过选举机制选举出新的Master,新的Master接管集群时需要从zookeeper获取持久化信息，并根据这些信息恢复集群状态

# FILESYSTEM: 集群的元数据持久化到文件系统，当Master出现异常的时候，只要在该机器上重启Master，启动后的Master获取持久化信息并根据持久化信息恢复集群状态

# CUSTOM: 自定义恢复模式，实现StandaloneRecoveryModeFactory抽象类进行实现，并把该类配置到配置文件，当Master出现异常，会根据用户自定义的方式进行恢复集群状况

# NONE: 不持久化集群元数据，当Master出现异常时，新启动的Master不进行恢复集群状态

 

我们知道Master继承了ThreadSafeRpcEndpoint，从而可以进行Rpc通信，而且它还继承了特质LeaderElectable，从而可以进行选举Leader和撤销Leader身份。

trait LeaderElectable {
  def electedLeader(): Unit
  def revokedLeadership(): Unit
}

 

一 HA相关的属性

state = RecoveryState.STANDBY: 初始状态设置为standby

PersistenceEngine persistenceEngine：持久化引擎，用于持久化集群元数据

LeaderElectionAgent leaderElectionAgent：用于Leader选举的代理

 

二 根据恢复模式策略，初始化对应的持久化对象和选举代理对象

在Master启动的时候，会根据当前配置的spark.deploy.recoveryMode策略，如果没有默认是None得到对应的持久化引擎和用于选举Leader的持久化代理

val (persistenceEngine_, leaderElectionAgent_) = RECOVERY_MODE match {
  // 如果恢复模式是ZOOKEEPER，那么通过zookeeper来持久化恢复状态
  case "ZOOKEEPER" =>
    logInfo("Persisting recovery state to ZooKeeper")
    val zkFactory =
      new ZooKeeperRecoveryModeFactory(conf, serializer)
    (zkFactory.createPersistenceEngine(), zkFactory.createLeaderElectionAgent(this))
  // 如果恢复模式是文件系统，那么通过文件系统来持久化恢复状态
  case "FILESYSTEM" =>
    val fsFactory =
      new FileSystemRecoveryModeFactory(conf, serializer)
    (fsFactory.createPersistenceEngine(), fsFactory.createLeaderElectionAgent(this))
  // 如果恢复模式是定制的，那么指定你定制的全路径类名，然后产生相关操作来持久化恢复状态
  case "CUSTOM" =>
    val clazz = Utils.classForName(conf.get("spark.deploy.recoveryMode.factory"))
    val factory = clazz.getConstructor(classOf[SparkConf], classOf[Serializer])
      .newInstance(conf, serializer)
      .asInstanceOf[StandaloneRecoveryModeFactory]
    (factory.createPersistenceEngine(), factory.createLeaderElectionAgent(this))
  case _ =>
    (new BlackHolePersistenceEngine(), new MonarchyLeaderAgent(this))
}
persistenceEngine = persistenceEngine_
leaderElectionAgent = leaderElectionAgent_

 

三 Master异常进行恢复

3.1 开始进行选举

case ElectedLeader =>
  // 根据配置的spark.deploy.recoveryMode，决定使用哪一种recovery 模式，然后决定采用什么持久化engine
  // 然后根据持久化engine，读取持久化数据，得到一个已经存储的(applicationInfo,driverInfo,workerInfo)
  // 的一个三元组
  val (storedApps, storedDrivers, storedWorkers) = persistenceEngine.readPersistedData(rpcEnv)
  // 根据读取的持久化数据是否都为空，判断RecoveryState状态是否是alive还是recovering
  state = if (storedApps.isEmpty && storedDrivers.isEmpty && storedWorkers.isEmpty) {
    RecoveryState.ALIVE
  } else {
    RecoveryState.RECOVERING
  }
  logInfo("I have been elected leader! New state: " + state)
  // 如果处于recovering状态
  if (state == RecoveryState.RECOVERING) {
    // 开始恢复数据
    beginRecovery(storedApps, storedDrivers, storedWorkers)
    // 后台线程调度一个线程去定期检查master完成了恢复工作
    recoveryCompletionTask = forwardMessageThread.schedule(new Runnable {
      override def run(): Unit = Utils.tryLogNonFatalError {
        self.send(CompleteRecovery)
      }
    }, WORKER_TIMEOUT_MS, TimeUnit.MILLISECONDS)
  }
// 如果是CompleteRecovery，则调用completeRecovery
case CompleteRecovery => completeRecovery()

 

 

3.2 开始进行恢复

private def beginRecovery(storedApps: Seq[ApplicationInfo], storedDrivers: Seq[DriverInfo],
    storedWorkers: Seq[WorkerInfo]) {
  // 遍历每一个存储的application，注册该application,并且发送MasterChanged请求
  for (app <- storedApps) {
    logInfo("Trying to recover app: " + app.id)
    try {
      registerApplication(app)
      // 将该application状态置为UNKNOWN状态
      app.state = ApplicationState.UNKNOWN
      // 然后这个app向master发送MasterChanged请求
      app.driver.send(MasterChanged(self, masterWebUiUrl))
    } catch {
      case e: Exception => logInfo("App " + app.id + " had exception on reconnect")
    }
  }
  // 遍历每一个存储的driver, 更新master所维护的driver集合
  for (driver <- storedDrivers) {
    drivers += driver
  }
  // 遍历每一个存储的wroker,然后向master注册worker
  for (worker <- storedWorkers) {
    logInfo("Trying to recover worker: " + worker.id)
    try {
      // 注册worker，就是更新master的woker集合，和worker相关的映射列表
      registerWorker(worker)
      // 将该worker状态置为UNKNOWN状态
      worker.state = WorkerState.UNKNOWN
      // 然后改worker向master发送MasterChanged请求
      worker.endpoint.send(MasterChanged(self, masterWebUiUrl))
    } catch {
      case e: Exception => logInfo("Worker " + worker.id + " had exception on reconnect")
    }
  }
}

3.3 worker接受到MasterChange消息，向master发送消息：

WorkerSchedulerStateResponse

case MasterChanged(masterRef, masterWebUiUrl) =>
  logInfo("Master has changed, new master is at " + masterRef.address.toSparkURL)
  // 获取新的master的url和master，连接状态置为true,取消之前的尝试重新注册
  changeMaster(masterRef, masterWebUiUrl)
  // 创建当前节点executors的简单描述对象ExecutorDescription
  val execs = executors.values.
    map(e => new ExecutorDescription(e.appId, e.execId, e.cores, e.state))
  // 向新的master发送WorkerSchedulerStateResponse消息，然后会做一些操作
  masterRef.send(WorkerSchedulerStateResponse(workerId, execs.toList, drivers.keys.toSeq))

3.4 Application接受到MasterChange消息，向Master发送消息：MasterChangeAcknowledged

case MasterChanged(masterRef, masterWebUiUrl) =>
  logInfo("Master has changed, new master is at " + masterRef.address.toSparkURL)
  master = Some(masterRef)
  alreadyDisconnected = false
  masterRef.send(MasterChangeAcknowledged(appId.get))

3.5 Master接收到WorkerSchedulerStateResponse和MasterChangeAcknowledged消息后，调用completeRecovery操作，kill掉那些不响应但是状态不是UNKNOWN的worker和application

case WorkerSchedulerStateResponse(workerId, executors, driverIds) =>
  // 根据workerId获取worker
  idToWorker.get(workerId) match {
    case Some(worker) =>
      logInfo("Worker has been re-registered: " + workerId)
      // worker状态置为alive
      worker.state = WorkerState.ALIVE
      // 从指定的executor中过滤出哪些是有效的executor
      val validExecutors = executors.filter(exec => idToApp.get(exec.appId).isDefined)
      // 遍历有效的executors
      for (exec <- validExecutors) {
        // 获取executor所对应的app
        val app = idToApp.get(exec.appId).get
        // 为app设置executor，比如哪一个worker,多少核数等资源
        val execInfo = app.addExecutor(worker, exec.cores, Some(exec.execId))
        // 将该executor添加到该woker上
        worker.addExecutor(execInfo)
        execInfo.copyState(exec)
      }
      // 将所有的driver设置为RUNNING然后加入到worker中
      for (driverId <- driverIds) {
        drivers.find(_.id == driverId).foreach { driver =>
          driver.worker = Some(worker)
          driver.state = DriverState.RUNNING
          worker.drivers(driverId) = driver
        }
      }
    case None =>
      logWarning("Scheduler state from unknown worker: " + workerId)
  }
  // 判断当前是否可以进行completeRecovery操作，如果可以进行completeRecovery操作
  if (canCompleteRecovery) { completeRecovery() }

case MasterChangeAcknowledged(workerId, executors, driverIds) =>
  // 根据workerId获取worker
  idToWorker.get(workerId) match {
    case Some(worker) =>
      logInfo("Worker has been re-registered: " + workerId)
      // worker状态置为alive
      worker.state = WorkerState.ALIVE
      // 从指定的executor中过滤出哪些是有效的executor
      val validExecutors = executors.filter(exec => idToApp.get(exec.appId).isDefined)
      // 遍历有效的executors
      for (exec <- validExecutors) {
        // 获取executor所对应的app
        val app = idToApp.get(exec.appId).get
        // 为app设置executor，比如哪一个worker,多少核数等资源
        val execInfo = app.addExecutor(worker, exec.cores, Some(exec.execId))
        // 将该executor添加到该woker上
        worker.addExecutor(execInfo)
        execInfo.copyState(exec)
      }
      // 将所有的driver设置为RUNNING然后加入到worker中
      for (driverId <- driverIds) {
        drivers.find(_.id == driverId).foreach { driver =>
          driver.worker = Some(worker)
          driver.state = DriverState.RUNNING
          worker.drivers(driverId) = driver
        }
      }
    case None =>
      logWarning("Scheduler state from unknown worker: " + workerId)
  }
  // 判断当前是否可以进行completeRecovery操作，如果可以进行completeRecovery操作
  if (canCompleteRecovery) { completeRecovery() }

3.6 完成恢复

private def completeRecovery() {
  // 如果状态不是recovering则返回
  if (state != RecoveryState.RECOVERING) { return }
  // 然后状态置为completing_recovery（正处于恢复中）
  state = RecoveryState.COMPLETING_RECOVERY

  // 杀掉那些不响应但是状态不是UNKNOWN的worker和application
  workers.filter(_.state == WorkerState.UNKNOWN).foreach(removeWorker)
  apps.filter(_.state == ApplicationState.UNKNOWN).foreach(finishApplication)

  // 重新调度未被任何worker声称的driver，即还没有worker来运行
  drivers.filter(_.worker.isEmpty).foreach { d =>
    logWarning(s"Driver ${d.id} was not found after master recovery")
    // 如果是driver是监管者，则重新发起driver,否则删除driver
    if (d.desc.supervise) {
      logWarning(s"Re-launching ${d.id}")
      relaunchDriver(d)
    } else {
      removeDriver(d.id, DriverState.ERROR, None)
      logWarning(s"Did not re-launch ${d.id} because it was not supervised")
    }
  }
  // 然后状态置为alive
  state = RecoveryState.ALIVE
  // 重新分配资源,调度driver和application
  schedule()
  logInfo("Recovery complete - resuming operations!")
}

 

四 触发选举流程

我们这里主要以zookeeper为例子：

当节点启动的时候，会调用onstart方法，然后个根据恢复模式zookeeper，会初始化ZooKeeperPersistenceEngine和ZooKeeperLeaderElectionAgent。ZooKeeperPersistenceEngine主要负责持久化app,driver，woker等信息，ZooKeeperLeaderElectionAgent主要负责监听master状态变化和选举Leader。

ZooKeeperLeaderElectionAgent在初始化的时候，就会启动，它在启动的时候会创建LeaderLatch，Leader启动就会启动一个后台线程去检测Leader状态，然后ZooKeeperLeaderElectionAgent根据当前节点状态判断是不是Leader，向Master发送ElectedLeader和RevokedLeadership消息