Hadoop yarn源码分析(一) ApplicationMaster源码分析 2021SC@SDUSC

2021SC@SDUSC

一、AM简介

Yarn的三大核心组件为ResourceManager(RM)，ApplicationMaster(AM)，NodeMananger(NM)。ApplicationMaster实际上是特定计算框架的一个实例，负责单个作业的管理和任务监控。AM负责与RM协商资源，并和NM协同来执行和监控Container。MapReduce只是可以运行在Yarn上的一种计算框架。

二、生命周期

ApplicationMaster管理部分主要由三个服务构成，分别是ApplicationMasterLauncher、AMLivelinessMonitor 和 ApplicationMasterService，他们共同管理ApplicationMaster的生命周期。ApplicationMaster从创建到销毁的流程如下：
1.启动：用户向RM提交应用程序，RM收到提交请求后，先向资源调度器申请以启动ApplicationMaster的资源。待申请到资源后，再由ApplicationMasterMasterLauncher与对应的NM通信，从而启动应用程序的ApplicationMaster。
2.注册：ApplicationMaster启动完成后，ApplicationMasterLauncher会通过事件的形式，将刚刚启动的ApplicationMaster注册到AMLivenessMonitor，以启动心跳监控。启动后，先向ApplicationMasterService注册，将所在host、端口号等信息汇报给它。
3.运行：ApplicationMaster运行过程中，周期性地向ApplicationMasterService汇报"心跳"信息确认它还"活着"("心跳"信息中含想要申请的资源描述)。ApplicationMasterService每次收到ApplicationMaster的"心跳"信息后，将通知AMLivenessMonitor更新该应用程序的最近"心跳"汇报时间。
3.注销：当应用程序运行完成后，ApplicationMaster向ApplicationMasterService发送请求，注销自己。当ApplicationMasterService收到注销请求后，标注应用程序运行状态为完成，同时通知AMLivenessMonitor移除对它"心跳"的监控。

三、源码分析

为了进一步理解ApplicationMaster的生命周期，我们从ApplicationMaster的启动、心跳、资源申请三个角度来分析源码。

3.1 AM启动流程

启动是AM生命周期的第一步，我们先从应用程序的提交，到APP/Attempt状态转换过程，再到具体的AM启动来分析。

3.1.1 应用程序提交

应用程序要先提交到yarn入口，通过YarnClient接口提交，具体方法为submitApplication()

//位置:org.apache.hadoop.yarn.client.api.YarnClient.java
public abstract ApplicationId submitApplication(
      ApplicationSubmissionContext appContext) throws YarnException,
      IOException;

YarnClientImpl中的提交入口：

//org.apache.hadoop.yarn.client.api.implYarnClientImpl.java
@Override
  public ApplicationId
      submitApplication(ApplicationSubmissionContext appContext)
          throws YarnException, IOException {
    ...
    //构建应用程序请求的上文信息
    SubmitApplicationRequest request =
        Records.newRecord(SubmitApplicationRequest.class);
    request.setApplicationSubmissionContext(appContext);

    // 自动将时间线DT添加到CLC中，仅当安全性和时间线服务都已启用时
    if (isSecurityEnabled() && timelineV1ServiceEnabled) {
      addTimelineDelegationToken(appContext.getAMContainerSpec());
    }

    //Client提交应用程序，在提交应用程序调用期间处理RM故障切换
    rmClient.submitApplication(request);
    
    ...
    while (true) {  
      // 对未能及时提交的应用程序不断重试
    }
    
    return applicationId;
  }

YarnClient与RM进行RPC通信是通过ClientRMService服务实现的，应用程序提交到服务端，会调用RMAppManager类的对应方法来处理应用程序`

//org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.java
@Override
  public SubmitApplicationResponse submitApplication(
      SubmitApplicationRequest request) throws YarnException, IOException {
    ApplicationSubmissionContext submissionContext = request
        .getApplicationSubmissionContext();
    ApplicationId applicationId = submissionContext.getApplicationId();
    CallerContext callerContext = CallerContext.getCurrent();
    //ApplicationSubmissionContext需要进行安全性验证-此处仅检查独立于RM配置的字段，依赖于RM配置的字段在RMAppManager中进行验证。
    ...
    try {
      // 调用RMAppManager来直接提交应用程序
      rmAppManager.submitApplication(submissionContext,
          System.currentTimeMillis(), user);

      LOG.info("Application with id " + applicationId.getId() + 
          " submitted by user " + user);
      RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST,
          "ClientRMService", applicationId, callerContext,
          submissionContext.getQueue(),
          submissionContext.getNodeLabelExpression());
    } catch (YarnException e) {
      LOG.info("Exception in submitting " + applicationId, e);
      RMAuditLogger.logFailure(user, AuditConstants.SUBMIT_APP_REQUEST,
          e.getMessage(), "ClientRMService",
          "Exception in submitting application", applicationId, callerContext,
          submissionContext.getQueue(),
          submissionContext.getNodeLabelExpression());
      throw e;
    }

    return recordFactory
        .newRecordInstance(SubmitApplicationResponse.class);
  }

3.1.2 APP/AppAttempt状态转换过程

从 RMAppManager 类的 rmAppManager.submitApplication() 方法，可以看到它向调度器发送 RMAppEventType.START 事件。

//org.apache.hadoop.yarn.server.resourcemanager.RMAppManager
@SuppressWarnings("unchecked")
  protected void submitApplication(
      ApplicationSubmissionContext submissionContext, long submitTime,
      String user) throws YarnException {
    ...
    try {
      if (UserGroupInformation.isSecurityEnabled()) {
        this.rmContext.getDelegationTokenRenewer()
            .addApplicationAsync(applicationId,
                BuilderUtils.parseCredentials(submissionContext),
                submissionContext.getCancelTokensWhenComplete(),
                application.getUser(),
                BuilderUtils.parseTokensConf(submissionContext));
      } else {
        // 向调度器发送 RMAppEventType.START 事件
        this.rmContext.getDispatcher().getEventHandler()
            .handle(new RMAppEvent(applicationId, RMAppEventType.START));
      }
    } catch (Exception e) {
      ...
    }
  }

RMAppAttemptState.ALLOCATED_SAVING 事件的注册状态机为 AttemptStoredTransition，此时 AppAttempt 状态已从 ALLOCATED_SAVING 转换为 ALLOCATED。

//org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.java
//  ALLOCATED_SAVING  状态转换
      .addTransition(RMAppAttemptState.ALLOCATED_SAVING, 
          RMAppAttemptState.ALLOCATED,
          RMAppAttemptEventType.ATTEMPT_NEW_SAVED, new AttemptStoredTransition())

AttemptStoredTransition 状态机，运行 AppAttempt，发送 AMLauncherEventType.LAUNCH 事件启动 AM Container

//org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.java
private static final class AttemptStoredTransition extends BaseTransition {
    @Override
    public void transition(RMAppAttemptImpl appAttempt,
                                                    RMAppAttemptEvent event) {

      appAttempt.registerClientToken();
      //运行AppAttempt
      appAttempt.launchAttempt();
    }
  }
private void launchAttempt(){
    launchAMStartTime = System.currentTimeMillis();
    // 发送 AMLauncherEventType.LAUNCH 事件启动 AM Container
    eventHandler.handle(new AMLauncherEvent(AMLauncherEventType.LAUNCH, this));
  }

3.1.3 AM启动

AM的启动主要靠ApplicationMasterLauncher来完成，ApplicationMasterLauncher继承自AbstractService，维护了一个线程池，主要处理AMLaunchEvent型的事件，分别是请求启动AM的"LAUNCH"事件和请求清理AM的"CLEANUP"事件。

ApplicationMasterLauncher的基本属性

//org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher
  //创建线程池，每一个AM都启动一个线程
  private ThreadPoolExecutor launcherPool;
  //创建独立线程，处理AM的LAUNCH,EVENT事件
  private LauncherThread launcherHandlingThread;
  
  //事件接收和等待队列
  private final BlockingQueue<Runnable> masterEvents
    = new LinkedBlockingQueue<Runnable>();
  
  //创建资源管理器上下文
  protected final RMContext context;
  
  //构造器
  public ApplicationMasterLauncher(RMContext context) {
    super(ApplicationMasterLauncher.class.getName());
    this.context = context;
    //新建线程，用于处理事件
    this.launcherHandlingThread = new LauncherThread();
  }

//初始化Init
@Override
  protected void serviceInit(Configuration conf) throws Exception {
    int threadCount = conf.getInt(
        YarnConfiguration.RM_AMLAUNCHER_THREAD_COUNT,
        YarnConfiguration.DEFAULT_RM_AMLAUNCHER_THREAD_COUNT);
    ThreadFactory tf = new ThreadFactoryBuilder()
        .setNameFormat("ApplicationMasterLauncher #%d")
        .build();
        //初始化线程池
    launcherPool = new ThreadPoolExecutor(threadCount, threadCount, 1,
        TimeUnit.HOURS, new LinkedBlockingQueue<Runnable>());
    launcherPool.setThreadFactory(tf);

  }

处理LAUNCH，CLEANUP事件

@Override
  public synchronized void  handle(AMLauncherEvent appEvent) {
    //获取AMLauncherEvent类型
    AMLauncherEventType event = appEvent.getType();
    RMAppAttempt application = appEvent.getAppAttempt();
    switch (event) {
    //处理LAUNCH事件
    case LAUNCH:
      launch(application);
      break;
    //处理CLEANUP事件
    case CLEANUP:
      cleanup(application);
      break;
    default:
      break;
    }
  }

处理AppAttempt发送来AMLauncherEventType.LAUNCH 事件，以启动AM Container

private void launch(RMAppAttempt application) {
    //创建AMLauncher实例
    Runnable launcher = createRunnableLauncher(application, 
        AMLauncherEventType.LAUNCH);
    //将事件添加到masterEvents队列
    masterEvents.add(launcher);
  }

//构造一个线程LauncherThread
private class LauncherThread extends Thread {
    
    public LauncherThread() {
      super("ApplicationMaster Launcher");
    }

    @Override
    public void run() {
      while (!this.isInterrupted()) {//死循环不停处理请求
        Runnable toLaunch;
        try {
          //从masterEvents队列中取出事件
          toLaunch = masterEvents.take();
          //从线程池中取出一个线程用于执行事件
          launcherPool.execute(toLaunch);
        } catch (InterruptedException e) {
          LOG.warn(this.getClass().getName() + " interrupted. Returning.");
          return;
        }
      }
    }
  }   

从masterEvents队列中取出事件后，交给AMLauncher来处理

//org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher
  @SuppressWarnings("unchecked")
  public void run() {
    switch (eventType) {
    case LAUNCH:   //LAUNCH事件
      try {
        LOG.info("Launching master" + application.getAppAttemptId());
        //启动launch()方法
        launch();
        //发送RMAppAttemptEventType.LAUNCHED事件
        handler.handle(new RMAppAttemptEvent(application.getAppAttemptId(),
            RMAppAttemptEventType.LAUNCHED, System.currentTimeMillis()));
      } catch(Exception ie) {
        onAMLaunchFailed(masterContainer.getId(), ie);
      }
      break;
    }
  }

launch()方法调用RPC函数，来启动AM Container。在NodeManager中，Container启动。至此，AM启动完成。

    //创建Container的请求信息
    StartContainerRequest scRequest =
        StartContainerRequest.newInstance(launchContext,
          masterContainer.getContainerToken());
    List<StartContainerRequest> list = new ArrayList<StartContainerRequest>();
    list.add(scRequest);
    StartContainersRequest allRequests =
        StartContainersRequest.newInstance(list);

    //调用RPC函数启动Container
    StartContainersResponse response =
        containerMgrProxy.startContainers(allRequests);
    if (response.getFailedRequests() != null
        && response.getFailedRequests().containsKey(masterContainerID)) {
      Throwable t =
          response.getFailedRequests().get(masterContainerID).deSerialize();
      parseAndThrowException(t);
    } else {
      LOG.info("Done launching container " + masterContainer + " for AM "
          + application.getAppAttemptId());
    }

3.2 AM注册和心跳

AM启动后，向RM申请注册和心跳，并申请Container资源。

AM注册。更新AM在AMLivelinessMonitor中的最新事件，发送RMAppAttemptEventType.REGISTERED事件，并将AppAttempt的状态从LAUNCHED转化为RUNNING。

//org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.java
// 一次只允许一个线程注册
    synchronized (lock) {
      ...
      
      //更新最近汇报的事件
      this.amLivelinessMonitor.receivedPing(applicationAttemptId);

      // 将响应id设置为0，以标识application master是否注册了相应的attemptid
      lastResponse.setResponseId(0);
      lock.setAllocateResponse(lastResponse);

      RegisterApplicationMasterResponse response =
          recordFactory.newRecordInstance(
              RegisterApplicationMasterResponse.class);
      this.amsProcessingChain.registerApplicationMaster(
          amrmTokenIdentifier.getApplicationAttemptId(), request, response);
      return response;
    }

心跳流程，利用heartbeatThread 线程处理AM

//org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.java
  //初始化AM心跳对象
  private final HeartbeatThread heartbeatThread;
  //构造器
  @Private
  @VisibleForTesting
  @Deprecated
  public AMRMClientAsyncImpl(AMRMClient<T> client, int intervalMs,
      CallbackHandler callbackHandler) {
    super(client, intervalMs, callbackHandler);
    //初始化AM心跳线程实例
    heartbeatThread = new HeartbeatThread();
    handlerThread = new CallbackHandlerThread();
    //响应队列
    responseQueue = new LinkedBlockingQueue<Object>();
    keepRunning = true;
  }

AM 向 RM 注册后，周期性地通过 RPC 函数 ApplicationMasterProtocol的allocate() 函数与 RM 通信，用来请求申请，获取新分配的资源，形成周期性心跳，告诉 RM 自己还活着。

  private class HeartbeatThread extends Thread {
    //心跳线程
    public HeartbeatThread() {
      super("AMRM Heartbeater thread");
    }
    
    public void run() {
      while (true) {
        Object response = null;
        // 同步确保我们在注销后不会发送心跳
        synchronized (unregisterHeartbeatLock) {
          if (!keepRunning) {
            return;
          }

          try {
          //心跳线程，就是周期性地调用allocate() 函数
            response = client.allocate(progress);
          } catch (ApplicationAttemptNotFoundException e) {
            handler.onShutdownRequest();
            LOG.info("Shutdown requested. Stopping callback.");
            return;
          } catch (Throwable ex) {
            LOG.error("Exception on heartbeat", ex);
            response = ex;
          }
          if (response != null) {
            while (true) {
              try {
                responseQueue.put(response);
                break;
              } catch (InterruptedException ex) {
                LOG.debug("Interrupted while waiting to put on response queue", ex);
              }
            }
          }
        }
      }

3.3 监控活跃度

AMLivelinessMonitor作为AM的三个主要服务之一，会周期性地遍历所有应用程序的AM状态。若一个AM在一定的时间(默认为10min)内没有汇报心跳信息，则将它上面所有正在运行的Container设置为运行失败，即认为它"死掉了"。若AM运行失败，RM会重新为重新为它申请资源，以便能够重新分配到另一个节点上运行。

AMLivelinessMonitor构造器，需要两个属性Dispatcher d, Clock clock

//org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.AMLivelinessMonitor.java
public AMLivelinessMonitor(Dispatcher d) {
    super("AMLivelinessMonitor");
    this.dispatcher = d.getEventHandler();
  }

  public AMLivelinessMonitor(Dispatcher d, Clock clock) {
    super("AMLivelinessMonitor", clock);
    this.dispatcher = d.getEventHandler();
  }

初始化方法

public void serviceInit(Configuration conf) throws Exception {
    super.serviceInit(conf);
    //设置过期时间间隔:10min
    int expireIntvl = conf.getInt(YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS,
            YarnConfiguration.DEFAULT_RM_AM_EXPIRY_INTERVAL_MS);
    setExpireInterval(expireIntvl);
    //设置监控间隔，为过期时间间隔的1/3
    setMonitorInterval(expireIntvl/3);
  }

超时回调，使用dispatcher进行处理。

  protected void expire(ApplicationAttemptId id) {
    dispatcher.handle(
        new RMAppAttemptEvent(id, RMAppAttemptEventType.EXPIRE));
  }

3.4 注销AM

当AM运行结束后，需要通过RPC函数ApplicationMasterProtocol #finishApplicationMaster告诉RM自己运行结束，可以回收资源和清理各种数据结果了。

@Override
  public FinishApplicationMasterResponse finishApplicationMaster(
      FinishApplicationMasterRequest request) throws YarnException,
      IOException {

    // 获取 applicationAttemptId
    ApplicationAttemptId applicationAttemptId =
        YarnServerSecurityUtils.authorizeRequest().getApplicationAttemptId();

    // 获取 ApplicationId
    ApplicationId appId = applicationAttemptId.getApplicationId();

    // 获取RMApp
    RMApp rmApp =
        rmContext.getRMApps().get(applicationAttemptId.getApplicationId());

    // 当app完成后删除所有collector地址
    if (timelineServiceV2Enabled) {
      ((RMAppImpl) rmApp).removeCollectorData();
    }
    // 首先检查app是否存在于RMStateStore中，以确保在RM工作重新启动之前和之后不引发ApplicationDoesNotExistInCacheException
    if (rmApp.isAppFinalStateStored()) {
      LOG.info(rmApp.getApplicationId() + " unregistered successfully. ");
      return FinishApplicationMasterResponse.newInstance(true);
    }

    AllocateResponseLock lock = responseMap.get(applicationAttemptId);
    if (lock == null) {
      throwApplicationDoesNotExistInCacheException(applicationAttemptId);
    }

    // 一次只允许AM中的一个线程执行finishApp
    synchronized (lock) {
      if (!hasApplicationMasterRegistered(applicationAttemptId)) {
        String message =
            "Application Master is trying to unregister before registering for: "
                + appId;
        LOG.error(message);
        RMAuditLogger.logFailure(
            this.rmContext.getRMApps()
                .get(appId).getUser(),
            AuditConstants.UNREGISTER_AM, "", "ApplicationMasterService",
            message, appId,
            applicationAttemptId);
        throw new ApplicationMasterNotRegisteredException(message);
      }

      FinishApplicationMasterResponse response =
          FinishApplicationMasterResponse.newInstance(false);

      // finishedAttemptCache 是否存在applicationAttemptId
      if (finishedAttemptCache.putIfAbsent(applicationAttemptId, true)
          == null) {
        // 没有处理过,直接处理
        this.amsProcessingChain
            .finishApplicationMaster(applicationAttemptId, request, response);
      }
      // 处理监控心跳
      this.amLivelinessMonitor.receivedPing(applicationAttemptId);
      return response;
    }
  }

四、结语

作为yarn三大核心组件之一，ApplicationMaster与ResourceManager以及NodeManager共同实现资源管理和任务调度，为集群在利用率、资源统一管理和数据共享等方面带来了巨大好处。