Yarn中MapReduce任务提交步骤

Submit ApplicationStage

  1. job.waitForCompletion 
  1. Job.submit()
    1. final JobSubmitter submitter =getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
    2. submitter.submitJobInternal(Job.this, cluster);
  1. JobSubmitter.submitJobInternal() 在本地设置好job的一些参数,取到job token等信息,通过inputformat计算输入的split(map)信息
  1. ClientProtocolsubmitJob RPC方法进入到YARNRunner
  2. Construct necessary information to start the MR AM
  3. Submit to ResourceManager
  4. Get the ApplicationId of the submitted application.
  5. YarnClientImplappConect取一个ApplicationId,封装为一个SubmitApplicationRequestrmClient进行提交,并等待提交后该application状态。
  6. ApplicationClientProtocol rmClient.submitApplication(request);

Yarn中MapReduce任务提交步骤_第1张图片

RM中协议实现

进入yarn.server.resourcemanager包  ClientRMService

  1. 安全用户校验
  2. ApplicationId校验
  3. 取出request中的queue,ApplicationName,ApplicationType等信息,由rmAppManager -> submit

RMAppManager中根据请求信息创建RMAppImpl实例

  1. SubmitApplicationResponse response = recordFactory.newRecordInstance(SubmitApplicationResponse.class);

你可能感兴趣的:(mapreduce,hadoop,yarn)