YARN源码分析(二)-----ResourceManager中的NM节点管理

前言

继上一篇文章讲述完ApplicationMaster的相关用法,核心主题都是围绕着2个字"应用",当然在RM中还有另外一项比较重要的服务也很重要,他就是节点管理服务,在RM中是如何维系管理多个节点,对于应用管理的话,在RM中已经有了ApplicationMasterService这个服务对象了,那么对应于节点NodeManager来说,难道叫做NodeManagerService吗,听起来非常顺,其实他叫做?ResourceTrackerService,当然名称叫什么都无所谓啦,他扮演的功能就是类似于节点NodeManager大管家的角色了.OK,在这里我们就以NodeManager管理为核心线索,逐步分析RM在此方面的设计思想.

相关涉及类

在分析之前,还是需要了解一下相关类,在阅读本篇文章之前,可以建议大家阅读我的上一篇文章ApplicationMaster文章的分析,因为NM和AM管理许多思想共同,也有共同的父类,比如AbstractService这样的抽象服务类.下面是我归纳出的几个类.

1.NodeManager.java--节点管理类,这个类是yarn-resourcemanager包中的类,不是yarn-nodemanager中的同名类,这个类是本篇文章的核心角色类,

2.NodesListManager--节点列表管理类，这个类中管理了类似黑名单，白名单的节点列表形式。

3.NMLivelinessMonitor--节点存活状态监控线程类，与之前的AMLivelinessMonitor线程的原理类似，最简单的心跳更新检查。

4.ResourceTrackerService--节点服务管理对象，负责与各个NodeManager通信。包括NM在此服务上的注册请求处理，心跳更新操作等等。

下面是一张结构简图帮助大家宏观上理解RM中的NM管理：

NodeManager节点注册

我们从一个比较初始的状态出发，比如说节点注册开始，一步步的贯穿的去分析整个流程。节点注册操作，在NodeManager类自身中。这个类中定义的基本信息如下

 //ResourceManager下资源管理器类
 public class NodeManager implements ContainerManagementProtocol {
   private static final Log LOG = LogFactory.getLog(NodeManager.class);
   private static final RecordFactory recordFactory = RecordFactoryProvider.getRecordFactory(null);

   final private String containerManagerAddress;
   //节点通信地址
   final private String nodeHttpAddress;
   //所在机架名称
   final private String rackName;
   //节点ID
   final private NodeId nodeId;
   final private Resource capability;
   Resource available = recordFactory.newRecordInstance(Resource.class);
   Resource used = recordFactory.newRecordInstance(Resource.class);

注册操作并没有独立出方法来，而是包含在了构造函数中，也就是说，当你构造新的NodeManager的时候，你已经在注册节点到ResourceTrackerService。

 public NodeManager(String hostName, int containerManagerPort, int httpPort,
       String rackName, Resource capability,
       ResourceTrackerService resourceTrackerService, RMContext rmContext)
       throws IOException, YarnException {
     this.containerManagerAddress = hostName + ":" + containerManagerPort;
     this.nodeHttpAddress = hostName + ":" + httpPort;
     this.rackName = rackName;
     this.resourceTrackerService = resourceTrackerService;
     this.capability = capability;
     Resources.addTo(available, capability);

     this.nodeId = NodeId.newInstance(hostName, containerManagerPort);
     //新建nodemanager注册请求
     RegisterNodeManagerRequest request = recordFactory
         .newRecordInstance(RegisterNodeManagerRequest.class);
     //往请求内写入状态信息
     request.setHttpPort(httpPort);
     request.setNodeId(this.nodeId);
     request.setResource(capability);
     request.setNodeId(this.nodeId);
     //调用resourceTrackerService服务对象进行节点注册操作
     resourceTrackerService.registerNodeManager(request);
     this.schedulerNode = new FiCaSchedulerNode(rmContext.getRMNodes().get(
         this.nodeId), false);
    .....
   }

顺着这行代码，来看一下服务端处理注册请求的方法。服务端的类对象是ResourceTrackerService。

 //节点资源跟踪服务,与各个节点的NodeManager通信服务
 public class ResourceTrackerService extends AbstractService implements
     ResourceTracker {

   private static final Log LOG = LogFactory.getLog(ResourceTrackerService.class);

   private static final RecordFactory recordFactory =
     RecordFactoryProvider.getRecordFactory(null);
   //资源管理器上下文
   private final RMContext rmContext;
   //节点列表管理器
   private final NodesListManager nodesListManager;
   //节点存活状态监控
   private final NMLivelinessMonitor nmLivelinessMonitor;
   //节点安全认证相关
   private final RMContainerTokenSecretManager containerTokenSecretManager;
   private final NMTokenSecretManagerInRM nmTokenSecretManager;

   //心跳间隔
   private long nextHeartBeatInterval;
   //远程RPC服务
   private Server server;
   private InetSocketAddress resourceTrackerAddress;

   private static final NodeHeartbeatResponse resync = recordFactory
       .newRecordInstance(NodeHeartbeatResponse.class);
   private static final NodeHeartbeatResponse shutDown = recordFactory
   .newRecordInstance(NodeHeartbeatResponse.class);

   //最小分配的内存的大小
   private int minAllocMb;
   //最小分配的核数大小
   private int minAllocVcores;

也是继承了抽象服务类，这里面包含的内容就多了许多，重点关注，节点列表管理器对象NodesListManager和NMLivelinessMonitor，这2者与本文叙述所相关。然后跳到节点注册请求处理操作。在节点注册请求进来的时候，首先会做一些请求的过滤条件的验证，过滤不符合要求的节点。

 //响应NodeManager的节点注册请求方法
   @SuppressWarnings("unchecked")
   @Override
   public RegisterNodeManagerResponse registerNodeManager(
       RegisterNodeManagerRequest request) throws YarnException,
       IOException {

     NodeId nodeId = request.getNodeId();
     String host = nodeId.getHost();
     .....

     // Check if this node is a 'valid' node
     //如果此节点是在exclude名单中,注册请求将会被拒绝,调用的是节点列表管理器的isValidNode方法
     if (!this.nodesListManager.isValidNode(host)) {
       String message =
           "Disallowed NodeManager from  " + host
               + ", Sending SHUTDOWN signal to the NodeManager.";
       LOG.info(message);
       response.setDiagnosticsMessage(message);
       response.setNodeAction(NodeAction.SHUTDOWN);
       return response;
     }

     // Check if this node has minimum allocations
     //判断节点资源是否满足最小内存和核数的限制,如果没有同样拒绝注册
     if (capability.getMemory() < minAllocMb
         || capability.getVirtualCores() < minAllocVcores) {
       String message =
           "NodeManager from  " + host
               + " doesn't satisfy minimum allocations, Sending SHUTDOWN"
               + " signal to the NodeManager.";
       LOG.info(message);
       response.setDiagnosticsMessage(message);
       response.setNodeAction(NodeAction.SHUTDOWN);
       return response;
     }
     .....

2个条件，节点是否有效，有效的规则就是在NodeListManager中定义的，这个后面会提到，第二个是节点所剩资源是否足够启动NodeManager。如果这2个请求都过了的话，则表明可以进行注册，此节点将会被注册进行存活监控线程中。

 .....
     // On every node manager register we will be clearing NMToken keys if
     // present for any running application.
     this.nmTokenSecretManager.removeNodeKey(nodeId);
     //同时将节点注册到节点存活监控线程中
     this.nmLivelinessMonitor.register(nodeId);

     String message =
         "NodeManager from node " + host + "(cmPort: " + cmPort + " httpPort: "
             + httpPort + ") " + "registered with capability: " + capability
             + ", assigned nodeId " + nodeId;
     LOG.info(message);
     response.setNodeAction(NodeAction.NORMAL);
     response.setRMIdentifier(ResourceManager.clusterTimeStamp);
     return response;
   }

注册操作在上篇文章都详细讲述过了，在基础监控类中声明了，如下：

 //进程存活状态监控类
 public abstract class AbstractLivelinessMonitor<O> extends AbstractService {
   ......

   private final Clock clock;

   //保存了心跳检验的结果记录
   private Map<O, Long> running = new HashMap<O, Long>();

   //更新心跳监控检测最新时间
   public synchronized void receivedPing(O ob) {
     //only put for the registered objects
     if (running.containsKey(ob)) {
       running.put(ob, clock.getTime());
     }
   }

具体细节请求点击 YARN源码分析(一) 。

OK，回到之前没有说清楚的NodeListManager节点列表管理器类，这个类提供了节点有效性检查的方法

 .....

     // Check if this node is a 'valid' node
     //如果此节点是在exclude名单中,注册请求将会被拒绝,调用的是节点列表管理器的isValidNode方法
     if (!this.nodesListManager.isValidNode(host)) {
       String message =
           "Disallowed NodeManager from  " + host
               + ", Sending SHUTDOWN signal to the NodeManager.";
       LOG.info(message);
       response.setDiagnosticsMessage(message);
       response.setNodeAction(NodeAction.SHUTDOWN);
       return response;
     }
     .....

传入的是主机名，可以联系之前Decommision文章中提到的include，exclude名单列表的内容。

 //节点列表管理器,主要是根据include白名单和exclude黑名单属性进行判断,也是一个服务
 public class NodesListManager extends AbstractService implements
     EventHandler<NodesListManagerEvent> {

   private static final Log LOG = LogFactory.getLog(NodesListManager.class);
   //节点列表读取器
   private HostsFileReader hostsReader;
   private Configuration conf;
   //不允许使用的节点列表
   private Set<RMNode> unusableRMNodesConcurrentSet = Collections
       .newSetFromMap(new ConcurrentHashMap<RMNode,Boolean>());
   //资源管理上下文
   private final RMContext rmContext;
   .....

在这个类中写明了unsableNodes无法使用的节点列表名单，但是有效性检查的方法并没有使用到此变量。下面是真正的valid检测方法

 //输入主机名,判断是否是有效的节点,
   public boolean isValidNode(String hostName) {
     synchronized (hostsReader) {
       //获取可接入和不可接入主机名列表
       Set<String> hostsList = hostsReader.getHosts();
       Set<String> excludeList = hostsReader.getExcludedHosts();
       String ip = NetUtils.normalizeHostName(hostName);
       //判断是否在相应的列表中以此判断节点是否有效
       return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
           .contains(ip))
           && !(excludeList.contains(hostName) || excludeList.contains(ip));
     }
   }

也是通过hostReader对象读取配置文件中的include，exclude主机名列表做判断。初始主机从这里读取出来

 @Override
   protected void serviceInit(Configuration conf) throws Exception {

     this.conf = conf;

     // Read the hosts/exclude files to restrict access to the RM
     //在服务初始化的时候读取include和exclude文件信息,exclude的节点列表名单将会被RM拒绝接入
     try {
       this.hostsReader =
         new HostsFileReader(
             conf.get(YarnConfiguration.RM_NODES_INCLUDE_FILE_PATH,
                 YarnConfiguration.DEFAULT_RM_NODES_INCLUDE_FILE_PATH),
             conf.get(YarnConfiguration.RM_NODES_EXCLUDE_FILE_PATH,
                 YarnConfiguration.DEFAULT_RM_NODES_EXCLUDE_FILE_PATH)
                 );
       //输出节点信息
       printConfiguredHosts();
       .....

OK，节点注册操作分析完毕。

节点HeartBeat心跳

心跳方法在NodeManager中有直接定义

 public class NodeManager implements ContainerManagementProtocol {
    ....
   //周期心跳方法
   public void heartbeat() throws IOException, YarnException {
     NodeStatus nodeStatus =
       org.apache.hadoop.yarn.server.resourcemanager.NodeManager.createNodeStatus(
           nodeId, getContainerStatuses(containers));
     nodeStatus.setResponseId(responseID);
     NodeHeartbeatRequest request = recordFactory
         .newRecordInstance(NodeHeartbeatRequest.class);
     request.setNodeStatus(nodeStatus);
     //调用resourceTrackerService发送心跳包，并获取响应回复
     NodeHeartbeatResponse response = resourceTrackerService
         .nodeHeartbeat(request);
     responseID = response.getResponseId();
   }

也是远程调用ResourceTrackerService方法

 //节点心跳相应方法
   @SuppressWarnings("unchecked")
   @Override
   public NodeHeartbeatResponse nodeHeartbeat(NodeHeartbeatRequest request)
       throws YarnException, IOException {

     //从心跳中获取远程节点状态信息
     NodeStatus remoteNodeStatus = request.getNodeStatus();
     /**
      * Here is the node heartbeat sequence...
      * 1. Check if it's a registered node
      * 2. Check if it's a valid (i.e. not excluded) node
      * 3. Check if it's a 'fresh' heartbeat i.e. not duplicate heartbeat
      * 4. Send healthStatus to RMNode
      */

     NodeId nodeId = remoteNodeStatus.getNodeId();

     // 1. Check if it's a registered node
     RMNode rmNode = this.rmContext.getRMNodes().get(nodeId);
     if (rmNode == null) {
       /* node does not exist */
       String message = "Node not found resyncing " + remoteNodeStatus.getNodeId();
       LOG.info(message);
       resync.setDiagnosticsMessage(message);
       return resync;
     }

     // Send ping
     //更新心跳响应最新时间
     this.nmLivelinessMonitor.receivedPing(nodeId);

     // 2. Check if it's a valid (i.e. not excluded) node
     //每次心跳检测都会检查节点是否被拉入exclude名单
     if (!this.nodesListManager.isValidNode(rmNode.getHostName())) {
       String message =
           "Disallowed NodeManager nodeId: " + nodeId + " hostname: "
               + rmNode.getNodeAddress();
       LOG.info(message);
       shutDown.setDiagnosticsMessage(message);
       //如果是被拉入,则触发节点撤销事件
       this.rmContext.getDispatcher().getEventHandler().handle(
           new RMNodeEvent(nodeId, RMNodeEventType.DECOMMISSION));
       return shutDown;
     }

     .....

     // Heartbeat response
     //设置心跳回复
     NodeHeartbeatResponse nodeHeartBeatResponse = YarnServerBuilderUtils
         .newNodeHeartbeatResponse(lastNodeHeartbeatResponse.
             getResponseId() + 1, NodeAction.NORMAL, null, null, null, null,
             nextHeartBeatInterval);
     rmNode.updateNodeHeartbeatResponseForCleanup(nodeHeartBeatResponse);

心跳方法也不是特别的复杂。