ResourceManager High Availability

Apache 官方原文地址：http://hadoop.apache.org/docs/r2.5.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

一. 简介

这篇指南提供关于 YARN’s ResourceManager 的 High Availability 的概述，并详细介绍了如何配置和使用该特性。ResourceManager (RM) 负责追踪集群资源的使用情况，并且调度应用程序（例如，MapReduce 任务）。在 Hadoop 2.4 之前，ResourceManager 在 YARN 集群中是单点故障。 High Availability 特性以一对 Active/Standby ResourceManager 的形式增加 redundancy 以消除单点故障。

二. 架构

2.1 RM 切换

ResourceManager HA 是通过 Active/Standby 架构来实现的。在任何时刻，其中一个 RMs 是 Active 状态，其余的一个或多个 RMs 是 Standby 模式在 Active 节点发生不幸时随时切换。转换到 active 的触发点那么来自管理员（通过 CLI）或来内置的自动故障切换（这个需保证 automatic-failover 是开启的）

2.1.1 手工故障切换

当 automatic-failover 没有启动时，管理员必须手动将其中一个 RMs 切换到 Active 状态。当将一个 RM 切换到其他状态，首先要切换的是将 Acive 的RM 状态切换到 Standby 的状态和将 Standby 的RM 切换到 Active 状态。所有这些操作可以使用 “yarn rmadmin” CLI。

2.2.2 自动故障切换

有个嵌入到 RMs 中的选项是基于 ZooKeeper 的 ActiveStandbyElector ，它将决定哪个 RM 应该是 Active 状态。当 Active 节点失效或者无法响应了，另一个 RM 就自动被选举为 Active 并接管工作。注意到，没有必要像在 HDFS 中那样需要单独运行 ZKFC 的进程，因为内嵌在 RMs 中的 ActiveStandbyElector 扮演着故障检测器和 leader 选举器的角色，而并非是单独的 ZKFC 进程。

2.2.3 在 RM 故障切换中的客户端，ApplicationMaster 和 NodeManager

当有多个 RMs 存在时， clients 和节点所使用的配置文件 (yarn-site.xml) 应该需要列出所有的 RMs。Clients, ApplicationMasters (AMs) and NodeManagers (NMs) 将以轮询调度方法（Round-Robin fashion) 尝试连接 RMs直到它们连接到 Active RM。如果 Active 失效了，它们将继续采用轮询调度的方式直到联系到新的 “Active”。这个默认重试的逻辑是 org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider 的实现。你也可以通过实现 org.apache.hadoop.yarn.client.RMFailoverProxyProvider 来覆写逻辑。

2.2 恢复之前 Active-RM 的状态

伴随着 ResourceManger Restart 生效， 被选举未新的 active 状态的 RM 将装载内部状态并尽可能继续从之前故障失效的地方操作，这取决于 RM 的 restart 特性。新的尝试被用于每个先前提交到 RM 的所管理的应用。应用可以周期性地 checkpoint 以避免丢失任何任务。状态-存储 都必须对 Active/Standby RMs 可见。目前，有 persistence - FileSystemRMStateStore 和 ZKRMStateStore 这两种的方式实现 RMStateStore 。ZKRMStateStore 在任何时刻毫无疑问只允许向一个 RM 写操作，也因此被推荐用于 HA 集群的存储。当使用 ZKRMStateStore 时就没有独立地隔离机制来解决潜在的多个 RMs 都可以承担 Active 角色所引发的 ” split-brain” 情况。

三. 部署

3.1 配置

大部分的故障切换功能都可以在使用不同的配置属性调整的。以下是一系列 必须/重要 的配置属性。yarn-default.xml carries a full-list of knobs。查看 yarn-default.xml 更多包括默认值在内的信息。查看 the document for ResourceManger Restart 也可以作为启动 state-store 的启动指南。

配置属性	描述
yarn.resourcemanager.zk-address	Address of the ZK-quorum. Used both for the state-store and embedded leader-election
yarn.resourcemanager.ha.enabled	Enable RM HA
yarn.resourcemanager.ha.rm-ids	List of logical IDs for the RMs. e.g., “rm1,rm2”
yarn.resourcemanager.hostname.rm-id	For each rm-id, specify the hostname the RM corresponds to. Alternately, one could set each of the RM’s service addresses
yarn.resourcemanager.ha.id	Identifies the RM in the ensemble. This is optional; however, if set, admins have to ensure that all the RMs have their own IDs in the config
yarn.resourcemanager.ha.automatic-failover.enabled	Enable automatic failover; By default, it is enabled only when HA is enabled
yarn.resourcemanager.ha.automatic-failover.embedded	Use embedded leader-elector to pick the Active RM, when automatic failover is enabled. By default, it is enabled only when HA is enabled
yarn.resourcemanager.cluster-id	Identifies the cluster. Used by the elector to ensure an RM doesn’t take over as Active for another cluster
yarn.client.failover-proxy-provider	The class to be used by Clients, AMs and NMs to failover to the Active RM
yarn.client.failover-max-attempts	The max number of times FailoverProxyProvider should attempt failover
yarn.client.failover-sleep-base-ms	The sleep base (in milliseconds) to be used for calculating the exponential delay between failovers
yarn.client.failover-sleep-max-ms	The maximum sleep time (in milliseconds) between failovers
yarn.client.failover-retries	The number of retries per attempt to connect to a ResourceManager
yarn.client.failover-retries-on-socket-timeouts	The number of retries per attempt to connect to a ResourceManager on socket timeouts

3.1.1 配置样例

以下是一个启动 RM 故障切换所需最小样例

<code class="hljs xml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.resourcemanager.ha.enabled<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>true<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.resourcemanager.cluster-id<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>cluster1<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.resourcemanager.ha.rm-ids<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>rm1,rm2<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.resourcemanager.hostname.rm1<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>master1<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.resourcemanager.hostname.rm2<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>master2<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>yarn.resourcemanager.zk-address<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">name</span>></span>
   <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>zk1:2181,zk2:2181,zk3:2181<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">value</span>></span>
 <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">property</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

3.2 管理命令

yarn rmadmin 有一些关于检查 RM 健康/状态、切换到 Active/Standby 状态的 HA-specific 命令。HA 命令将由 yarn.resourcemanager.ha.rm-ids 所设置的 RM 的服务 id 作为参数。

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"> <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yarn rmadmin -getServiceState rm1
 active

 <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yarn rmadmin -getServiceState rm2
 standby</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>

如果自动故障切换生效，你不能手工故障切换命令。

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$ yarn rmadmin -transitionToStandby rm1
 Automatic failover is enabled <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> org.apache.hadoop.yarn.client.RMHAServiceTarget@<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>d8299fd
 Refusing <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">to</span> manually manage HA state, since <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">it</span> may cause
 <span class="hljs-operator" style="box-sizing: border-box;">a</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">split</span>-brain scenario <span class="hljs-operator" style="box-sizing: border-box;">or</span> other incorrect state.
 If you are very sure you know what you are doing, please
 specify <span class="hljs-operator" style="box-sizing: border-box;">the</span> forcemanual flag.</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

重点在于这句： it may cause a split-brain scenario or other incorrect state。 
查看 YarnCommands 中更多相关信息。

3.3 ResourceManager 网页界面服务

假设一个 standby RM 已经启动并且正在运行中，那么 Standby 将所有的 web 请求自动重定向到 Active，除了 “About” 界面。

3.4 网页服务

假设一个 standby RM 已经启动并且正在运行中，当援引一个 standby RM ，在 ResourceManager REST APIs 中所描述RM 网页服务将自动重定向到 Active RM。

http://blog.csdn.net/u011414200/article/details/50281029