HBase中的RIT机制(Region in Transcation)

每天到公司第一件事，就是检查测试环境和线上环境的几个hbase集群运行状态。由于测试环境用的几台机器都是虚拟机，因此不可避免的很容易出现regionserver宕机的情况。每次对于宕机的regionserver重启后的一段时间内，部门里的其他一些需要调用hbase的开发人员就开始抱怨，hbase 客户端连接错误 超时等等各种抱怨，其实也是因为regionserver 启动后，本身会进行一次 RIT操作，这时候 client 服务会暂时关闭掉，这里稍微解释下RIT机制，以便我们更好的进行hbase集群的维护工作。

RIT 的全称是region in transcation. 每次hbase master 对region 的一个open 或一个close 操作都会想Master 的RIT中插入一条记录,因为master 对region 的操作要保持原子性,region 的 open 和 close 是通过Hmaster 和 region server 协助来完成的. 所以为了满足这些操作的协调,回滚,和一致性.Hmaster 采用了 RIT 机制并结合Zookeeper 中Node的状态来保证操作的安全和一致性.

   为了弄清楚RIT的状态机制, 我们先来了解下下面的2个enum. 第一个是 RegionState的状态, 它的作用是用于记录Hmaster在操作region时region 所处的状态.

 /**
  * State of a Region while undergoing transitions.
  */
 public static class RegionState implements Writable {
   private HRegionInfo region;

   public enum State {
     OFFLINE,        // region is in an offline state
     PENDING_OPEN,   // sent rpc to server to open but has not begun
     OPENING,        // server has begun to open but not yet done
     OPEN,           // server opened region and updated meta
     PENDING_CLOSE,  // sent rpc to server to close but has not begun
     CLOSING,        // server has begun to close but not yet done
     CLOSED          // server closed region and updated meta
   }

   private State state;
   private long stamp;

第二个 enum 是 Hbase Event Handler types. 此处我们着重关注下面5中状态.这5种状态是在open/close region时设置的.可以用于region server 和 hmaster 的通讯和状态记录,保证hmaster可以及时了解region serve 的动作是否完成并在失败时可以回滚.

 RS_ZK_REGION_CLOSING, RS_ZK_REGION_CLOSED ,RS_ZK_REGION_OPENING ,RS_ZK_REGION_OPENED ,M_ZK_REGION_OFFLINE

/**

  * List of all HBase event handler types.  Event types are named by a
  * convention: event type names specify the component from which the event
  * originated and then where its destined -- e.g. RS2ZK_ prefix means the
  * event came from a regionserver destined for zookeeper -- and then what
  * the even is; e.g. REGION_OPENING.
  *
  * We give the enums indices so we can add types later and keep them
  * grouped together rather than have to add them always to the end as we
  * would have to if we used raw enum ordinals.
  */
 public enum EventType {
   // Messages originating from RS (NOTE: there is NO direct communication from
   // RS to Master). These are a result of RS updates into ZK.
   RS_ZK_REGION_CLOSING      (1),   // RS is in process of closing a region
   RS_ZK_REGION_CLOSED       (2),   // RS has finished closing a region
   RS_ZK_REGION_OPENING      (3),   // RS is in process of opening a region
   RS_ZK_REGION_OPENED       (4),   // RS has finished opening a region

   // Messages originating from Master to RS
   M_RS_OPEN_REGION          (20),  // Master asking RS to open a region
   M_RS_OPEN_ROOT            (21),  // Master asking RS to open root
   M_RS_OPEN_META            (22),  // Master asking RS to open meta
   M_RS_CLOSE_REGION         (23),  // Master asking RS to close a region
   M_RS_CLOSE_ROOT           (24),  // Master asking RS to close root
   M_RS_CLOSE_META           (25),  // Master asking RS to close meta

   // Messages originating from Client to Master
   C_M_DELETE_TABLE          (40),   // Client asking Master to delete a table
   C_M_DISABLE_TABLE         (41),   // Client asking Master to disable a table
   C_M_ENABLE_TABLE          (42),   // Client asking Master to enable a table
   C_M_MODIFY_TABLE          (43),   // Client asking Master to modify a table
   C_M_ADD_FAMILY            (44),   // Client asking Master to add family to table
   C_M_DELETE_FAMILY         (45),   // Client asking Master to delete family of table
   C_M_MODIFY_FAMILY         (46),   // Client asking Master to modify family of table

   // Updates from master to ZK. This is done by the master and there is
   // nothing to process by either Master or RS
   M_ZK_REGION_OFFLINE       (50),  // Master adds this region as offline in ZK

   // Master controlled events to be executed on the master
   M_SERVER_SHUTDOWN         (70),  // Master is processing shutdown of a RS
   M_META_SERVER_SHUTDOWN    (72);  // Master is processing shutdown of RS hosting a meta region (-ROOT- or .META.).

   /**
    * Constructor
    */
   EventType(int value) {}
 }

    Hmaster中的AssignmentManager是主要来管理region 上线下线操作的。我们可以暂时理解为Assign就是将region 上线，Unassign就是将region下线。

    Assign region 时会先将regionstate 置为OFFLINE,等getregionplan结束后取得了region on line 的server 。 后将其置为PENDING_OPEN并将reionstatus加入到RIT之中.随后调用RPC让regionserver 中OpenRegionHandler来在region server端处理open region的动作，其中包括将zk中node的状态置为RS_ZK_REGION_OPENING  然后对hregion 进行初始话，最后将zk中 node 的状态置为RS_ZK_REGION_OPENED。 后续master端会通过zk 的watch机制发现node的状态发生变化。详见图三中左半部分流程。

同样Unassign region 时会先将regionstate 置为PENDING_CLOSE状态并将reionstatus加入到RIT之中，在完成一系列检查之后会调用RPC让regionserver 中CloseRegionHandler来在region server端处理open region的动作，其中包括将zk中node的状态置为RS_ZK_REGION_CLOSEING  然后对hregion 进行CLOSE，最后将zk中 node 的状态置为RS_ZK_REGION_CLOSEED。 后续master端会通过zk 的watch机制发现node的状态发生变化。详见图三中左半部分流程。

在图三所示中是hmaster在对RIT的处理。如果是第一次启动，会监视ZK的变化，发现ZK变化后根据ZK中node的状态调用hmaster中的hander(ClosedRegionHandler,OpeneRegionHandler)来处理，如果node 状态为RS_ZK_REGION_CLOSED ,RS_ZK_REGION_OPENED,

表示region server动作已经完成，master 完成相应动作后会其对于的RIT中的regionstate删除。这样就完成了RIT中regionstate的一个生命过程。 同样如果是FAILOVER中做recover的master会对ZK中的node状态进行扫描，然后处理对应的RIT。

RIT中保存的长时间没有处理的regionstate会被chore线程处理掉，重新加入assign和unassign队列之中。