HBase Region在两个RegionServer出现分析

文章转自：http://punishzhou.iteye.com/blog/1264874

HBase的操作一般都是以region为粒度的，如split，compact，move等操作。因此对于每个region在集群的唯一性就需要得到保证。若region在两个regionserver中出现显然会有各种各样的bug出现。 

这里我们通过以下的分析来看看region在上吗情况下会出现分配到两个rs中。 


对于一个RS1上的Region A将其move到RS2中，调用HBaseAdmin的move操作： 

  1.创建一个到B的RegionPlan 

  2.然后unassigned Region A，即将RS1上的Region A的region关闭，主要是关闭region上每个storefile的读数据流，在这个过程中会把memstore内容刷一次磁盘 

  3.然后在RS2中open Region A，主要操作是对Region A的初始化，将A的rs地址信息更新Meta表中 


假设在move的过程中RS1 down掉了，那么master会调用servershutdownhandler来处理该事件。主要处理步骤如下：

 public void process() throws IOException {
     final String serverName = this.hsi.getServerName();

 
拆分日志，把regionserver的hlog按照region的不同进行拆分，分别写入各个region目录下 

 LOG.info("Splitting logs for " + serverName);
     this.services.getMasterFileSystem().splitLog(serverName);

 

 在master的内存中清除rs中的region和server，并返回在master的RIT队列上的该RS的online region

 

 // Clean out anything in regions in transition.  Being conservative and
    // doing after log splitting.  Could do some states before -- OPENING?
    // OFFLINE? -- and then others after like CLOSING that depend on log
    // splitting.
    List regionsInTransition =
      this.services.getAssignmentManager().processServerShutdown(this.hsi);

 [如该RS包含ROOT或META则需首先分配之 

 // Assign root and meta if we were carrying them.
    if (isCarryingRoot()) { // -ROOT-
      try {
    this.services.getAssignmentManager().assignRoot();
      } catch (KeeperException e) {
        this.server.abort("In server shutdown processing, assigning root", e);
        throw new IOException("Aborting", e);
      }
    }

    // Carrying meta?
    if (isCarryingMeta())this.services.getAssignmentManager().assignMeta();

     
从meta表中获得RS1上的的region

 // Wait on meta to come online; we need it to progress.
     // TODO: Best way to hold strictly here?  We should build this retry logic
     //       into the MetaReader operations themselves.
     NavigableMap hris = null;
     while (!this.server.isStopped()) {
       try {
         this.server.getCatalogTracker().waitForMeta();
         hris = MetaReader.getServerUserRegions(this.server.getCatalogTracker(),
             this.hsi);
         break;
       } catch (InterruptedException e) {
         Thread.currentThread().interrupt();
         throw new IOException("Interrupted", e);
       } catch (IOException ioe) {
         LOG.info("Received exception accessing META during server shutdown of " +
             serverName + ", retrying META read");
       }
     }

 
移除RIT中状态为Closing或是PendingClose的region，得到的就是需要重新分配的regions[
 

 // Skip regions that were in transition unless CLOSING or PENDING_CLOSE
    for (RegionState rit : regionsInTransition) {
      if (!rit.isClosing() && !rit.isPendingClose()) {
        LOG.debug("Removed " + rit.getRegion().getRegionNameAsString() +
          " from list of regions to assign because in RIT");
        hris.remove(rit.getRegion());
      }
    }

     
我们可以看到，在move过程中，若region尚未上线，此时master的RIT队列中region的状态是OFFLINE，而该region在master的角度来看是offline的，然而在severshutdown的处理中认为该region是要重新分配的。若在此时region在RS2上线了，那么而master依然要对该region进行分配这就导致了region的两次分配过程[code="java"]    

 LOG.info("Reassigning " + hris.size() + " region(s) that " + serverName +
       " was carrying (skipping " + regionsInTransition.size() +
       " regions(s) that are already in transition)");

     // Iterate regions that were on this server and assign them
     for (Map.Entry e: hris.entrySet()) {
       if (processDeadRegion(e.getKey(), e.getValue(),
           this.services.getAssignmentManager(),
           this.server.getCatalogTracker())) {
         this.services.getAssignmentManager().assign(e.getKey(), true);
       }
     }
     this.deadServers.finish(serverName);
     LOG.info("Finished processing of shutdown of " + serverName);
   }

  

整个过程起始是在rs shundown以后master需要对rs上的region重新assign到其他rs中去，但是在需要assign的region的序曲上出现了一些问题，将在master中已经offline的但是依然在RIT队列中的region重新分配，由于这些region可能已经在其他地方正在分配，但是还没有上线。当master开始assign region的时候此时region上线了，master依旧会继续分配重而导致region的两次分配