HBase启动

# HBase中组件及启动流程记录

 

## 1.服务组件

- Client

    -包含访问HBase的接口,并维护cache来加快对HBase的访问,比如region的位置信息

   

- Master

    - 为Region server分配region

    - 负责Region server的负载均衡

    - 发现失效的Region server并重新分配其上的region

    - 管理用户对table的增删改查操作 (这个和client是怎么关联的)

   

- Region Server

    - Regionserver维护region,处理对这些region的IO请求

    - Regionserver负责切分在运行过程中变得过大的region

    - HStore

        - HBase存储的核心。由MemStore和StoreFile组成

    - HLog

        -防止HRegionServer意外退出,MemStore中的内存数据就会丢失

   

- ZooKeeper

    -通过选举,保证任何时候,集群中只有一个master,Master与RegionServers 启动时会向ZooKeeper注册

    - 存贮所有Region的寻址入口

    - 实时监控Region server的上线和下线信息。并实时通知给Master

    - 存储HBase的schema和table元数据

    - 默认情况下,HBase 管理ZooKeeper 实例,比如,启动或者停止ZooKeeper

    - Zookeeper的引入使得Master不再是单点故障

    - Zookeeper Quorum存储-ROOT-表地址、HMaster地址

 

## 2.调用图

- HBase服务架构图

> ![alttext](./HBase服务架构图.png "调用图片")

 

 

-HRegionServer下结构解析图

> ![alttext](./HRegionServer下结构解析图.png "调用图片")

 

 

- HBase服务间调度流程图

> ![alttext](./HBase服务间调度流程.png "调用图片")

 

 

## 3.Client组件

 

 

## 4.Master组件

### HMaster的main方法调用

1.HMaster的入口是main方法,main方法需要传递一个参数,start或者stop。

```java

public classHMasterStarter {

    public static void main(String[] args)throws Exception {

            File f = TestBase.getTestDir();

            //删除临时测试目录

            deleteRecursive(f.listFiles());

            //zookeeper类启动

            new ZookeeperThread().start();

            Thread.sleep(1000);

            //运行HMaster服务,这里是开始服务

            HMaster.main(new String[] {"start" }); 

    }

   

    public static void deleteRecursive(File[]files) {

        if (files == null)

            return;

        for (File f : files) {

            if (f.isDirectory()) {

                deleteRecursive(f.listFiles());

            }

            f.delete();

        }

    }

   

    public static class ZookeeperThread extendsThread {

        public void run() {

            MiniZooKeeperClusterzooKeeperCluster = new MiniZooKeeperCluster();

 

            File zkDataPath = newFile(TestBase.sharedConf.get(HConstants.ZOOKEEPER_DATA_DIR));

            int zkClientPort =TestBase.sharedConf.getInt(HConstants.ZOOKEEPER_CLIENT_PORT, 2181);

           zooKeeperCluster.setDefaultClientPort(zkClientPort);

            try {

               zooKeeperCluster.startup(zkDataPath);

            } catch (Exception e) {

                e.printStackTrace();

            }

        }

    }

}

```

   

2.main方法内首先打印hbase版本信息,然后在调用HMasterCommandLine的doMain方法。

```java

packageorg.apache.hadoop.hbase.master;

 

@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.TOOLS)

@SuppressWarnings("deprecation")

public class HMasterextends HRegionServer implements MasterServices, Server {

    /**

      * @seeorg.apache.hadoop.hbase.master.HMasterCommandLine

      */

    public static void main(String [] args) {

        VersionInfo.logVersion();

        newHMasterCommandLine(HMaster.class).doMain(args);

    }

   

}

```

 

3.doMain方法内会调用ToolRunner的run方法,ToolRunner为Hadoop的类。HMasterCommandLine继承自ServerCommandLine类,ServerCommandLine类实现了Tool接口。所以最后会调用HMasterCommandLine的run方法。

```java

packageorg.apache.hadoop.hbase.util;

 

/**

 * Base class for command lines that start upvarious HBase daemons.

 */

@InterfaceAudience.Private

public abstractclass ServerCommandLine extends Configured implements Tool {

    public void doMain(String args[]) {

        try {

           //this将实例HMasterCommandLine实现的run方法传递进去,达到调用的效果

            int ret =ToolRunner.run(HBaseConfiguration.create(), this, args);

            if (ret != 0) {

                System.exit(ret);

            }

        } catch (Exception e) {

            LOG.error("Failed torun", e);

            System.exit(-1);

        }

    }

}

```

```java

packageorg.apache.hadoop.util;

 

public classToolRunner {

    public static int run(Configuration conf,Tool tool, String[] args) throws Exception {

        if(conf == null) {

            conf = new Configuration();

        }

 

        GenericOptionsParser parser = newGenericOptionsParser(conf, args);

        tool.setConf(conf);

        String[] toolArgs =parser.getRemainingArgs();

        //运行外部传入的实现方法

        return tool.run(toolArgs);

    }

}

```

 

4.解析参数,根据参数值判断是执行startMaster方法还是stopMaster方法。

```java

packageorg.apache.hadoop.hbase.master;

 

@InterfaceAudience.Private

public classHMasterCommandLine extends ServerCommandLine {

    public int run(String args[]) throwsException {

        Options opt = new Options();

       opt.addOption("localRegionServers", true,

          "RegionServers to start inmaster process when running standalone");

        opt.addOption("masters",true, "Masters to start in this process");

       opt.addOption("minRegionServers", true, "MinimumRegionServers needed to host user tables");

        opt.addOption("backup",false, "Do not try to become HMaster until the primary fails");

   

        CommandLine cmd;

        try {

          cmd = new GnuParser().parse(opt,args);

        } catch (ParseException e) {

          LOG.error("Could not parse:", e);

          usage(null);

          return 1;

        }

   

   

        if(cmd.hasOption("minRegionServers")) {

          String val =cmd.getOptionValue("minRegionServers");

         getConf().setInt("hbase.regions.server.count.min",Integer.parseInt(val));

          LOG.debug("minRegionServers setto " + val);

        }

   

        // minRegionServers used to beminServers.  Support it too.

        if(cmd.hasOption("minServers")) {

          String val =cmd.getOptionValue("minServers");

         getConf().setInt("hbase.regions.server.count.min",Integer.parseInt(val));

          LOG.debug("minServers set to" + val);

        }

   

        // check if we are the backup master -override the conf if so

        if (cmd.hasOption("backup")){

         getConf().setBoolean(HConstants.MASTER_TYPE_BACKUP, true);

        }

   

        // How many regionservers to startup inthis process (we run regionservers in same process as

        // master when we are inlocal/standalone mode. Useful testing)

        if(cmd.hasOption("localRegionServers")) {

          String val =cmd.getOptionValue("localRegionServers");

         getConf().setInt("hbase.regionservers",Integer.parseInt(val));

          LOG.debug("localRegionServersset to " + val);

        }

        // How many masters to startup insidethis process; useful testing

        if (cmd.hasOption("masters")){

          String val =cmd.getOptionValue("masters");

         getConf().setInt("hbase.masters", Integer.parseInt(val));

          LOG.debug("masters set to "+ val);

        }

   

       @SuppressWarnings("unchecked")

        List remainingArgs =cmd.getArgList();

        if (remainingArgs.size() != 1) {

          usage(null);

          return 1;

        }

   

        String command = remainingArgs.get(0);

   

        if ("start".equals(command)){

          return startMaster();

        } else if("stop".equals(command)) {

          return stopMaster();

        } else if("clear".equals(command)) {

          return (ZNodeClearer.clear(getConf())? 0 : 1);

        } else {

          usage("Invalid command: " +command);

          return 1;

        }

      }

}

```

 

5.假如是startMaster,分两种情况:本地模式和分布式模式。如果是分布式模式,通过反射调用HMaster的构造方法,并调用其start和join方法。如果是本地模式,将会运行一个LocalHBaseCluster,包括:LocalHMaster、HRegionServer、MiniZooKeeperCluster

```java

packageorg.apache.hadoop.hbase.master;

 

@InterfaceAudience.Private

public classHMasterCommandLine extends ServerCommandLine {

    private int startMaster() {

        Configuration conf = getConf();

        try {

            // If 'local', defer toLocalHBaseCluster instance.  Startsmaster

            // and regionserver both in the oneJVM.

            //判断是本地模式还是分布式模式

            if(LocalHBaseCluster.isLocal(conf)) {

               DefaultMetricsSystem.setMiniClusterMode(true);

                final MiniZooKeeperClusterzooKeeperCluster = new MiniZooKeeperCluster(conf);

                File zkDataPath = newFile(conf.get(HConstants.ZOOKEEPER_DATA_DIR));

 

                // find out the default clientport

                int zkClientPort = 0;

 

                // If the zookeeper client portis specified in server quorum, use it.

                String zkserver =conf.get(HConstants.ZOOKEEPER_QUORUM);

                if (zkserver != null) {

                    String[] zkservers =zkserver.split(",");

 

                    if (zkservers.length >1) {

                        // In local modedeployment, we have the master + a region server and zookeeper server

                        // started in the sameprocess. Therefore, we only support one zookeeper server.

                        String errorMsg ="Could not start ZK with " + zkservers.length +

                                " ZKservers in local mode deployment. Aborting as clients (e.g. shell) will not"

                                + "be ableto find this ZK quorum.";

                       System.err.println(errorMsg);

                        throw newIOException(errorMsg);

                    }

 

                    String[] parts =zkservers[0].split(":");

 

                    if (parts.length == 2) {

                        // the second part isthe client port

                        zkClientPort =Integer.parseInt(parts[1]);

                    }

                }

                // If the client port could notbe find in server quorum conf, try another conf

                if (zkClientPort == 0) {

                    zkClientPort =conf.getInt(HConstants.ZOOKEEPER_CLIENT_PORT, 0);

                    // The client port has tobe set by now; if not, throw exception.

                    if (zkClientPort == 0) {

                        throw newIOException("No config value for " +HConstants.ZOOKEEPER_CLIENT_PORT);

                    }

                }

               zooKeeperCluster.setDefaultClientPort(zkClientPort);

                // set the ZK tick time ifspecified

                int zkTickTime =conf.getInt(HConstants.ZOOKEEPER_TICK_TIME, 0);

                if (zkTickTime > 0) {

                   zooKeeperCluster.setTickTime(zkTickTime);

                }

 

                // login the zookeeper serverprincipal (if using security)

                ZKUtil.loginServer(conf,HConstants.ZK_SERVER_KEYTAB_FILE,

                       HConstants.ZK_SERVER_KERBEROS_PRINCIPAL, null);

                intlocalZKClusterSessionTimeout =

                       conf.getInt(HConstants.ZK_SESSION_TIMEOUT +".localHBaseCluster", 10 * 1000);

               conf.setInt(HConstants.ZK_SESSION_TIMEOUT,localZKClusterSessionTimeout);

                LOG.info("Starting azookeeper cluster");

                int clientPort =zooKeeperCluster.startup(zkDataPath);

                if (clientPort != zkClientPort){

                    String errorMsg ="Could not start ZK at requested port of " +

                            zkClientPort +".  ZK was started at port: " +clientPort +

                            ".  Aborting as clients (e.g. shell) will not beable to find " +

                            "this ZKquorum.";

                   System.err.println(errorMsg);

                    throw newIOException(errorMsg);

                }

               conf.set(HConstants.ZOOKEEPER_CLIENT_PORT,Integer.toString(clientPort));

 

                // Need to have the zk clustershutdown when master is shutdown.

                // Run a subclass that does thezk cluster shutdown on its way out.

                int mastersCount =conf.getInt("hbase.masters", 1);

                int regionServersCount =conf.getInt("hbase.regionservers", 1);

                LOG.info("Starting upinstance of localHBaseCluster; master=" + mastersCount +

                        ",regionserversCount=" + regionServersCount);

               //在这里启动一个本地集群,看到里面带有一个LocalHMaster和一个HRegionServer

                LocalHBaseCluster cluster = newLocalHBaseCluster(conf, mastersCount, regionServersCount,

                        LocalHMaster.class,HRegionServer.class);

                //在里面再加入一个MiniZooKeeperCluster

                ((LocalHMaster)cluster.getMaster(0)).setZKCluster(zooKeeperCluster);

                cluster.startup();

                waitOnMasterThreads(cluster);

            } else {

                logProcessInfo(getConf());

                CoordinatedStateManager csm =

                       CoordinatedStateManagerFactory.getCoordinatedStateManager(conf);

                //构造HMaster

                HMaster master =HMaster.constructMaster(masterClass, conf, csm);

                if (master.isStopped()) {

                    LOG.info("Won't bringthe Master up as a shutdown is requested");

                    return 1;

                }

                master.start();

                master.join();

                if (master.isAborted())

                    throw newRuntimeException("HMaster Aborted");

            }

        } catch (Throwable t) {

            LOG.error("Masterexiting", t);

            return 1;

        }

        return 0;

    }

}

```

 

6.如果是stopMaster,将会尝试调用HBaseAdmin的shutdown方法,最后使用close方法。

```java

packageorg.apache.hadoop.hbase.master;

 

public classHMasterCommandLine extends ServerCommandLine {

    private int stopMaster() {

        Admin adm = null;

        try {

            Configuration conf = getConf();

            // Don't try more than once

           conf.setInt(HConstants.HBASE_CLIENT_RETRIES_NUMBER, 1);

            adm = new HBaseAdmin(getConf());

            adm.shutdown();

        } catch (MasterNotRunningException e) {

            LOG.error("Master notrunning");

            return 1;

        } catch (ZooKeeperConnectionExceptione) {

            LOG.error("ZooKeeper notavailable");

            return 1;

        } catch (IOException e) {

            LOG.error("Got IOException:" + e.getMessage(), e);

            return 1;

        } catch (Throwable t) {

            LOG.error("Failed to stopmaster", t);

            return 1;

        } finally {

            if (adm != null) {

                try {

                    adm.close();

                } catch (Throwable t) {

                    LOG.error("Failed toclose Admin", t);

                    return 1;

                }

            }

        }

        return 0;

    }

}

```

 

 

7.HMaster继承自HasThread类,而HasThread类实现了Runnable接口,故HMaster也是一个线程。

```java

public class HMasterextends HRegionServer implements MasterServices, Server{};

public classHRegionServer extends HasThread implements RegionServerServices,LastSequenceId, ConfigurationObserver {};

public abstractclass HasThread implements Runnable {};

 

```

 

8. HMaster类继承关系

> ![alttext](./HMaster类继承关系.jpg)

 

 

### HMaster的构造方法调用

1.创建Configuration并设置和获取一些参数,默认参数在HConstants中。(发现新版HBase1.3.0代码已和老版0.9.6有较大差异,配置的运行和实现在startActiveMasterManager()方法下;而且HMaster继承HRegionServer,大部分操作实际都在HRegionServer中进行)

    - 在master上禁止block cache

    - 设置服务端重试次数

    - 获取主机名称和master绑定的ip和端口号,端口号默认为60000

    - 设置regionserver的coprocessorhandler线程数为0

    - 创建rpcServer(见下文分析)

    -初始化serverName,其值为:192.168.1.129,60000,1404117936154

    - zk授权登录和hbase授权

    - 设置当前线程名称:master + "-" +this.serverName.toString()

    -判断是否开启复制:Replication.decorateMasterConfiguration(this.conf);

    -设置mapred.task.id,如果其为空,则其值为:"hb_m_" + this.serverName.toString()

    -创建ZooKeeperWatcher监听器(见下文分析),并在zookeeper上创建一些节点

    - 启动rpcServer中的线程

    - 创建一个MasterMetrics

    - 判断是否进行健康检测:HealthCheckChore

    -另外还初始化两个参数:shouldSplitMetaSeparately、waitingOnLogSplitting

   

    ```java

    package org.apache.hadoop.hbase.master;

   

    public class HMaster extends HRegionServerimplements MasterServices, Server {

        /**

         * Utility for constructing an instanceof the passed HMaster class.

         * @param masterClass

         * @param conf

         * @return HMaster instance.

         */

        public static HMasterconstructMaster(Class masterClass,

                                             final Configuration conf, final CoordinatedStateManager cp) {

            try {

                Constructor c =

                       masterClass.getConstructor(Configuration.class,CoordinatedStateManager.class);

                //这里的conf就是配置文件

                return c.newInstance(conf, cp);

            } catch (Exception e) {

                Throwable error = e;

                if (e instanceofInvocationTargetException &&

                       ((InvocationTargetException) e).getTargetException() != null) {

                    error =((InvocationTargetException) e).getTargetException();

                }

                throw newRuntimeException("Failed construction of Master: " +masterClass.toString() + ". "

                        , error);

            }

        }

    }

    ```

 

    以下为涉及参数(网上复制,没有验证准确性,和代码核对后,发现后很多都已经改动或者移除)

    >

        hfile.block.cache.size

        hbase.master.dns.interface

        hbase.master.dns.nameserver

        hbase.master.port

        hbase.master.ipc.address

        hbase.master.handler.count

        hbase.regionserver.handler.count

        hbase.master.buffer.for.rs.fatals

        hbase.zookeeper.client.keytab.file

       hbase.zookeeper.client.kerberos.principal

        hbase.master.keytab.file

        hbase.master.kerberos.principal

        hbase.master.logcleaner.plugins

        mapred.task.id

        hbase.node.health.script.frequency

       hbase.regionserver.separate.hlog.for.meta

        hbase.master.wait.for.log.splitting

 

2.创建rpcServer并启动其中的线程

 

    2.1通过反射创建RpcEngine的实现类,实现类可以在配置文件中配置(hbase.rpc.engine现在已变为hbase.security.authentication),默认实现为WritableRpcEngine。调用getServer方法,其实也就是new一个HBaseServer类。

   

    2.2 运行构造方法

    >

       启动一个Listener线程,功能是监听client的请求,将请求放入nio请求队列,逻辑如下:

          –>创建nselector,和一个n个线程的readpoolnipc.server.read.threadpool.size决定,默认为10

          –>读取每个请求的头和内容,将内容放入priorityQueue

       

       启动一个Responder线程,功能是将响应队列里的数据写给各个client的connection通道,逻辑如下:

          –>创建nio selector

          –>默认超时时间为15 mins

          –>依次将responseQueue中的内容写回各通道,并关闭连接,buffer=8k

          –>如果该请求的返回没有写完,则放回队列头,推迟再发送

          –>对于超时未完成的响应,丢弃并关闭相应连接

       

       启动N(n默认为10)个Handler线程,功能是处理请求队列,并将结果写到响应队列

          –>读取priorityQueue中的call,调用对应的call方法获得value,写回out并调用doRespond方法,处理该请求,并唤醒writable selector

          –>启动M(m默认为0)Handler线程以处理priority

         

        启动一个Scheduler线程,是一个调度器(自己添加的,网上都没说,在代码中发现)

 

    2.3 创建ZooKeeperWatcher

    构造函数中生成如下持久节点:

    >

        /hbase

        /hbase/root-region-server

        /hbase/rs

        /table/draining

        /hbase/master

        /hbase/backup-masters

        /hbase/shutdown

        /hbase/unassigned

        /hbase/table94

        /hbase/table

        /hbase/hbaseid

        /hbase/splitlog

 

 

### run方法调用

-我这里已经修改不下去了,1.3.0版本的HMaster根本就没有run方法,下面的步骤基本还在,但是都转移到HRegionServer的run方法下、构造方法、构造方法的startActiveMasterManager()里去了。

1. 总体过程

    - 创建MonitoredTask,并把HMaster的状态设置为Masterstartup(新版本1.3.0放入HBase自封装的线程类中运行)

    - 启动infoserver,即Jetty服务器,端口默认为60010,其对外提供两个接口:/master-status和/dump(新版本1.3.0在HMaster的构造方法中运行)

    -调用becomeActiveMaster方法(见下文分析),阻塞等待直至当前master成为activemaster(新版本1.3.0修改为startActiveMasterManager方法)

    -当成为了master之后并且当前master进程正在运行,则调用finishInitialization方法(见下文分析),并且调用loop方法循环等待,一直到stop发生

    - 当HMaster停止运行时候,会做以下事情:

        - 清理startupStatus

        - 停止balancerChore和catalogJanitorChore

        - 让RegionServers shutdown

        -停止服务线程:rpcServer、logCleaner、hfileCleaner、infoServer、executorService、healthCheckChore

        -停止以下线程:activeMasterManager、catalogTracker、serverManager、assignmentManager、fileSystemManager、snapshotManager、zooKeeper

           

2.becomeActiveMaster方法

    - 创建ActiveMasterManager

    - ZooKeeperWatcher注册activeMasterManager监听器

    - 调用stallIfBackupMaster:–>先检查配置项 “hbase.master.backup”,自己是否backup机器,如果是则直接block直至检查到系统中的active master挂掉(zookeeper.session.timeout,默认每3分钟检查一次)

    - 创建clusterStatusTracker并启动

    -调用activeMasterManager的blockUntilBecomingActiveMaster方法。

        -创建短暂的”/hbase/master”,此节点值为version+ServerName,如果创建成功,则删除备份节点;否则,创建备份节点

        -获得”/hbase/master”节点上的数据,如果不为null,则获得ServerName,并判断是否是在当前节点上创建了”/hbase/master”,如果是则删除该节点,这是因为该节点已经是备份节点了。

        ```java

        public class HMaster{

            private booleanbecomeActiveMaster(MonitoredTask startupStatus)throws InterruptedException {

                // TODO: This is wrong!!!!Should have new servername if we restart ourselves,

                // if we come back to life.

                //创建activeMasterManager对象

                this.activeMasterManager = newActiveMasterManager(zooKeeper, this.serverName,

                    this);

               //注册activeMasterManager到zookeeper

               this.zooKeeper.registerListener(activeMasterManager);

                stallIfBackupMaster(this.conf,this.activeMasterManager);

           

                // The ClusterStatusTracker issetup before the other

                // ZKBasedSystemTrackersbecause it's needed by the activeMasterManager

                // to check if the clustershould be shutdown.

                this.clusterStatusTracker = newClusterStatusTracker(getZooKeeper(), this);

               this.clusterStatusTracker.start();

                returnthis.activeMasterManager.blockUntilBecomingActiveMaster(startupStatus);

              }

        }

      ```

3.finishInitialization方法

    -创建MasterFileSystem对象,封装了master常用的一些文件系统操作,包括splitlogfile、删除region目录、删除table目录、删除cf目录、检查文件系统状态等.

    -创建FSTableDescriptors对象(新版本1.3.0在HRegionServer的构造函数上创建)

    - 设置集群id(publish cluster ID)

    ```

    ZKClusterId.setClusterId(this.zooKeeper,fileSystemManager.getClusterId());

    ```

    - 如果不是备份master:

        -创建ExecutorService,维护一个ExecutorMap,一种Event对应一个Executor(线程池)。可以提交EventHandler来执行异步事件;(新版本1.3.0在HRegionServer的构造函数中创建)

        - 创建serverManager,管理regionserver信息,维护着onlineregion server和deadregionserver列表,处理regionserver的startups、shutdowns、 deaths,同时也维护着每个regionserver rpcstub。

    -调用initializeZKBasedSystemTrackers,初始化zk文件系统:

        - 创建CatalogTracker,它包含RootRegionTracker和MetaNodeTracker,对应”/hbase/root-region-server”和/”hbase/unassigned/1028785192”这两个结点(1028785192是.META.的分区名)。如果之前从未启动过hbase,那么在startCatalogTracker时这两个结点不存在。”/hbase/root-region-server”是一个持久结点,在RootLocationEditor中建立

        - 创建LoadBalancer,负责region在regionserver之间的移动,关于balancer的策略,可以通过hbase.regions.slop来设置load区间

        - 创建AssignmentManager,负责管理和分配region,同时它也会接受zk上关于region的event,根据event来完成region的上下线、关闭打开等工作。

        - 创建 RegionServerTracker:监控”/hbase/rs”结点,通过ZK的Event来跟踪onlineregion servers,如果有rs下线,删除ServerManager中对应的onlineregions.

        - 创建 DrainingServerTracker:监控”/hbase/draining”结点.在大型集群中, 用户有时需要把一部分regionserver服务器一起停止服务迁出HBase集群,这些将要被迁出的region server称为draining region server。

        - 创建ClusterStatusTracker,监控”/hbase/shutdown”结点维护集群状态

        - 创建SnapshotManager:负责管理HBase快照功能,包括创建快照,恢复快照。快照是HBase从0.94.6开始提供的功能,创建快照时并不复制数据,因此速度很快,对regionserver性能基本没有影响。

    -如果不是备份master,初始化MasterCoprocessorHost并执行startServiceThreads()。说明:infoserver的启动移到构造函数了去了,这样可以早点通过Jetty服务器查看HMaster启动状态。

        - 创建一些executorService

        - 创建logCleaner并启动

        - 创建hfileCleaner并启动

        - 启动healthCheckChore

        - 打开rpcServer

    - 等待RegionServer注册。满足以下这些条件后返回当前所有regionserver上的region数后继续:

        - a至少等待4.5s,"hbase.master.wait.on.regionservers.timeout"

        - b成功启动regionserver节点数>=1,"hbase.master.wait.on.regionservers.mintostart"

        - c1.5s内没有regionsever死掉或重新启动,"hbase.master.wait.on.regionservers.interval"

    - serverManager注册新的在线region server

    - 如果不是备份master,启动assignmentManager

    - 获取下线的Region server,然后拆分HLog

        ```

        // get a list for previously failed RSwhich need log splitting work

        // we recover hbase:meta region serversinside master initialization and

        // handle other failed servers in SSHin order to start up master node ASAP

        SetpreviouslyFailedServers =

                       this.fileSystemManager.getFailedServersFromLogFolders();

                       

        // log splitting for hbase:metaserver               

        ServerName oldMetaServerLocation =metaTableLocator.getMetaRegionLocation(this.getZooKeeper());

        if (oldMetaServerLocation != null&& previouslyFailedServers.contains(oldMetaServerLocation)) {

           splitMetaLogBeforeAssignment(oldMetaServerLocation);

            // Note: we can't removeoldMetaServerLocation from previousFailedServers list because it

            // may also host user regions

        }

        ```

        - –>依次检查每一个hlog目录,查看它所属的region server是否online,如果是则不需要做任何动作,region server自己会恢复数据,如果不是,则需要将它分配给其它的region server

        - –>split是加锁操作:

        - –> 创建一个新的hlogsplitter,遍历每一个server目录下的所有hlog文件,依次做如下操作。(如果遇到文件损坏等无法跳过的错误,配 hbase.hlog.split.skip.errors=true 以忽略之)

        - –>启动hbase.regionserver.hlog.splitlog.writer.threads(默认为3)个线程,共使用128MB内存,启动这些写线程

        - –>先通过lease机制检查文件是否能够append,如果不能则死循环等待

            - –>hlog中的内容全部加载到内存中(内存同时被几个写线程消费)

            - –>把有损坏并且跳过的文件移到/hbase/.corrupt/目录中

            - –> 把其余己经处理过的文件移到/hbase/.oldlogs中,然后删除原有的server目录

            - –> 等待写线程结束,返回新写的所有路径

        - –>解锁

        - 写线程逻辑:

            - –>从内存中读出每一行数据的keyvalue,然后查询相应的region路径。如果该region路径不存在,说明该region很可能己经被split了,则不处理这部分数据,因为此时忽略它们是安全的。

            - –>如果上一步能查到相应的路径,则到对应路径下创建”recovered.edits”文件夹(如果该文件夹存在则删除后覆盖之),然后将数据写入该文件夹

    -调用assignRoot方法,检查是否分配了-ROOT-表,如果没有,则通过assignmentManager.assignRoot()来分配root表,并激活该表

    - 运行this.serverManager.enableSSHForRoot()方法

    - 拆分.META. server上的HLog

    - 分配.META.表

    - enableServerShutdownHandler

    - 处理dead的server

    - assignmentManager.joinCluster();

    - 设置balancer

    - fixupDaughters(status)

    - 如果不是备份master

        - 启动balancerChore线程,运行LoadBalancer

        -启动startCatalogJanitorChore,周期性扫描.META.表上未使用的region并回收

        - registerMBean

    -serverManager.clearDeadServersWithSameHostNameAndPortOfOnlineServer(),清理dead的server

    - 如果不是备份master,cpHost.postStartMaster()

   

   

###MasterFileSystem构造方法调用

1.在HMaster.finishInitialization方法中触发了MasterFileSystem的构造方法,该类在HMaster类中会被以下类使用:

    - LogCleaner

    - HFileCleaner

   

   另外该类可以完成拆分log的工作(新版本1.3.0中由SplitLogManager完成该工作):

    ```

    /**

      * Override to change master'ssplitLogAfterStartup. Used testing

      * @param mfs

      */

    protected void splitLogAfterStartup(finalMasterFileSystem mfs){

        mfs.splitLogAfterStartup();

    }

    ```

   

2. 构造方法运行过程

    - 获取rootdir:由参数hbase.rootdir配置

    - 获取tempdir:${hbase.rootdir}/.tmp

    -获取文件系统的uri,并设置到fs.default.name和fs.defaultFS

    -判断是否进行分布式文件拆分,参数:hbase.master.distributed.log.splitting,如果需要,则创建SplitLogManager

    -创建oldLogDir,调用createInitialFileSystemLayout方法

        - checkRootDir

            -等待fs退出安全模式(默认10秒钟轮循一次,可通过参数hbase.server.thread.wakefrequency调整

            -如果hbase.rootdir目录不存在则创建它,然后在此目录中创建名为”hbase.version”的文件,内容是文件系统版本号,当前为7;如果hbase.rootdir目录已存在,则读出”hbase.version”文件的内容与当前的版本号相比,如果不相等,则打印错误信息(提示版本不对),抛出异常FileSystemVersionException

            -检查${hbase.rootdir}目录下是否有名为”hbase.id”的文件,如果没有则创建它,内容是随机生成的UUID(总长度36位,由5部份组成,用”-“分隔),如:6c43f934-37a2-4cae-9d49-3f5abfdc113d

            - 读出”hbase.id”的文件的内容存到clusterId字段

            -判断hbase.rootdir目录中是否有”-ROOT-/70236052”目录,没有的话说明是第一次启动hbase,进入bootstrap方法

            - createRootTableInfo建立”-ROOT-“表的描述文件,判断hbase.rootdir/-ROOT-目录中是否存在tableinfo开头的文件,另外还创建了.tmp目录

        - checkTempDir

        -如果oldLogDir(${hbase.rootdir}/.oldlogs)不存在,则创建

           

3. bootstrap方法运行过程:

    -调用HRegion.createHRegion建立”-ROOT-“分区和”.META.”分区

    - 把”.META.”分区信息加到”-ROOT-“表,并关闭分区和hlog

   

 

### 总结

 

1.zookeeper创建目录分布式与监控类之间关系

 

|目录                       |监控类               |作用                      |

|------------------------ | ------------------- | ------------------------ |

|/hbase                    |                     |                          |

|/hbase/root-region-server|RootRegionTracker   |监控root所在的regionserver |

|/hbase/rs                 |RegionServerTracker  |监控regionserver的上线和下线|

|/table/draining          |DrainingServerTracker|监听regionserver列表的变化  |

|/hbase/master             |                    |在HMaster中建立,并且是一个短暂结点,结点的值是HMaster的ServerName:hostname,port,当前毫秒|

|/hbase/backup-masters     |                     |                          |

|/hbase/shutdown           |ClusterStatusTracker|当HMaster启动之后,会将当前时间(Bytes.toBytes(new java.util.Date().toString()))存到该节点|

|/hbase/unassigned         |MetaNodeTracker      |                          |

|/hbase/table94            |                     |                          |

|/hbase/table              |                     |                          |

|/hbase/hbaseid            |                    |在HMaster.finishInitialization方法中调用ClusterId.setClusterId建立,结点值是UUID|

|/hbase/splitlog           |                     |                          |

 

2.HMaster启动之后,${hbase.rootdir}目录

>

    .

    ├── -ROOT-                           //"-ROOT-"表名

    │  ├── ..tableinfo.0000000001.crc   //crc校验文件

    │  ├── .tableinfo.0000000001

    │  ├── .tmp

    │  └── 70236052                     //"-ROOT-"分区名

    │      ├── ..regioninfo.crc

    │      ├── .oldlogs                 //存放hlog文件

    │      │   ├── .hlog.1402551641526.crc

    │      │   └── hlog.1402551641526

    │      ├── .regioninfo              //"-ROOT-"分区描述表件

    │      ├── .tmp

    │      └── info                     //列族名

    │          ├── .5037e69a0c244bd78945aaa333d0230a.crc

    │          └── 5037e69a0c244bd78945aaa333d0230a //存放".META."分区信息的StoreFile

    ├── .META.

    │  └── 1028785192

    │      ├── ..regioninfo.crc

    │      ├── .oldlogs

    │      │   ├── .hlog.1402551641701.crc

    │      │   └── hlog.1402551641701

    │      ├── .regioninfo

    │      └── info

    ├── .hbase.id.crc

    ├── .hbase.version.crc

    ├── .oldlogs

    ├── .tmp

    ├── hbase.id                         //集群uuid

    └── hbase.version                    //hbase版本

   

3. HMaster启动主要运行过程

    - 创建rpcServer,及HBaseServer

    - 创建ZooKeeperWatcher监听器

    - 阻塞等待成为activeMaster

    - 创建master的一些文件目录

    - 初始化一些基于zk的跟踪器

    - 创建LoadBalancer

    - 创建SnapshotManager

    - 如果不是备份master

        - 创建logCleaner并启动

        - 创建hfileCleaner并启动

        - 创建jetty的infoServer并启动

        - 启动健康检查

        - 打开rpcServer

    - 等待RegionServer注册

    - 从hlog中恢复数据

    - 分配root和meta表

    - 分配region

    - 运行负载均衡线程

    - 周期性扫描.META.表上未使用的region并回收

   

 

## 5.Region Server组件

启动了HMaster之后,再启动HRegionServer。

 

###HRegionServer的mian方法调用

 

1.HRegionServer的入口是main方法,main方法需要传递一个参数,start或者stop。

```java

importorg.apache.hadoop.hbase.regionserver.HRegionServer;

 

public classHRegionServerStarter {

    public static void main(String[] args)throws Exception {

        //newHMasterStarter.ZookeeperThread().start();

        HRegionServer.main(new String[] {"start" });

    }

}

```

 

2.通过反射创建一个HRegionServer实例,然后调用其run方法。(其实步骤都和HMaster一样,难怪新版本1.3.0中HMaster已经继承HRegionServer)

 

3.HRegionServer类继承关系

> ![alttext](./HRegionServer类继承关系.jpg)

 

 

###HRegionServer构造方法调用

 

-设置服务端HConnection重试次数

-检查压缩编码,通过hbase.regionserver.codecs可以配置编码类,一一检测,判断是否支持其压缩算法。

-获取useHBaseChecksum值,是否开启hbase checksum校验

-获取hbase.regionserver.separate.hlog.for.meta参数值

- 获取客户端重复次数

-获取threadWakeFrequency值

-获取hbase.regionserver.msginterval值

-创建Sleeper对象,用于周期性休眠线程

-获取最大扫描结果集大小,hbase.client.scanner.max.result.size,默认无穷大

-获取hbase.regionserver.numregionstoreport值

-获取rpctimeout值,hbase.rpc.timeout,默认60000

-获取主机名和绑定的ip和端口,端口默认为60020

- 创建rpcServer

- zk授权登录和hbase授权

-创建RegionServerAccounting

- 创建CacheConfig

 

 

### run方法

   

-preRegistrationInitialization

    - initializeZooKeeper,此方法不会创建任何节点

        -创建ZooKeeperWatcher(新版本1.3.0修改为在构造函数中运行)

        - 创建MasterAddressTracker并等到”/hbase/master”节点有数据为止

        - 创建ClusterStatusTracker并等到”/hbase/shutdown”节点有数据为止

        - 创建CatalogTracker 不做任何等待

        - 创建RegionServerSnapshotManager

        - 设置集群id

    - 初始化线程:initializeThreads

        - 创建 cacheFlusher

        - 创建 compactSplitThread

        - 创建 compactionChecker

        - 创建 periodicFlusher

        - 创建 healthCheckChore

        - 创建 Leases

        - 判断是否启动 HRegionThriftServer

    -参数hbase.regionserver.nbreservationblocks默认为4,默认会预留20M(每个5M,20M = 4*5M)的内存防止OOM

    - 初始化rpcEngine =HBaseRPC.getProtocolEngine(conf)

-reportForDuty,轮询,向汇报master自己已经启动

    -getMaster(),取出”/hbase/master”节点中的数据,构造一个master的ServerName,然后基于此生成一个HMasterRegionInterface接口的代理,此代理用于调用master的方法

    - regionServerStartup

-当轮询结果不为空时,调用handleReportForDutyResponse

    -regionServerStartup会返回来一个MapWritable,这个MapWritable有三个值,这三个key的值会覆盖rs原有的conf:

        -“hbase.regionserver.hostname.seen.by.master” =master为rs重新定义的hostname(通常跟rs的InetSocketAddress.getHostName一样)rs会用它重新得到serverNameFromMasterPOV

        - “fs.default.name” = “file:///”

        -“hbase.rootdir”        = “file:///E:/hbase/tmp”

    - 查看conf中是否有”mapred.task.id”,没有就自动设一个(格式:“hb_rs_“+serverNameFromMasterPOV),例如: hb_rs_localhost,60050,1323525314060

    - createMyEphemeralNode:在zk中建立短暂节点”/hbase/rs/localhost,60050,1323525314060”,也就是把当前rs的serverNameFromMasterPOV(为null的话用rs的InetSocketAddress、port、startcode构建新的ServerName)放到/hbase/rs节点下,”/hbase/rs/localhost,60050,1323525314060”节点没有数据

    - 设置fs.defaultFS值为hbase.rootdir的值

    -生成一个只读的FSTableDescriptors(新版本1.3.0修改为构造函数中运行)

    - 调用setupWALAndReplication

    - 初始化 hlog、metrics、dynamicMetrics、rsHost

    - 调用startServiceThreads启动服务线程

        - 启动一些ExecutorService

        - 启动hlogRoller

        - 启动cacheFlusher

        - 启动compactionChecker

        - 启动healthCheckChore

        - 启动periodicFlusher

        - leases.start()

        - 启动jetty的infoServer,默认端口为60030(新版本1.3.0修改至构造函数的putUpWebUI)

        -启动复制相关打的一些handler:replicationSourceHandler、replicationSourceHandler、replicationSinkHandler

        - rpcServer启动

        - 创建并启动SplitLogWorker

- registerMBean

-snapshotManager启动快照服务

-在master上注册之后,进入运行模式,周期性(msgInterval默认3妙)调用doMetrics,tryRegionServerReport

-isHealthy健康检查,只要Leases、MemStoreFlusher、LogRoller、periodicFlusher、CompactionChecker有一个线程退出,regionServer就停止

    - doMetrics

    -tryRegionServerReport向master汇报rs的负载HServerLoad

- shutdown之后的一些操作

    - unregisterMBean

    -停掉thriftServer、leases、rpcServer、splitLogWorker、infoServer、cacheConfig

    -中断一些线程:cacheFlusher、compactSplitThread、hlogRoller、metaHLogRoller、compactionChecker、healthCheckChore

    - 停掉napshotManager

    - 停掉 catalogTracker、compactSplitThread

    - 等待所有region关闭

    - 关闭wal

    - 删除zk上的一些临时节点,zooKeeper关闭

 

 

### 总结

 

1. 功能总结

    - 在zk上注册自己,表明自己上线了

    - 跟master汇报

    - 设置wal和复制

    - 注册协作器RegionServerCoprocessorHost

    - 启动hlogRoller

    - 定期刷新memstore

    - 定期检测是否需要压缩合并

    - 启动租约

    - 启动jetty的infoserver

    - 创建SplitLogWorker,用于拆分HLog

    - 快照管理

 

2. 对象总结

    - HBaseServer:处理客户端请求

    - Leases:租约

    - InfoServer:Jetty服务器

    - RegionServerMetrics:

    - RegionServerDynamicMetrics:

    - CompactSplitThread:合并文件线程

    - MemStoreFlusher:刷新memstore线程

    - 两个Chore:compactionChecker、periodicFlusher

    - 两个LogRoller:hlogRoller、metaHLogRoller

    - MasterAddressTracker:跟踪master地址

    - CatalogTracker:跟踪-ROOT-和.META.表

    - ClusterStatusTracker:跟踪集群状态

    - SplitLogWorker:拆分log

    - Sleeper:

    - ExecutorService:

    -ReplicationSourceService和ReplicationSinkService:复制服务

    - RegionServerAccounting:

    - CacheConfig:缓存配置和block

    -RegionServerCoprocessorHost:RegionServer协作器

    - HealthCheckChore:健康检查

 

 

## 6.ZooKeeper组件

 

 

 

 

 

你可能感兴趣的:(做过的技术)