hbase异常解决小记1

查询hbase1中表d是报异常，如下：

Tue Feb 02 13:48:50 CST 2016, RpcRetryingCaller{globalStartTime=1454392129833, pause=100, retries=3}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region d,,1451161931525.ebebcda3b62fbe7c7bce561d24f64df2. is not online on datanode2,16020,1453890442843
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2234)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
    at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
    at java.lang.Thread.run(Thread.java:745)


    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
    at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region d,,1451161931525.ebebcda3b62fbe7c7bce561d24f64df2. is not online on datanode2,16020,1453890442843
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2234)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
    at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
    at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
    at java.lang.Thread.run(Thread.java:745)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
    at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:380)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
    ... 4 more

定位问题：看hbase程序是否挂掉(没有)，查询其他表看是否报异常(没有)，可以看出只是查该表才会出问题，该表的数据量比较大，所以怀疑表d在hbase中出了问题(该表需要经常分裂，无法预知rowkey的内容，所以只能交给hbase做分裂)。

了解一下hbase查询数据的流程，到.META中查询表的region server信息，然后再去对应的region上去取数据，看上面报的异常中有：Region d,,1451161931525.ebebcda3b62fbe7c7bce561d24f64df2. is not online on datanode2，再看代码中hBaseAdmin.tableExists(tableName)返回true，

初步可以判断表d的meta信息出现了问题，看看hbase的ui界面中Tables那个单元中也没有d表，所以决定把表信息删掉再重建。

    // 初始化配置
    static {
        conf = HBaseConfiguration.create();
          try {
            connection  = ConnectionFactory.createConnection(conf);
        } catch (IOException e) {
            logger.error("", e);
        }
    }
    
    public static void dropTable(String tableName) {  
        HBaseAdmin hBaseAdmin =  null; 
        try {  
            hBaseAdmin =  (HBaseAdmin) connection.getAdmin(); 
            if (hBaseAdmin.tableExists(tableName)) {
                hBaseAdmin.disableTable(tableName);
                hBaseAdmin.deleteTable(tableName);
                System.out.println(tableName + " is deleted....");  
                return ;
            }  
            HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf(tableName));  
            tableDescriptor.addFamily(new HColumnDescriptor(Constants.FAMILY1));  
            hBaseAdmin.createTable(tableDescriptor); 
        } catch (Exception e) {  
            logger.error("", e);  
        } finally {
            if(hBaseAdmin != null)
                try {
                    hBaseAdmin.close();
                } catch (IOException e) {
                    logger.error("", e);
                }
        }
    }

执行上面代码后是成功了，表的数据也没有丢，所以猜测是.meta给出的表d的region信息是错误的，把.meta的信息重构了一下。

上面只是解决问题，具体内部细节需要慢慢深入学习hbase了~~~