HBase Balancer失败日志报错 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler

现象

HBase shell下面打开Balance并手动执行Balancer很快结束，显示为false（此时balance_switch为true）

hbase(main):007:0* balancer
false  

查看RegionServer日志有如下报错信息，

2020-07-08 16:49:01,821 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => 830a8f32d019fa5122cd533fc68ef1a1, NAME => 'TRAFODION.V7PE
RF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 7
2020-07-08 16:49:01,965 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Open TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.
2020-07-08 16:49:01,971 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122c
d533fc68ef1a1., starting to roll back the global memstore size.
java.lang.IllegalStateException: Could not instantiate a region instance.
        at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6351)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6652)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6624)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6580)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6531)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
        at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6348)
        ... 10 more
Caused by: java.lang.IllegalArgumentException: Need table descriptor
        at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:657)
        at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:634)
        at org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion.(TransactionalRegion.java:70)
        ... 15 more

解决

1. hbase shell使用list 'TRAFODION.V7PERF.KAPB_CPXXHZ,'检查表是否存在 --不存在
2. hadoop fs -ls /hbase/data/default/TRAFODION.V7PERF.KAPB_CPXXHZ检查目录是否存在 --不存在
3. scan hbase:meta表检查是否有残留数据 --是

scan 'hbase:meta', {STARTROW=>'TRAFODION.V7PERF.KAPB_CPXXHZ', LIMIT=>1} 
hbase(main):002:0> scan 'hbase:meta', {STARTROW=>'TRAFODION.V7PERF.KAPB_CPXXHZ', LIMIT=>1} 
ROW                                                 COLUMN+CELL                                                                                                                                           
 TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:regioninfo, timestamp=1579251646633, value={ENCODED => 830a8f32d019fa5122cd533fc68ef1a1, NAME => 'TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646
 32d019fa5122cd533fc68ef1a1.                        163.830a8f32d019fa5122cd533fc68ef1a1.', STARTKEY => '', ENDKEY => ''}                                                                                 
 TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:seqnumDuringOpen, timestamp=1579251646633, value=\x00\x00\x00\x00\x00\x00\x00\x02                                                         
 32d019fa5122cd533fc68ef1a1.                                                                                                                                                                              
 TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:server, timestamp=1579251646633, value=hxpernode08.esgyn.cn:60020                                                                         
 32d019fa5122cd533fc68ef1a1.                                                                                                                                                                              
 TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f column=info:serverstartcode, timestamp=1579251646633, value=1579054953133                                                                             
 32d019fa5122cd533fc68ef1a1.                                                                                                                                                                              
1 row(s) in 0.0150 seconds

4. delete根据table/rowkey/column删除对应信息

delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:regioninfo' 
delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:seqnumDuringOpen' 		
delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:server' 
delete 'hbase:meta','TRAFODION.V7PERF.KAPB_CPXXHZ,,1579251646163.830a8f32d019fa5122cd533fc68ef1a1.','info:serverstartcode'

5. 重启HBase，问题解决