HBase错误解决:Master is initializing/org.apache.hadoop.hbase.NotServingRegionException)

HBase错误解决

在IDEA中远程连接时提示错误

2019-10-08 15:17:28 DEBUG org.apache.zookeeper.ClientCnxn    Reading reply sessionid:0x10000192d2c000b, packet:: clientPath:/hbase/meta-region-server serverPath:/hbase/meta-region-server finished:false header:: 11,4  replyHeader:: 11,94489280688,0  request:: '/hbase/meta-region-server,F  response:: #ffffffff000146d61737465723a3136303030ffffff9b44fffffff0ffffff8b5f48ffffffbbffffffbc50425546a12a6736c6176653210ffffff947d18ffffffffffffffbfffffff8cffffffb0ffffffcf2d100183,s{34359738425,60129596943,1561451869773,1569062411965,17,0,0,0,53,0,34359738425} 
2019-10-08 15:17:28 DEBUG org.apache.zookeeper.ClientCnxn    Got ping response for sessionid: 0x10000192d2c000b after 2ms
2019-10-08 15:17:28 DEBUG org.apache.hadoop.hbase.client.RpcRetryingCallerImpl    Call exception, tries=9, retries=16, started=29249 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave2,16020,1570518833642
	...
, details=row 'user_table' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=slave2,16020,1567495495679, seqNum=-1, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave2,16020,1570518833642
	...
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.NotServingRegionException): org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on slave2,16020,1570518833642
	...

接着在hbase shell中尝试创建表失败,错误提示:

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
	at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2946)
	at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1942)
	at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:603)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)

查看hbase log文件,错误提示:

2019-10-08 15:41:19,966 WARN  [Thread-14] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1570520219373, server=slave2,16020,1567495495679}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined; operator intervention required. Schedule an assign.
2019-10-08 15:45:35,967 WARN  [Thread-14] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1570520219373, server=slave2,16020,1567495495679}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined; operator intervention required. Schedule an assign.

解决办法

# 进入zookeeper目录
/home/hadoop/zookeeper/apache-zookeeper-3.5.5/bin
# 运行zookeeper环境命令
zkCli.sh -server localhost:2181
# 删除zookeeper下的hbase(会提示可以用deleteall代替,但该命令还是可以用)
rmr /hbase

之后远程IDEA连接以及建表都可以正常使用,原因就是在关闭重启集群前得关闭hadoop\hbase\zookpeer,不然再次启动会有奇奇怪怪的问题.