启动Hbase时其中一个节点连不上zookeeper问题解决（maxClientCnxns）

阅读更多

安装hbase集群时，到启动集群时其中一个节点报错如下：
2016-09-09 15:32:57,749 INFO  [main-SendThread(madwx31:2181)]

zookeeper.ClientCnxn: Socket connection established to madwx31/192.168.32.31:2181, initiating session
2016-09-09 15:32:57,749 INFO  [main-SendThread(madwx31:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-09-09 15:32:59,379 INFO  [main-SendThread(madwx51:2181)] zookeeper.ClientCnxn: Opening socket connection to server madwx51/192.168.32.51:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-09 15:32:59,380 INFO  [main-SendThread(madwx51:2181)] zookeeper.ClientCnxn: Socket connection established to madwx51/192.168.32.51:2181, initiating session
2016-09-09 15:32:59,380 WARN  [main-SendThread(madwx51:2181)] zookeeper.ClientCnxn: Session 0x0 for server madwx51/192.168.32.51:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-09-09 15:32:59,481 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2016-09-09 15:32:59,481 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster. 
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2422)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:138)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2432)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: master:160000x0, quorum=madwx51:2181,madwx41:2181,madwx31:2181, baseZNode=/hbase Unexpected KeeperException creating base node
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:206)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:187)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:585)
	at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:379)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2415)
	... 5 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:575)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:554)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1204)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1182)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:194)
	... 13 more

一看，大概意思就是连不上zookeeper，但是检查zookeeper服务一切都正常，但是当查看到zookeeper的日志文件时候发现报too many connections from  host - max is 10 的错误，明显是hbase连接zookeeper的连接数太多了，所以在zookeeper的zoo.cfg配置文件中加上如下配置：
maxClientCnxns=300
然后重启zookeeper，hbase问题解决。


maxClientCnxns=300
默认zookeeper给每个客户端IP使用的连接数为10个，经常会出现连接不够用的情况。修改连接数目前好像只支持zoo.cfg配置文件修改，所以需要zookeeper重启才能生效。
zoo.cfg:
maxClientCnxns=300
否则报错如下：2011-10-28 09:39:44,856 – WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5858:NIOServerCnxn$Factory@253] – Too many connections from /192.168.*.* – max is 10