zookpeer客户端狂刷KeeperErrorCode = ConnectionLoss问题

问题是这样的,在测试环境平时都可以正常使用zookeeper,但是每次准备上版本之前有频繁的测试。则久不久就会出现日志刷屏的情况。 使用的zk客户端框架是Curator,项目里面kafka/dubbo/elastic-job都使用到zk。千丝万缕而且问题又不能随时重现,所以排查了很久一直找不出问题原因。

日志如下:

[2016-12-07 21:11:04,435] [525133fa-24f2-44bf-beab-58d6ff36b9ea] [main-EventThread] [INFO] [ConnectionStateManager.java:228] State change: SUSPENDED

[2016-12-07 21:11:04,435] [5c4a12ce-7e81-435f-80c7-420996812c78] [main-EventThread] [WARN] [ConnectionStateManager.java:235] ConnectionStateManager queue full - dropping events to make room

[2016-12-07 21:11:05,015] [c6425e13-048a-486d-a9f1-211566fc8221] [Curator-Framework-0] [INFO] [ConnectionStateManager.java:228] State change: LOST

[2016-12-07 21:11:05,015] [7bc4a190-6f0f-4206-b6b9-9a02c793619d] [Curator-Framework-0] [WARN] [ConnectionStateManager.java:235] ConnectionStateManager queue full - dropping events to make room

[2016-12-07 21:11:05,017] [54fcf829-4285-4fd2-a889-b67c28e1e50a] [Curator-Framework-0] [ERROR] [CuratorFrameworkImpl.java:537] Background operation retry gave up

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:708) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) [curator-framework-2.8.0.jar:na]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]

at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

[2016-12-07 21:11:05,019] [6ced4c75-20cf-4fdc-a90d-ac303d6abc11] [Curator-Framework-0] [ERROR] [CuratorFrameworkImpl.java:537] Background retry gave up

org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss

at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:809) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) [curator-framework-2.8.0.jar:na]

at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) [curator-framework-2.8.0.jar:na]

at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]

at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

经过查看依赖的zk/zk客户端有多个不同的版本,而且在其他地方看到了这个博客(http://blog.csdn.net/azhao_dn/article/details/8469680)所以就一一的在pom中排掉低版本的zk和zk客户端(Curator)。但是后来问题依然重现,没法发只能看源码了,翻看了下Curator的源码发现出现KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss因为初始化zk的等待连接的时间使用的是默认的15秒

zookpeer客户端狂刷KeeperErrorCode = ConnectionLoss问题_第1张图片


所以在初始话的操作中将这个时间稍微再加大一些。但是后来问题依然重现无奈之下只能再翻看代码。

CuratorFrameworkImpl.java

private void backgroundOperationsLoop()

{

while ( !Thread.currentThread().isInterrupted() )

{

OperationAndData operationAndData;

try

{

operationAndData = backgroundOperations.take();

if ( debugListener != null )

{

debugListener.listen(operationAndData);

}

}

catch ( InterruptedException e )

{

Thread.currentThread().interrupt();

break;

}

performBackgroundOperation(operationAndData);

}

}

private void performBackgroundOperation(OperationAndData operationAndData)

{

try

{

if ( client.isConnected() )

{

operationAndData.callPerformBackgroundOperation();

}

else

{

client.getZooKeeper();  // important - allow connection resets, timeouts, etc. to occur

if ( operationAndData.getElapsedTimeMs() >= client.getConnectionTimeoutMs() )

{

throw new CuratorConnectionLossException();

}

operationAndData.sleepFor(1, TimeUnit.SECONDS);

queueOperation(operationAndData);

}

}

catch ( Throwable e )

{

/**

* Fix edge case reported as CURATOR-52. ConnectionState.checkTimeouts() throws KeeperException.ConnectionLossException

* when the initial (or previously failed) connection cannot be re-established. This needs to be run through the retry policy

* and callbacks need to get invoked, etc.

*/

if ( e instanceof CuratorConnectionLossException )

{

WatchedEvent watchedEvent = new WatchedEvent(Watcher.Event.EventType.None, Watcher.Event.KeeperState.Disconnected, null);

CuratorEvent event = new CuratorEventImpl(this, CuratorEventType.WATCHED, KeeperException.Code.CONNECTIONLOSS.intValue(), null, null, operationAndData.getContext(), null, null, null, watchedEvent, null);

if ( checkBackgroundRetry(operationAndData, event) )

{

queueOperation(operationAndData);

}

else

{

logError("Background retry gave up", e);

}

}

else

{

handleBackgroundOperationException(operationAndData, e);

}

}

}

OperationAndData.java

longgetElapsedTimeMs()

{

returnSystem.currentTimeMillis() -startTimeMs;

}

DelayQueue.java:

public E take() throws InterruptedException {

final ReentrantLock lock = this.lock;

lock.lockInterruptibly();

try {

for (;;) {

E first = q.peek();//队列中为空,firsr值为Null

if (first == null)

available.await();

else {

long delay = first.getDelay(NANOSECONDS);

if (delay <= 0)

return q.poll();

first = null; // don't retain ref while waiting

if (leader != null)

available.await();

else {

Thread thisThread = Thread.currentThread();

leader = thisThread;

try {

available.awaitNanos(delay);

} finally {

if (leader == thisThread)

leader = null;

}

}

}

}

} finally {

if (leader == null && q.peek() != null)

available.signal();

lock.unlock();

}

}

看到这里,我猜测应该是从队列中获取到的元素为空,所以程序block在这里导致超时。所以问题就转变成为什么优先级队列(PriorityQueue)为空。导致获取不到元素,程序block

你可能感兴趣的:(zookpeer客户端狂刷KeeperErrorCode = ConnectionLoss问题)