问题是这样的，在测试环境平时都可以正常使用zookeeper，但是每次准备上版本之前有频繁的测试。则久不久就会出现日志刷屏的情况。 使用的zk客户端框架是Curator，项目里面kafka/dubbo/elastic-job都使用到zk。千丝万缕而且问题又不能随时重现，所以排查了很久一直找不出问题原因。
日志如下：
[2016-12-07 21:11:04,435] [525133fa-24f2-44bf-beab-58d6ff36b9ea] [main-EventThread] [INFO] [ConnectionStateManager.java:228] State change: SUSPENDED
[2016-12-07 21:11:04,435] [5c4a12ce-7e81-435f-80c7-420996812c78] [main-EventThread] [WARN] [ConnectionStateManager.java:235] ConnectionStateManager queue full - dropping events to make room
[2016-12-07 21:11:05,015] [c6425e13-048a-486d-a9f1-211566fc8221] [Curator-Framework-0] [INFO] [ConnectionStateManager.java:228] State change: LOST
[2016-12-07 21:11:05,015] [7bc4a190-6f0f-4206-b6b9-9a02c793619d] [Curator-Framework-0] [WARN] [ConnectionStateManager.java:235] ConnectionStateManager queue full - dropping events to make room
[2016-12-07 21:11:05,017] [54fcf829-4285-4fd2-a889-b67c28e1e50a] [Curator-Framework-0] [ERROR] [CuratorFrameworkImpl.java:537] Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:708) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:826) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) [curator-framework-2.8.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
[2016-12-07 21:11:05,019] [6ced4c75-20cf-4fdc-a90d-ac303d6abc11] [Curator-Framework-0] [ERROR] [CuratorFrameworkImpl.java:537] Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:809) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:792) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62) [curator-framework-2.8.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257) [curator-framework-2.8.0.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
经过查看依赖的zk/zk客户端有多个不同的版本，而且在其他地方看到了这个博客(http://blog.csdn.net/azhao_dn/article/details/8469680)所以就一一的在pom中排掉低版本的zk和zk客户端(Curator)。但是后来问题依然重现，没法发只能看源码了，翻看了下Curator的源码发现出现KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss因为初始化zk的等待连接的时间使用的是默认的15秒
所以在初始话的操作中将这个时间稍微再加大一些。但是后来问题依然重现无奈之下只能再翻看代码。
CuratorFrameworkImpl.java
private void backgroundOperationsLoop()
{
while ( !Thread.currentThread().isInterrupted() )
{
OperationAndData operationAndData;
try
{
operationAndData = backgroundOperations.take();
if ( debugListener != null )
{
debugListener.listen(operationAndData);
}
}
catch ( InterruptedException e )
{
Thread.currentThread().interrupt();
break;
}
performBackgroundOperation(operationAndData);
}
}
private void performBackgroundOperation(OperationAndData operationAndData)
{
try
{
if ( client.isConnected() )
{
operationAndData.callPerformBackgroundOperation();
}
else
{
client.getZooKeeper(); // important - allow connection resets, timeouts, etc. to occur
if ( operationAndData.getElapsedTimeMs() >= client.getConnectionTimeoutMs() )
{
throw new CuratorConnectionLossException();
}
operationAndData.sleepFor(1, TimeUnit.SECONDS);
queueOperation(operationAndData);
}
}
catch ( Throwable e )
{
/**
* Fix edge case reported as CURATOR-52. ConnectionState.checkTimeouts() throws KeeperException.ConnectionLossException
* when the initial (or previously failed) connection cannot be re-established. This needs to be run through the retry policy
* and callbacks need to get invoked, etc.
*/
if ( e instanceof CuratorConnectionLossException )
{
WatchedEvent watchedEvent = new WatchedEvent(Watcher.Event.EventType.None, Watcher.Event.KeeperState.Disconnected, null);
CuratorEvent event = new CuratorEventImpl(this, CuratorEventType.WATCHED, KeeperException.Code.CONNECTIONLOSS.intValue(), null, null, operationAndData.getContext(), null, null, null, watchedEvent, null);
if ( checkBackgroundRetry(operationAndData, event) )
{
queueOperation(operationAndData);
}
else
{
logError("Background retry gave up", e);
}
}
else
{
handleBackgroundOperationException(operationAndData, e);
}
}
}
OperationAndData.java
longgetElapsedTimeMs()
{
returnSystem.currentTimeMillis() -startTimeMs;
}
DelayQueue.java:
public E take() throws InterruptedException {
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
for (;;) {
E first = q.peek();//队列中为空，firsr值为Null
if (first == null)
available.await();
else {
long delay = first.getDelay(NANOSECONDS);
if (delay <= 0)
return q.poll();
first = null; // don't retain ref while waiting
if (leader != null)
available.await();
else {
Thread thisThread = Thread.currentThread();
leader = thisThread;
try {
available.awaitNanos(delay);
} finally {
if (leader == thisThread)
leader = null;
}
}
}
}
} finally {
if (leader == null && q.peek() != null)
available.signal();
lock.unlock();
}
}
看到这里，我猜测应该是从队列中获取到的元素为空，所以程序block在这里导致超时。所以问题就转变成为什么优先级队列(PriorityQueue)为空。导致获取不到元素，程序block