dubbo Thread pool is EXHAUSTED-故障排除
整个排除故障过程使用了jstack命令
故障现象描述
dubbo Thread pool is EXHAUSTED
dubbo线程池耗尽dubbo日志如下：
16:54:34,026 WARN [New I/O server worker #1-4] - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.8.64.57:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 5823 (completed: 5623), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.8.64.57:20880!, dubbo version: 2.5.3, current host: 10.8.64.57
16:54:34,027 WARN [New I/O server worker #1-4] - [DUBBO] Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.8.64.57:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 5823 (completed: 5623), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://10.8.64.57:20880!, dubbo version: 2.5.3, current host: 10.8.64.57
16:54:34,027 WARN [New I/O server worker #1-4] - [DUBBO] An exception was thrown by a user handler while handling an exception event ([id: 0x7434600d, /10.8.64.57:59102 => /10.8.64.57:20880] EXCEPTION: com.alibaba.dubbo.remoting.ExecutionException: class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process received event .), dubbo version: 2.5.3, current host: 10.8.64.57
com.alibaba.dubbo.remoting.ExecutionException: class com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler error when process caught event .
at com.alibaba.dubbo.remoting.transport.dispatcher.all.AllChannelHandler.caught(AllChannelHandler.java:67)
at com.alibaba.dubbo.remoting.transport.AbstractChannelHandlerDelegate.caught(AbstractChannelHandlerDelegate.java:44)
at com.alibaba.dubbo.remoting.transport.AbstractChannelHandlerDelegate.caught(AbstractChannelHandlerDelegate.java:44)
at com.alibaba.dubbo.remoting.transport.AbstractPeer.caught(AbstractPeer.java:127)
at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.exceptionCaught(NettyHandler.java:112)
at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.exceptionCaught(NettyCodecAdapter.java:165)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:148)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.RejectedExecutionException: Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-10.8.64.57:20880, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 5823 (com
查看jstack 历史dump文件(card-service是一个采用了Dubbo框架的服务程序)
/data/service/card-service_20161104/logs/dump/20170829171658/jstack-93064.dump
如果现在dubbo刚刚挂掉。
可以根据card-service的pid来获取jstack日志
==
获取dubbo服务card-service的jstack日志的步骤如下：
1. 去程序源码找到这个dubbo服务的端口为20880
2. 根据20880查找对应的pid，发现pid为99668
[root@NYSJHL64-57 20170829171658]# netstat -ntlp |grep 20880
tcp 0 0 0.0.0.0:20880 0.0.0.0:* LISTEN 99668/java
3. jstack 'pid' > xxxx.log
(xxx.log是你希望生成的jstack dump文件)
jstack 99668 > card-service_down.log
------
现在查看这个
jstack-93064.dump 或者xxxlog
"DubboServerHandler-10.8.64.57:20880-thread-111" daemon prio=10 tid=0x00007fd574193800 nid=0x16cf1 waiting for monitor entry [0x00007fd691887000..0x00007fd691888810]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
- waiting to lock <0x00007fd6c9fa4ba8> (a java.lang.Object)
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
at org.apache.activemq.ActiveMQSession.(ActiveMQSession.java:250)
at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:318)
at sun.reflect.GeneratedMethodAccessor222.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:550)
at $Proxy49.createSession(Unknown Source)
at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:457)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
at com.frank.searchClient.home.CardIndexerHome.createOrUpdate(CardIndexerHome.java:72)
at com.frank.card.service.provider.UserCardProvider.createOrUpdateIndex(UserCardProvider.java:616)
at com.frank.card.service.provider.UserCardProvider.updateCardNormal(UserCardProvider.java:432)
at com.frank.card.worker.UpdateCardWorker.doJob(UpdateCardWorker.java:85)
at com.frank.card.service.CardServiceImpl.updateCard(CardServiceImpl.java:300)
at com.alibaba.dubbo.common.bytecode.Wrapper15.invokeMethod(Wrapper15.java)
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.AccessLogFilter.invoke(AccessLogFilter.java:199)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:112)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619) 关键点java.lang.Thread.State: BLOCKED (on object monitor)
dubbo的线程被activemq连接的线程所阻塞BLOCKED
由此推测测试环境的activemq运行遇到性能瓶颈（比如内存耗尽，或者cpu耗尽等）
因为activemq是轻量级中间件， cpu很少耗尽。 估计是内存参数设置太小， 内存被耗尽。
查看 下。 果然
-Xmx256m -Xms256m 256M内存显然太小。 改成下面这样的
set ACTIVEMQ_OPTS=-Xms1G-Xmx1G改成1G内存， 大功告成，dubbo不再down了。
附：
jstack是jdk自带的一个命令
$JAVA_HOME/bin/jstack
例如
/opt/j2sdk/bin/jstack