并发过大druid异常：Thread pool is EXHAUSTED!

项目源码地址：https://gitee.com/lilyssh/high-concurrency

当并发量太高，就会导致部分请求调用服务失败，我们就需要定位瓶颈在哪里，再对症下药，本篇就是为了说明线程池资源耗尽的情况下该怎么办。

1. 用jmeter压测

2. 异常描述

运行，会看到部分请求失败：

在项目控制台会看到报错信息：threadpool is exhausted，线程池耗尽。

2018-09-14 11:04:30.365 [http-nio-1111-exec-435] ERROR cn.lilyssh.common.exception.ExceptionAdviceHandler  - 处理出现异常
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method save in the service cn.lilyssh.order.api.service.OrderServiceApi. Tried 3 times of the providers [192.168.31.31:3333] (1/1) from the registry ssh.qianxunclub.com:2181 on the consumer 192.168.31.31 using the dubbo version 2.6.0. Last error is: Failed to invoke remote method: save, provider: dubbo://192.168.31.31:3333/cn.lilyssh.order.api.service.OrderServiceApi?anyhost=true&application=order-consumer&check=false&default.check=false&dubbo=2.6.0&generic=false&interface=cn.lilyssh.order.api.service.OrderServiceApi&methods=save,list&pid=2372&register.ip=192.168.31.31&remote.timestamp=1536893088718&side=consumer&timestamp=1536893890966, cause: Server side(192.168.31.31,3333) threadpool is exhausted ,detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.31.31:3333, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 4395 (completed: 4195), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.31.31:3333!
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: com.alibaba.dubbo.remoting.RemotingException: Server side(192.168.31.31,3333) threadpool is exhausted ,detail msg:Thread pool is EXHAUSTED! Thread Name: DubboServerHandler-192.168.31.31:3333, Pool Size: 200 (active: 200, core: 200, max: 200, largest: 200), Task: 4395 (completed: 4195), Executor status:(isShutdown:false, isTerminated:false, isTerminating:false), in dubbo://192.168.31.31:3333!
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:218)
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
	... 69 common frames omitted

3. 异常分析

从异常的信息看，有两种可能：

1. dubbo线程池耗尽
2. druid连接池耗尽
   先增加dubbo线程池的数量，默认为200。改大后又报超时的错误：

2018-09-14 15:04:45.223 [http-nio-1111-exec-194] ERROR cn.lilyssh.common.exception.ExceptionAdviceHandler  - 处理出现异常
com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method exist in the service cn.lilyssh.user.api.service.UserServiceApi. Tried 3 times of the providers [192.168.31.31:6666] (1/1) from the registry ssh.qianxunclub.com:2181 on the consumer 192.168.31.31 using the dubbo version 2.6.0. Last error is: Invoke remote method timeout. method: exist, provider: dubbo://192.168.31.31:6666/cn.lilyssh.user.api.service.UserServiceApi?anyhost=true&application=order-consumer&check=false&default.check=false&dubbo=2.6.0&generic=false&interface=cn.lilyssh.user.api.service.UserServiceApi&methods=exist,list&pid=3307&register.ip=192.168.31.31&remote.timestamp=1536908653179&side=consumer&timestamp=1536908663628, cause: Waiting server-side response timeout. start time: 2018-09-14 15:04:43.837, end time: 2018-09-14 15:04:44.860, client elapsed: 1 ms, server elapsed: 1022 ms, timeout: 1000 ms, request: Request [id=590, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=exist, parameterTypes=[class java.lang.Integer], arguments=[4], attachments={path=cn.lilyssh.user.api.service.UserServiceApi, interface=cn.lilyssh.user.api.service.UserServiceApi, version=0.0.0}]], channel: /192.168.31.31:61708 -> /192.168.31.31:6666
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)
	at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: com.alibaba.dubbo.remoting.TimeoutException: Waiting server-side response timeout. start time: 2018-09-14 15:04:43.837, end time: 2018-09-14 15:04:44.860, client elapsed: 1 ms, server elapsed: 1022 ms, timeout: 1000 ms, request: Request [id=590, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=exist, parameterTypes=[class java.lang.Integer], arguments=[4], attachments={path=cn.lilyssh.user.api.service.UserServiceApi, interface=cn.lilyssh.user.api.service.UserServiceApi, version=0.0.0}]], channel: /192.168.31.31:61708 -> /192.168.31.31:6666
	at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:134)
	at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)
	... 69 common frames omitted

把timeout默认为1秒，改为5秒：

spring:
  dubbo:
    registry:
      address: zookeeper://ssh.qianxunclub.com:2181
    provider:
      port: 6666
#      threads: 1000
      timeout: 5000
#    protocol:
#      threads: 1000

依旧报错：threadpool is exhausted.
druid最大连接数max-active默认为8，改为20：

spring:
  datasource:
    druid:
      url: jdbc:mysql://db.qianxunclub.com:3306/demo
      username: xxx
      password: xxx
      max-active: 20

继续测试，jmeter报错：

java.net.NoRouteToHostException: Can't assign requested address (Address not available)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:121)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
	at org.apache.jmeter.protocol.http.sampler.hc.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:318)
	at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.open(MeasuringConnectionManager.java:114)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:697)
	at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:455)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1189)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1178)
	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:490)
	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:416)
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:250)
	at java.lang.Thread.run(Thread.java:748)

修改了jmeter的implementation参数设置，就解决了报错的问题。修改如下：

4. 解决办法

druid最大连接数max-active默认为8，改为20：

spring:
  datasource:
    druid:
      url: xxx
      username: xxx
      password: xxx
      max-active: 20

大功告成！