记一次应用配置的数据库连接被打满问题
线上应用，配置的数据库连接数为50，正常情况是已经够用了，但是有天发现50个连接全部被占满，并且长时间无法恢复，重启服务后会好一段时间。
1、问题现象
Druid获取MySQL数据库连接超时，超时时间设置的为60s
2019-09-23 15:38:24.627-|XNIO-3 task-80-|ERROR-|7fba6a1c917f42878b43f937df37e575-|10.123.18.241-|-|/api/v1/qimao/admin/distributor/add-|-|Java/1.8.0_181-|c.q.k.q.web.controller.admin.DistributorController-|method:addDistributor
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 10, maxActive 10, creating 0
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:289)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.qiyi.kpp.qimao.web.service.impl.DistributorServiceImpl$$EnhancerBySpringCGLIB$$e95f0879.addDistributor(<generated>)
at com.qiyi.kpp.qimao.web.controller.admin.DistributorController.addDistributor(DistributorController.java:174)
at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$FastClassBySpringCGLIB$$9c955533.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$EnhancerBySpringCGLIB$$100c45af.addDistributor(<generated>)
at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
.............
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 10, maxActive 10, creating 0
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1512)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1255)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5007)
at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:680)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5003)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1233)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1225)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:90)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
... 100 common frames omitted
2、解决办法
添加配置，当连接超过30秒钟后会强制进行回收
<!-- 超过时间限制是否回收 -->
druid.removeAbandoned=true
<!-- 超时时间；单位为秒。 -->
druid.removeAbandonedTimeout=30
<!-- 关闭abanded连接时输出错误日志 -->
druid.logAbandoned=true
最终重现了这个问题，打印日志如下
2019-09-27 16:07:21.369-|AsyncResolver-bootstrap-executor-0-|INFO-|-|-|-|-|-|-|c.n.d.shared.resolver.aws.ConfigClusterResolver-|Resolving eureka endpoints via configuration
2019-09-27 16:07:28.954-|Druid-ConnectionPool-Destroy-786728464-|ERROR-|-|-|-|-|-|-|com.alibaba.druid.pool.DruidDataSource-|abandon connection, owner thread: XNIO-3 task-862, connected at : 1569571561703, open stackTrace
at java.lang.Thread.getStackTrace(Thread.java:1552)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1313)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5007)
at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:680)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5003)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1233)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1225)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:90)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:246)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:447)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:277)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.qiyi.kpp.qimao.web.service.impl.DistributorServiceImpl$$EnhancerBySpringCGLIB$$ec30275b.addDistributor(<generated>)
at com.qiyi.kpp.qimao.web.controller.admin.DistributorController.addDistributor(DistributorController.java:174)
at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$FastClassBySpringCGLIB$$9c955533.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656)
at com.qiyi.kpp.qimao.web.controller.admin.DistributorController$$EnhancerBySpringCGLIB$$34df12db.addDistributor(<generated>)
运行程序，当连接超过30秒钟后会强制进行回收，并输出异常日志。再根据异常日志分析（看看owner thread在干啥，为什么一直占据数据库连接）即可