记一次Could not get JDBC Connection、java.lang.InterruptedException问题排查

现象：

本地自测的一次http的请求中，使用mybatis执行的sql查询，报错如下：

### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:79)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:447)
	at com.sun.proxy.$Proxy112.selectOne(Unknown Source)
	at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:167)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:75)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53)
	...
	at com.netflix.hystrix.contrib.javanica.command.MethodExecutionAction.execute(MethodExecutionAction.java:116)
	at com.netflix.hystrix.contrib.javanica.command.MethodExecutionAction.executeWithArgs(MethodExecutionAction.java:93)
	at com.netflix.hystrix.contrib.javanica.command.MethodExecutionAction.execute(MethodExecutionAction.java:78)
	...
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:150)
	...
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:434)
	... 41 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
	...
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
	... 48 common frames omitted
Caused by: java.sql.SQLException: null
	at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1153)
	at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1012)
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:992)
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:982)
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:102)
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
	... 68 common frames omitted
Caused by: java.lang.InterruptedException: null
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
	at com.alibaba.druid.pool.DruidDataSource.takeLast(DruidDataSource.java:1504)
	at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1141)
	... 74 common frames omitted
org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 

先说结果：

因为代码执行较慢，超出了hystrix的默认超时时长，导致了hystrix对当前线程执行了中断：interrupt()。

排查过程：

1、debug异常抛出的地方，发现线程有被interrupt()，因为深入了druid连接池代码，debug一通毫无头绪。

2、网络检索连接池druid的CannotGetJdbcConnectionException、InterruptedException异常

发现可能是连接池版本低，于是升级版本，升级到最新版，问题并未解决。

3、灵光一闪，想起来可能是hystrix导致的，网络检索信息，果然如此。

4、删除方法体上：@HystrixCommand，问题解决。

ps：hystrix超时中断有下面两个属性控制：

execution.isolation.thread.timeoutInMilliseconds（1秒）、execution.isolation.thread.interruptOnTimeout（true）

这一折腾就是两个多小时，能解决还是靠自己的灵光一闪，完全是浪费时间。

得失：

1、堆栈中明显有hystrix相关内容，由于堆栈较长，一开始被忽略了

at com.netflix.hystrix.contrib.javanica.command.MethodExecutionAction.execute(MethodExecutionAction.java:116)
at com.netflix.hystrix.contrib.javanica.command.MethodExecutionAction.executeWithArgs(MethodExecutionAction.java:93)
at com.netflix.hystrix.contrib.javanica.command.MethodExecutionAction.execute(MethodExecutionAction.java:78)

2、如果对hystrix不了解，不知道会引发线程中断，也不应该盲目debug druid代码，应该直接debug Thread类的interrupt()方法，这样一下就能定位到hystrix。