关于java.sql.SQLRecoverableException: Closed Connection异常的解决方案
在项目中碰到了一个应用异常，从表象来看应用僵死。查看Weblogic状态为Running,内存无溢出，但是出现多次线程堵塞。查看Weblogic日志，发现程序出现多次Time Out。
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "640" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 640083 ms ", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) oracle.net.ns.Packet.receive(Packet.java:300) oracle.net.ns.DataPacket.receive(DataPacket.java:106) oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315) oracle.net.ns.NetInputStream.read(NetInputStream.java:260) oracle.net.ns.NetInputStream.read(NetInputStream.java:185) oracle.net.ns.NetInputStream.read(NetInputStream.java:102) oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124) oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80) oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137) oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290) oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192) oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:531) oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207) oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:884) oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1167) oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289) oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3593) oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3637) oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1495) com.trs.infra.persistent.BaseObjs.loadPage(BaseObjs.java:615) com.trs.infra.persistent.BaseObjs.ensurePageLoaded(BaseObjs.java:552) com.trs.infra.persistent.BaseObjs.getAt(BaseObjs.java:1500) com.trs.infra.support.log.LogServer.initOperTypeMap(LogServer.java:570) com.trs.infra.support.log.LogServer.getOperTypeAsInt(LogServer.java:105) com.trs.infra.support.log.LogServer.save(LogServer.java:81) com.trs.infra.support.log.LogServer.startRecord(LogServer.java:630) com.trs.infra.support.log.LogServer.startRecord(LogServer.java:646) com.trs.presentation.util.LoginHelper.login(LoginHelper.java:342) jsp_servlet._app.__login_dowith._jspService(__login_dowith.java:446) weblogic.servlet.jsp.JspBase.service(JspBase.java:34) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338) weblogic.servlet.internal.ServletStubImpl.onAddToMapException(ServletStubImpl.java:453) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:364) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) com.trs.deployer.domain.filter.DeployerControllerFilter.doFilter(DeployerControllerFilter.java:88) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) com.trs.servlet.AclFilter.doFilter(AclFilter.java:54) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) com.trs.servlet.LoginCheckFilter.doFilter(LoginCheckFilter.java:126) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) com.trs.cluster.ext.wcm.ClusterProxyFilter.doFilter(ClusterProxyFilter.java:65) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) com.trs.infra.session.util.SessionFilterBase.doFilter(SessionFilterBase.java:128) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3296) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3262) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57) weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2171) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2097) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2075) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1514) weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254) weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) weblogic.work.ExecuteThread.run(ExecuteThread.java:221)我们知道，Weblogic会自动检测线程运行超时，当超过特点时间（默认600S），即认为此线程为堵塞线程。在日志中发现多次堵塞线程，通过查找资料，发现Weblogic在发生多次线程堵塞后，会自动把应用挂起。默认次数为15次。
是什么造成了线程堵塞呢？通过进一步分析日志，我们发现在线程堵塞之前，发生了多次java.sql.SQLRecoverableException: Closed Connection异常。异常情况：
java.sql.SQLRecoverableException: Closed Connection at oracle.jdbc.driver.PhysicalConnection.needLine(PhysicalConnection.java:5389) at oracle.jdbc.driver.OracleStatement.closeOrCache(OracleStatement.java:1578) at oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:1563) at oracle.jdbc.driver.OracleStatementWrapper.close(OracleStatementWrapper.java:94) at oracle.jdbc.driver.OraclePreparedStatementWrapper.close(OraclePreparedStatementWrapper.java:82) at com.trs.infra.persistent.BaseObj.update(BaseObj.java:1733) at com.trs.cms.content.CMSObj.update(CMSObj.java:402) at com.trs.cms.auth.persistent.User.update(User.java:798) at com.trs.cms.auth.persistent.User.login(User.java:890) at com.trs.presentation.util.LoginHelper.login(LoginHelper.java:371) at jsp_servlet._app.__login_dowith._jspService(__login_dowith.java:446) at weblogic.servlet.jsp.JspBase.service(JspBase.java:34) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.trs.deployer.domain.filter.DeployerControllerFilter.doFilter(DeployerControllerFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.trs.servlet.AclFilter.doFilter(AclFilter.java:54) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.trs.servlet.LoginCheckFilter.doFilter(LoginCheckFilter.java:126) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.trs.cluster.ext.wcm.ClusterProxyFilter.doFilter(ClusterProxyFilter.java:65) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.trs.infra.session.util.SessionFilterBase.doFilter(SessionFilterBase.java:128) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3296) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3262) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2171) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2097) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2075) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1514) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)从表现来看是数据库连接出了异常。我们对数据库和网络进行了分析，确定数据库和网络都无异常。我们的另外一个应用在Weblogic运行没有类似问题。
最后在Oracle的论坛上找到了问题的根结，由于我们的应用是自己开发的数据库连接池，应用和数据库之间有一层防火墙。防火墙策略是对于1800s未使用的Socket连接将自动关闭。Oracle的日志中也发现Socket异常关闭的异常。我们对应用进行了调整，当连接池中的连接15分钟不用时，自动回收，问题解决。