线上APPARENT DEADLOCK!!! - C3P0连接池DeadLock机制分析

转载: https://www.jianshu.com/p/1a0d5129b884

APPARENT DEADLOCK

线上系统出现APPARENT DEADLOCK 报错,并导致了堆溢出问题 :

2020-01-19 08:37:40.499 WARN  [] [] com.mchange.v2.async.ThreadPoolAsynchronousRunner - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b3326ae -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 3
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@63df98b7
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqsxt2a71c4wj4wglzuwt|70619635]-HelperThread-#1
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@61dbd434
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqsxt2a71c4wj4wglzuwt|70619635]-HelperThread-#2
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1c047962
			on thread: C3P0PooledConnectionPoolManager[identityToken->1bqsxt2a71c4wj4wglzuwt|70619635]-HelperThread-#0
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@78943254
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@679107d1
		com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@58d40563
		com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@67a4a872
		com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3e7b2040
		com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@513b07d1
		com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@293cff88
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@6b1925e5
		com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@c8761e6
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@58bea28
Pool thread stack traces:
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqsxt2a71c4wj4wglzuwt|70619635]-HelperThread-#1,5,main]
		com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
		com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
		com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
		com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
		com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
		com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
		com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2194)
		com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
		com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
		com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
		com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
		sun.reflect.GeneratedConstructorAccessor3889.newInstance(Unknown Source)
		sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		java.lang.reflect.Constructor.newInstance(Constructor.java:422)
		com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
		com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
		com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
		com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
		com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
		com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqsxt2a71c4wj4wglzuwt|70619635]-HelperThread-#0,5,main]
		com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
		com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
		com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
		com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
		com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
		com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
		com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2194)
		com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
		com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
		com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
		com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
		sun.reflect.GeneratedConstructorAccessor3889.newInstance(Unknown Source)
		sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		java.lang.reflect.Constructor.newInstance(Constructor.java:422)
		com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
		com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
		com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
		com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
		com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
		com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
	Thread[C3P0PooledConnectionPoolManager[identityToken->1bqsxt2a71c4wj4wglzuwt|70619635]-HelperThread-#2,5,main]
		java.lang.ClassLoader.defineClass1(Native Method)
		java.lang.ClassLoader.defineClass(ClassLoader.java:760)
		java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
		org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2263)
		org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:799)
		org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1244)
		org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1104)
		com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3559)
		com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3459)
		com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3900)
		com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
		com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
		com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
		com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2194)
		com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2225)
		com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2024)
		com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
		com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
		sun.reflect.GeneratedConstructorAccessor3889.newInstance(Unknown Source)
		sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		java.lang.reflect.Constructor.newInstance(Constructor.java:422)
		com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
		com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
		com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
		com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
		com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
		com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
		com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
		com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
		com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

最终导致堆内存溢出 :

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:978)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.wdw.common.filter.ClientLoginFilter.doFilter(ClientLoginFilter.java:69)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:198)
	at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:651)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:499)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1376)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.xmlbeans.impl.store.Cur.createElementXobj(Cur.java:260)
	at org.apache.xmlbeans.impl.store.Cur$CurLoadContext.startElement(Cur.java:2997)
	at org.apache.xmlbeans.impl.store.Locale.loadNode(Locale.java:1420)
	at org.apache.xmlbeans.impl.store.Locale.loadNodeChildren(Locale.java:1403)

1.配置

查询百度之后基本是说解决办法是修改数据库连接池配置 ,如下为生产上的配置属性

<bean class="com.mchange.v2.c3p0.ComboPooledDataSource" id="dataSource" destroy-method="close">
        <!-- 配置连接池属性 -->
        <property name="driverClass" value="${jdbc.driver}"/>
        <property name="jdbcUrl" value="${jdbc.url}"/>
        <property name="user" value="${jdbc.username}"/>
        <property name="password" value="${jdbc.password}"/>
 
        <!-- c3p0连接池的私有属性 -->
        <property name="maxPoolSize" value="30"/>
        <property name="minPoolSize" value="10"/>
        <!-- 关闭连接后不自动commit -->
        <property name="autoCommitOnClose" value="false"/>
        <!-- 获取连接超时时间 -->
        <property name="checkoutTimeout" value="10000"/>
        <!-- 当获取连接失败重试次数 -->
        <property name="acquireRetryAttempts" value="2"/>
        <!--每60秒检查所有连接池中的空闲连接。Default: 0 -->   
        <property name="idleConnectionTestPeriod" value="60"/>
        <!--最大空闲时间,60秒内未使用则连接被丢弃。若为0则永不丢弃。Default: 0 -->   
  		<property name="maxIdleTime" value="60"/>
    </bean>

以下分析过程内容摘自 转载博客:

1.3 初步判断

根据日志，C3P0判断其线程池（ThreadPoolAsynchronousRunner）中的任务出现死锁（DEADLOCK），此时线程处于检查空闲数据链接状态是否可用。单从日志字面暂时无法判断问题所在。

查阅了网上的资料，对此问题原因分析较少，绝大部分不甚了了。比较多的文章是尝试通过调整参数来避免问题的出现的帖子。

秉着 “知其所以然” 的态度，再加上一点点的好奇心，我翻阅了一下C3P0的代码，试图从中发现端倪。

2 C3P0的死锁（DeadLock）机制分析

2.1 C3P0类结构

为了后面展示代码的需要，先附上一个我整理的C3P0的类结构图，有助于我们分析问题时确定问题的位置。如果下面的图太小，可以点击这里浏览

我们重点关注右下角的几个类。

2.2 ThreadPoolAsynchronousRunner线程池

在 C3P0PooledConnectionPoolManager 类在初始化时中创建了一个taskRunner，其类型为 ThreadPoolAsynchronousRunner ， 是一个 C3P0 自定实现的线程池。

public final class C3P0PooledConnectionPoolManager{
...
    ThreadPoolAsynchronousRunner taskRunner;
...
    private synchronized void _poolsInit()
    {
    ...
        this.taskRunner = createTaskRunner( 
                num_task_threads, matt, timer, idStr + "-HelperThread" );
    ...
    }
 ...
    private ThreadPoolAsynchronousRunner createTaskRunner( 
                int num_threads, 
                int matt /* maxAdministrativeTaskTime */, 
                Timer timer, 
                String threadLabel )
    {
    ...
        out = new ThreadPoolAsynchronousRunner( 
                      num_threads, true, timer, threadLabel );
        return out;
    ...
    }
...
}

ThreadPoolAsynchronousRunner 线程池被作为成员注入到多个对象之中，最终用于执行 BasicResourcePool 类中定义的 ScatteredAcquireTask， AcquireTask， RemoveTask， DestroyResourceTask， RefurbishCheckinResourceTask， AsyncTestIdleResourceTask 等任务。

我们来看一下 ThreadPoolAsynchronousRunner 的代码。

public final class ThreadPoolAsynchronousRunner implements AsynchronousRunner
{
...
    TimerTask deadlockDetector = new DeadlockDetector();
...
    public ThreadPoolAsynchronousRunner( 
        int num_threads, boolean daemon, 
        Timer sharedTimer, String threadLabel )
    { 
        this( num_threads, 
          daemon, 
          DFLT_MAX_INDIVIDUAL_TASK_TIME, 
          DFLT_DEADLOCK_DETECTOR_INTERVAL, 
          DFLT_INTERRUPT_DELAY_AFTER_APPARENT_DEADLOCK, 
          sharedTimer, 
          false,
          threadLabel ); 
    }
...
    private ThreadPoolAsynchronousRunner( int num_threads, 
                      boolean daemon, 
                      int max_individual_task_time,
                      int deadlock_detector_interval, 
                      int interrupt_delay_after_apparent_deadlock,
                      Timer myTimer,
                      boolean should_cancel_timer,
                      String threadLabel )
    {
        this.num_threads = num_threads;
        this.daemon = daemon;
        this.max_individual_task_time = max_individual_task_time;
        this.deadlock_detector_interval = 
                deadlock_detector_interval;
        this.interrupt_delay_after_apparent_deadlock = 
                interrupt_delay_after_apparent_deadlock;
        this.myTimer = myTimer;
        this.should_cancel_timer = should_cancel_timer;
        this.threadLabel = threadLabel;

        recreateThreadsAndTasks();

        myTimer.schedule( deadlockDetector, 
                deadlock_detector_interval, 
                deadlock_detector_interval );
    }
...
}

可以看到， ThreadPoolAsynchronousRunner 线程池默认创建3个执行线程，并且创建了一个 死锁检测（DeadLockDetector） 的定时任务，默认情况下每10秒执行一次。

2.3 DeadLockDetector内部机制

public final class ThreadPoolAsynchronousRunner implements AsynchronousRunner
{
...
    class DeadlockDetector extends TimerTask
    {
        LinkedList last = null;
        LinkedList current = null;

        public void run()
        {
        ...
            current = (LinkedList) pendingTasks.clone();
        ...
            if ( current.equals( last ) )
            {
                //System.err.println(this + " -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!");
                ...
                recreateThreadsAndTasks();
                run_stray_tasks = true;
            }
                ...
                if (run_stray_tasks)
            {
                AsynchronousRunner ar = new ThreadPerTaskAsynchronousRunner( DFLT_MAX_EMERGENCY_THREADS, max_individual_task_time );
                for ( Iterator ii = current.iterator(); ii.hasNext(); )
                    ar.postRunnable( (Runnable) ii.next() );
                ar.close( false ); //tell the emergency runner to close itself when its tasks are complete
                last = null;
            }
            else
                last = current;

...

可以看到，DeadLockDetector是
ThreadPoolAsynchronousRunner 的内部类，是一个默认每10秒执行一次的定时任务。其工作原理是定时检查任务队列（pendingTasks）中的任务，如果前后两次检查发现待执行的任务没有变化，就认为可能产生了死锁，并额外创建线程执行等待任务。

3 问题分析

3.1 线程等待

从代码中可以看出，APPARENT DEADLOCK!!!报错原因主要是ThreadPoolAsynchronousRunner线程池的中的等待任务队列在经过一定时长（默认10秒）前后没有变化。

结合问题日志。我们可以初步判断问题的原因，是由于某些任务（如日志中显示AsyncTestIdleResourceTask）执行过程中失去响应（超过60s没有收到响应），导致后续的任务长时间等待，进而报错。

3.2 具体分析

进而观察执行线程的堆栈，可以发现三个任务都在执行检测连接有效性时，向服务端发送请求数据后，Oracle数据库未返回响应，导致线程长时间等待。

由于此问题无法通过构造场景重现，只能猜测，产生这种原因可能是连接在服务端已经超时，由服务端主动关闭连接，但由于TCP协议断开连接需要完成4次握手，而服务端只完成了前两次握手，这样就导致了客户端可以发送消息但接收不到任何内容，长时间等待直到超时。

3.3 解决方案

根据以上分析，我认为问题是maxIdleTime（连接最大空闲时间）与idleConnectionTestPeriod（空闲检测周期）参数配置与数据库不匹配引起的。

根据DBA的建议，将maxIdleTime调整为30秒，将idleConnectionTestPeriod调整为10秒。

4 后续

调整完后系统稳定运行，至今未再报出同样的错误。

在查看资料的过程中，发现有与我类似的情况，请看这里。问题出在同一块代码，但现象与原因均与我不同。我并不认为这是C3P0的缺陷，而是具体环境上的配置没有相匹配。

我再说一点废话，C3P0中大量使用了synchronized关键字进行加锁，由于单例没有使用double-check，导致正常代码中大量加锁，在高并发下效率很低。因此，如果性能对系统较为重要，还是推荐大家使用HikariCP替换C3P0。

PS: 后面导致的堆内存溢出是否为客户端不断发送请求产生大量内存占用造成?
PS: 除了配置问题以外,连接超时原因也应该分析,是大量连接,还是数据库处在高负载无法使用等?
PS: 我有追根溯源思想,但是又不愿看源码,只能抄抄笔记勉强度日这样子