Mysql空闲连接超时自动断开问题记录

Mysql数据库空闲连接默认8小时后会自动断开连接，此时由于业务处理使用C3P0连接池，业务在一个长时间休眠恢复后(其实是一个每天的定时任务)出现mysql 读写socket异常。异常栈信息如下：

com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException
MESSAGE: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.

STACKTRACE:

java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1997)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
    at com.mysql.jdbc.Connection.execSQL(Connection.java:3283)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1332)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1467)
    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418)
    at com.temp.datamigration.tool.DatasourceUtil.executMysqlQuery(DatasourceUtil.java:188)
    at com.temp.datamigration.processor.recordprocessor.datacloud.UserSelfDefineProcessor.initUserSelfDefineSdsMode(UserSelfDefineProcessor.java:106)
    at com.temp.datamigration.processor.recordprocessor.datacloud.UserSelfDefineProcessor.init(UserSelfDefineProcessor.java:79)
    at com.temp.datamigration.processor.WearProcessor.initDataCloudProfileProcessors(WearProcessor.java:176)
    at com.temp.datamigration.processor.WearProcessor.init(WearProcessor.java:156)
    at com.temp.datamigration.bootstrap.Bootstrap.startMerge(Bootstrap.java:155)
    at com.temp.datamigration.bootstrap.Bootstrap.access$000(Bootstrap.java:43)
    at com.temp.datamigration.bootstrap.Bootstrap$1.run(Bootstrap.java:125)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

或者如下

com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:2744)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1612)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)
        at com.mysql.jdbc.Connection.execSQL(Connection.java:3283)
        at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1332)
        at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1467)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:1418)
        at com.temp.datamigration.tool.DatasourceUtil.executMysqlQuery(DatasourceUtil.java:188)
        at com.temp.datamigration.processor.WearProcessor.obtailUserToBeMerged(WearProcessor.java:100)
        at com.temp.datamigration.processor.WearProcessor.init(WearProcessor.java:131)
        at com.temp.datamigration.bootstrap.Bootstrap.startMerge(Bootstrap.java:156)
        at com.temp.datamigration.bootstrap.Bootstrap.access$000(Bootstrap.java:43)
        at com.temp.datamigration.bootstrap.Bootstrap$1.run(Bootstrap.java:126)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

mysql server gone away
mysql wait_timeout变量说明
C3P0连接可用测试配置说明

下面是我们代码中初始化C3P0 DataSource的代码

ComboPooledDataSource dataSource = new ComboPooledDataSource();
dataSource.setDriverClass("com.mysql.jdbc.Driver");
dataSource.setJdbcUrl(mysqlConfig.getJdbcUrl());
dataSource.setUser(mysqlConfig.getUsername());
dataSource.setPassword(mysqlConfig.getPasswd());
dataSource.setMinPoolSize(5);
dataSource.setAcquireIncrement(5);
dataSource.setMaxPoolSize(maxPoolSize);
dataSource.setInitialPoolSize(2);
dataSource.setAcquireRetryDelay(400);

即未设置连接任何Connection Testing 配置，都使用默认值
preferredTestQuery 默认值null，没有校验连接可用性的测试语句
testConnectionOnCheckin 默认值false，创建连接时不会校验连接可用性
testConnectionOnCheckout 默认值false，从连接池borrow时不会校验连接可用性
idleConnectionTestPeriod 默认值为0，即不进行空闲测试逐出处理。

而连接池的minPoolSize却被设置为5，连接池默认最小有5个空闲连接。尼玛，这5个连接在长时间(8小时)后被mysql server断连了。

修改方案1：修改mysql wait_timeout值超过24小时。但不能从根本上解决问题，而且一般现网的mysql不能随意修改，该方案不可行。

修改方案2：配置C3P0的Connection Testing ，由连接池自行校验连接可用性。

dataSource.setPreferredTestQuery("select 1");
dataSource.setTestConnectionOnCheckout(true);
dataSource.setTestConnectionOnCheckin(true);

测试验证：
1)set global wait_timeout=30;—-将mysql的wait_timeout全局变量设置为30s。
2)将业务(定时读取Mysql的任务)定时周期挑战为1分钟。
即可复现和验证该问题。

而现网的Tomcat业务服务器未出现过类似问题，而我们的Tomcat业务服务使用Tomcat dbcp数据库连接池中间件，并设置了validationQuery为”select 1”，但未配置其他Connection Testing。
查看DBCP配置参数说明
testOnBorrow、testOnConnect、testOnReturn、testWhileIdle的默认值也都是False，按道理也应该会存在连接断连的问题的啊。
查看源码发现 org.apache.tomcat.dbcp.dbcp.BasicDataSource.java
testOnBorrow默认值是true。。。。和文档不符。。。