com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

背景

服务部署上线启动完成后调用数据库时报错，异常信息如下：

12:34:09.975 [DubboServerHandler--thread-10] ERROR com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Exception during pool initialization.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_231]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_231]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_231]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_231]
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2188) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2221) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2016) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:776) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_231]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_231]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_231]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_231]
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl.getInstance$original$9yKxqIJl(ConnectionImpl.java:386) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl.getInstance$original$9yKxqIJl$accessor$UpOPm7Ty(ConnectionImpl.java) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.ConnectionImpl$auxiliary$iVcojcKH.call(Unknown Source) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:89) ~[skywalking-agent.jar:6.5.0.11]
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.NonRegisteringDriver.connect$original$NhjNAWan(NonRegisteringDriver.java:330) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.NonRegisteringDriver.connect$original$NhjNAWan$accessor$P9cGUeBp(NonRegisteringDriver.java) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.NonRegisteringDriver$auxiliary$iSNhNzjS.call(Unknown Source) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:96) ~[skywalking-agent.jar:6.5.0.11]
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.Driver.connect$accessor$V88PeGez(Driver.java) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.Driver$auxiliary$glNVB9xh.call(Unknown Source) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:96) ~[skywalking-agent.jar:6.5.0.11]
	at com.mysql.jdbc.Driver.connect(Driver.java) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:117) ~[HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:123) ~[HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:365) ~[HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:194) ~[HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:460) [HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:534) [HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) [HikariCP-2.7.9.jar!/:?]
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112) [HikariCP-2.7.9.jar!/:?]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:151) [spring-jdbc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:115) [spring-jdbc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:78) [spring-jdbc-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
	at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:82) [mybatis-spring-1.3.2.jar!/:1.3.2]
	at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:68) [mybatis-spring-1.3.2.jar!/:1.3.2]
	at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:338) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:84) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) [mybatis-3.4.6.jar!/:3.4.6]
	at com.github.pagehelper.PageInterceptor.intercept(PageInterceptor.java:108) [pagehelper-5.1.10.jar!/:?]
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61) [mybatis-3.4.6.jar!/:3.4.6]
	at com.sun.proxy.$Proxy260.query(Unknown Source) [?:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) [mybatis-3.4.6.jar!/:3.4.6]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_231]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_231]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_231]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_231]
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) [mybatis-spring-1.3.2.jar!/:1.3.2]
	at com.sun.proxy.$Proxy120.selectList(Unknown Source) [?:?]
	at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:230) [mybatis-spring-1.3.2.jar!/:1.3.2]
	at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:139) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:76) [mybatis-3.4.6.jar!/:3.4.6]
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59) [mybatis-3.4.6.jar!/:3.4.6]
	at com.sun.proxy.$Proxy166.findUserIdsByDeviceId(Unknown Source) [?:?]
	at com.manniu.rds.core.impl.DeviceShareDaoImpl.findUsersByDeviceId(DeviceShareDaoImpl.java:51) [classes!/:?]
	at org.apache.dubbo.common.bytecode.Wrapper33.invokeMethod(Wrapper33.java) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:47) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:84) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.config.invoker.DelegateProviderMetaDataInvoker.invoke(DelegateProviderMetaDataInvoker.java:56) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:52) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke$original$NjCsa98Z(MonitorFilter.java:89) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke$original$NjCsa98Z$accessor$YlFDwOdd(MonitorFilter.java) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.monitor.support.MonitorFilter$auxiliary$nI4Wm5kr.call(Unknown Source) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:96) [skywalking-agent.jar:6.5.0.11]
	at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:44) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.ExecuteLimitFilter.invoke(ExecuteLimitFilter.java:58) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:102) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:149) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:41) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:81) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:150) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:100) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:175) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57) [dubbo-2.7.5.11-SNAPSHOT.jar!/:2.7.5.11-SNAPSHOT]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_231]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_231]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_231]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_231]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_231]
	at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_231]
	at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301) ~[mysql-connector-java-5.1.46.jar!/:5.1.46]
	... 95 more

问题分析

Stack Overflow 有人总结了 jdbc 连接失败的可能情况及解决方案（https://stackoverflow.com/questions/2983248/com-mysql-jdbc-exceptions-jdbc4-communicationsexception-communications-link-fai）：

如果您使用SQLException: Connection refused或Connection timed out或MySQL专用的CommunicationsException:
Communications link failure，则意味着该数据库根本无法访问。可能有以下一种或多种原因：

1. JDBC URL中的IP地址或主机名错误。
2. 本地DNS服务器无法识别JDBC URL中的主机名。
3. JDBC URL中的端口号丢失或错误。
4. 数据库服务器已关闭。
5. DB服务器不接受TCP / IP连接。
6. 数据库服务器已用尽连接。
7. Java和DB之间的某种事物正在阻止连接，例如防火墙或代理。

要解决一个或另一个，请遵循以下建议：

1. 使用验证并测试它们ping。
2. 刷新DNS或在JDBC URL中使用IP地址。
3. 验证基于my.cnfMySQL DB。
4. 启动数据库。
5. 验证mysqld是否在不带的情况下启动–skip-networking option。
6. 重新启动数据库，并相应地修复您的代码，使其关闭中的连接finally。
7. 禁用防火墙和/或配置防火墙/代理以允许/转发端口。

由于有其他服务正在正常的访问数据库，并且查看了数据库的最大连接数（show variables like ‘%max_connections%’;）和当前连接数（SHOW STATUS like “Threads%”;
）都是正常的，超时时间设置了 30s, 所以推测可能是该 k8s pod 到数据库的网络连接可能有问题。

因此写了个简单的 socket 连接程序检验网络是否有问题：

new Thread(() -> {
                while (true) {
                    SocketAddress saAdd1 = new InetSocketAddress("www.baidu.com", 443);
                    SocketAddress saAdd2 = new InetSocketAddress("www.google.com", 443);
                    SocketAddress saAdd3 = new InetSocketAddress("my.rds.amazonaws.com", 3306);
                    try (Socket sClient1 = new Socket()) {
                        sClient1.connect(saAdd1, 1000);
                        sClient1.close();
                        LOGGER.info("www.baidu.com connect success!!!");
                    } catch (Exception e) {
                        LOGGER.error("www.baidu.com connect exception: ", e);
                    }
                    try (Socket sClient2 = new Socket()) {
                        sClient2.connect(saAdd2, 1000);
                        sClient2.close();
                        LOGGER.info("google connect success!!!");
                    } catch (Exception e) {
                        LOGGER.error("google connect exception: ", e);
                    }
                    try (Socket sClient3 = new Socket()) {
                        sClient3.connect(saAdd3, 2000);
                        sClient3.close();
                        LOGGER.info("my.rds connect success!!!");
                    } catch (Exception e) {
                        LOGGER.error("my.rds connect exception: ", e);
                    }
                    try {
                        Thread.sleep(3000L);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }).start();

部署程序，查看日志发现开始时能访问其他网站，但是访问数据库失败，过了一段时间访问数据库成功。在服务器上通过 telnet 和 netstat 命令发现，连接数据库时会时而通过内网访问，时而通过公网访问。再去查看亚马逊上面数据库的安全组配置只允许通过内网访问，所以就会造成有时候连接失败的情况。解决方案就很明确了，添加公网访问安全组配置，或者连接数据库时直接使用 IP 访问，不通过域名访问，不过这样有风险，如果数据库的 IP 变的话程序会出问题。