一,背景pc端查询变慢,最后查询失败
二,查看服务器日志,报错如下
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 30, maxActive 30, creating 0, runningSqlCount 1 : SELECT ti_dp_all_feedstate_id, unique_code, command_field, delivery_no, vin, transporting_method, leaving_warehouse, arriving_warehouse, convey_code, memo, dispatch_order_no, carrier, motorcade, driver, transport_supplement, effective_time, feedstate_time, actual_leaving_time, sender, receiver, interface_status, original_spot_name, original_code, destination_spot_name, destination_code, retailer_code, retailer_name, dispatch_take_effect_time, plan_no, load_type, load_number, current_status, original_spot_description, destination_spot_description, planning_leaving_time, planning_arriving_time, model, model_desc, spare_01, spare_02, spare_03, spare_04, spare_05, remark, create_by, create_date, update_by, update_date, version
,final_transporting_method
看下数据库连接配置
##### Primary数据库连接池配置 - MySql数据库
spring.datasource.mall.druid.filters = stat
spring.datasource.mall.druid.maxActive = 30
spring.datasource.mall.druid.minIdle = 5
spring.datasource.mall.druid.initialSize = 5
spring.datasource.mall.druid.maxWait = 60000
spring.datasource.mall.druid.timeBetweenEvictionRunsMillis = 60000
spring.datasource.mall.druid.minEvictableIdleTimeMillis = 300000
spring.datasource.mall.druid.poolPreparedStatements = false
可以看出maxActive = 30,active = 30,可以看出当前查询无法获取数据库连接,导致查询失败。
三,原因分析
是哪里占用了大量的数据库连接
猜测:存在大量慢查询,无法释放,导致这里无法获取
skywalking分析
如上图,skywalking中,首先出现大量慢查询,红色报错应该也是没有获取数据库连接
其实OrderAlarmWithParamsJobHandler这个请求出现多次,查看代码发现,这是一个job,
且存在循环中,查询,当时数据大概有40条,也就是说查询了40次,最大连接数30,问题应该在这里。
但是有一个问题
无法获取数据库连接时间,开始时间14:51,结束时间:15:30,直到重启服务
持续近40min,按理说之前咱用比较多连接的job,已经执行结束。还是哪里占用了?
四,解决方案
1.优化skywalking中慢查询sql
2.优化循环中查询