java程序内存溢出java.lang.OutOfMemoryError: GC overhead limit exceeded问题排查

问题:java程序内存溢出java.lang.OutOfMemoryError: GC overhead limit exceeded问题

1)截取log4j相关日志:

java.net.SocketException: Socket input is already shutdown
    at java.net.Socket.shutdownInput(Socket.java:1525)
    at com.mysql.jdbc.MysqlIO.quit(MysqlIO.java:1683)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4388)
    at com.mysql.jdbc.ConnectionImpl.cleanup(ConnectionImpl.java:1368)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2654)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2570)
    at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1474)
    at com.mchange.v2.c3p0.impl.NewProxyStatement.executeQuery(NewProxyStatement.java:35)
    at com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:73)
    at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:374)
    at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
    at com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
    at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)


** END NESTED EXCEPTION **


Exception in thread "catalina-exec-63" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "catalina-exec-24" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "catalina-exec-97" java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "catalina-exec-52" java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "creditUserFeatureAsynRemoteService-pool-3" org.mybatis.spring.MyBatisSystemException: nested exception is org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
### The error may exist in sqlMap/tbCreditUserFeature-mybatis-sqlmap.xml
### The error may involve TbCreditUserFeatureMapper.MybatisTbCreditUserFeatureDAO_select_dynamic_count
### The error occurred while executing a query
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
    at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:75)
    at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:371)
    at com.sun.proxy.$Proxy25.selectOne(Unknown Source)
    at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:163)
    at mybatis.MybatisTbCreditUserFeatureDAO.selectCountDynamic(MybatisTbCreditUserFeatureDAO.java:45)
    at TbCreditUserFeatureServiceImpl.selectCountDynamic(TbCreditUserFeatureServiceImpl.java:73)
    at CreditUserFeatureBoImpl.getUserFeatureCount(CreditUserFeatureBoImpl.java:202)
    at CreditUserFeatureAsynRemoteServiceImpl$3.run(CreditUserFeatureAsynRemoteServiceImpl.java:159)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
### The error may exist in sqlMap/tbCreditUserFeature-mybatis-sqlmap.xml
### The error may involve TbCreditUserFeatureMapper.MybatisTbCreditUserFeatureDAO_select_dynamic_count
### The error occurred while executing a query

 

2)jstack输出日志:

Attaching to process ID 11048, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.74-b02
Deadlock Detection:

No deadlocks found.

Thread 12424: (state = BLOCKED)


Thread 11851: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11850: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11845: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11363: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11286: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11285: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11284: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11283: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11282: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11281: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11280: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11279: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11278: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11277: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11276: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11275: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11274: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11273: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11272: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11271: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11270: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11269: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11268: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11267: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11266: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11265: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11264: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11263: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11262: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11261: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11260: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11259: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11258: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11257: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11256: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11255: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11254: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11253: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11252: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11251: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11250: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 11249: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

 

 

线程程序:

ExecutorsUtil 类:

public class ExecutorsUtil {
    public ExecutorsUtil() {
    }

    public static ExecutorService createExecutorService(int corePoolSize, int maximumPoolSize, int queueSize, String threadName) {
        return new ThreadPoolExecutor(corePoolSize, maximumPoolSize, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue(queueSize), (new ThreadFactoryBuilder()).setNameFormat(threadName + "-pool-%d").build(), new AbortPolicy());
    }

}

ICreditUserFeatureAsynBo实现类:

public class CreditUserFeatureAsynBoImpl implement ICreditUserFeatureAsynBo{

private final static ExecutorService executorService = ExecutorsUtil.createExecutorService(1, THREAD_SIZE, 999, "creditUserFeatureAsynBo");

 

public Boolean polymerizeUser(final PagePeriod pagePeriod, final CreditRgisterUserQueryDTO requestDTO){
Integer startPage = pagePeriod.getBeginPage(),endPage = pagePeriod.getEndPage();
List>> callableList = new ArrayList>>();
while (startPage <= endPage) {
...
List userVoList = creditUserFeatureService.getRegisterUser(queryDTO);

callableList.add(new CreditUserFeatureCallable>>() {
    @Override
    public List call() {
        多线程调用ICreditUserFeatureConsumeService 
    }
});

....

List>> futureList = executorService.invokeAll(callableList);

数据库db操作

}

 

问题分析:

1.程序为三级线程 Timetask(调用发起) > ICreditUserFeatureAsynRemoteService(线程异步) > ICreditUserFeatureAsynBo(多线程并发) > CreditUserFeatureCallable (Callable实现类,共享CreditUserFeatureAsynBo中线程池)>ICreditUserFeatureConsumeService (单例服务类)> ICreditStatisticsUserService (多线程并发)

a)Timetask(调用发起):调用发起时,循环至少30次,每次sleep 20ms。

b)ICreditUserFeatureAsynRemoteService(线程异步)负责获取需处理数据页数并成每组1000页,异步hessian分发到 ICreditUserFeatureAsynRemoteService方法处理。

c)ICreditUserFeatureAsynBo(多线程并发):根据分页的组,对每页进行查询获取数据放入list,按每页一个CreditUserFeatureCallable ,一次到executorService.invokeAll中处理。

问题原因:

1.调用发起循环>30次,三级线程每级都是异步,必然会在LinkedBlockingQueue 排队。

2.ICreditUserFeatureAsynBo中根据组先查出每页数据,按组再批量给CreditUserFeatureCallable处理,导致a.提前查询数据占用内存  b.线程未能及时处理,会在LinkedBlockingQueue 垃圾回收器回收不了。

3.ICreditUserFeatureAsynBo中根据组先查出每页数据按组一次性推给多线程处理器,会导致内存占用暴增。

4.xxx-logging.xxx与xxx-common.xxx包,直接使用了 Executors.newFixedThreadPool内存溢出问题:
    private static ExecutorService fixedThreadPool = Executors.newFixedThreadPool(10, new ThreadFactory() {
        public Thread newThread(Runnable r) {
            Thread t = Executors.defaultThreadFactory().newThread(r);
            t.setDaemon(true);
            return t;
        }
    });

AppenderManager类中Executors.newFixedThreadPool代码中使用队列默认为Integer的最大值,会造成内存溢出:new LinkedBlockingQueue()

排查方法:
1.jstat -gcutil pid 看gc频次是否正常,看gc是否能回收掉垃圾; 
2.jmap -histo:live pid (这个会触发FGC) 
3.jstack pid > /data/jstack.log 看线程谁被阻塞 
4.如果是内存溢出再down(jmap -dump:live,format=b,file=/data/test-ccdbkd-10-24.bin 23548)出堆分析,同时打出线程栈。
EMA慢慢分析内存,并结合业务代码看,如果是第三方jar包的类,那么使用下载源码看; 
需要注意的是:分析问题要知道因果,一个事情发生相对于另外一个事情,是因,还是果,这个需要注意搞清楚;

5.sz /data/jstack.log 下载文件到本地
6.通过Eclipse Memory Analyzer (下载路径:http://www.eclipse.org/mat/)软件对dump文件进行分析。

 

java程序内存溢出java.lang.OutOfMemoryError: GC overhead limit exceeded问题排查_第1张图片

java程序内存溢出java.lang.OutOfMemoryError: GC overhead limit exceeded问题排查_第2张图片

解决问题:

1)将四级线程调用优化2级线程,

Timetask(调用发起) > ICreditUserFeatureAsynRemoteService(线程异步) > ICreditUserFeatureAsynBo(多线程并发) > ICreditUserFeatureService(Sercie服务类) > ICreditStatisticsUserService (Sercie服务类)

2)Timetask(调用发起)for30次调用改为扩大检索时间范围只一次调用,分摊到组分页中。

3)所有组一次性并发调用,调整为组分页中,每2页进行一次多线程调用;多线程中需要使用数据时,在线程处理中加载查询数据。

public Boolean polymerizeUser(final PagePeriod pagePeriod, final CreditRgisterUserQueryDTO requestDTO){
    logger.info("CreditUserFeatureAsynServiceImpl.polymerizeUser.begin...............................");
    //设置开始页,结束页
    Integer startPage = pagePeriod.getBeginPage(),endPage = pagePeriod.getEndPage();
    List>> callableList = new ArrayList>>();
    while (startPage <= endPage) {
        //多线程必须创建新对象设置当前页
       final CreditRgisterUserQueryDTO queryDTO = new CreditRgisterUserQueryDTO();
        queryDTO.setCurrentPage(startPage);
        queryDTO.setBeginTime(requestDTO.getBeginTime());
        queryDTO.setEndTime(requestDTO.getEndTime());
        queryDTO.setPageSize(requestDTO.getPageSize());

        if(startPage%THREAD_SIZE == 0) {
            callableList = new ArrayList>>();
        }
        callableList.add(new Callable>() {
            @Override
            public List call() {
                return creditUserFeatureService.getRegisterUser(queryDTO);
            }
        });

        //创建统计信息
        try {
            List>> futureList = executorService.invokeAll(callableList);
            for(Future> future : futureList){
                try {
                    List userDTOList = future.get();
                    if (CollectionUtil.isEmpty(userDTOList)) {
                       continue;
                    }
                    batchSaveRegisterUser(userDTOList);
                }catch (Exception ex) {
                    logger.error("CreditUserFeatureAsynServiceImpl.polymerizeUser.batchSaveRegisterUser.ex:",ex);
                }
            }
        } catch (Exception ex) {
            logger.error("CreditUserFeatureAsynServiceImpl.polymerizeUser.invokeAll.ex:",ex);
        }
        try {
            if(startPage%THREAD_SIZE == 0) {
                Thread.sleep(60000);
            }
        } catch (InterruptedException sellpTe) {
            logger.error("CreditUserFeatureAsynServiceImpl.polymerizeUser.sleep.te:",sellpTe);
        }
        startPage++;
    }
    logger.info("CreditUserFeatureAsynServiceImpl.polymerizeUser.end...............................");
    return Boolean.TRUE;
}

4)优化AppenderManager类中Executors.newFixedThreadPool代码中使用队列默认为Integer的最大值,改为自定义队列大小。

优化后dump分析图:

java程序内存溢出java.lang.OutOfMemoryError: GC overhead limit exceeded问题排查_第3张图片

java程序内存溢出java.lang.OutOfMemoryError: GC overhead limit exceeded问题排查_第4张图片

 

参考相关资料:
https://blog.csdn.net/nielinqi520/article/details/78455614
https://blog.csdn.net/chenxiusheng/article/details/74007040?utm_source=blogxgwz12
https://blog.csdn.net/bxyz1203/article/details/7465271

你可能感兴趣的:(技术类)