Debugging hangs in JVM (on AIX but methodology applicable to other platforms)
The
JVM is hanging if the process is still present
but is not responding
in some sense. The end user experience usually is timing out and get a 404 page not found error
in the browser. This lack of response can be caused because:
* The process has come to a complete halt because of a deadlock condition
* The process has become caught in an infinite loop
* The process is running very slowly
If the process is not taking up any CPU time, it is deadlocked. Use the "ps -fp [process id]" command to investigate whether the process is still using CPU time. The ps command is described in AIX debugging commands. For example:
root@pw4 # ps -fp 790652
UID PID PPID C STIME TTY TIME CMD
root 790652 1 0 13:54:37 pts/0 3:05 /usr/IBM/WebSphere/AppServer/java/bin/java -Declipse.se
[after 5 minutes while the app server is still being hit by requests:]
root@pw4 # ps -fp 790652
UID PID PPID C STIME TTY TIME CMD
root 790652 1 0 13:54:37 pts/0 3:05 /usr/IBM/WebSphere/AppServer/java/bin/java -Declipse.se
If the value of 'TIME' increases over the course of a few minutes, the process is still using the CPU and is not deadlocked. As the example shown above, the TIME column stayed "3:05" for the JVM process, indicating it hangs and mostly likely caused by deadlocks.
Next step is to take a java thread dump on the hanging JVM, by doing a "kill -3 [pid]", and look at the generated javacore file:
root@pw4 # kill -3 790652
root@pw4 # ls *790652*
should display a file like: javacore.20070126.122115.790652.txt
root@pw4 # vi javacore.20070126.122115.790652.txt
[and look for section similar to the following]
| LOCKS subcomponent dump routine
| ===============================
|
| Monitor pool info:
| Current total number of monitors: 2
|
|Monitor Pool Dump (flat & inflated object-monitors):
| sys_mon_t:0x00039B40 infl_mon_t: 0x00039B80:
| java/lang/Integer@004B22A0/004B22AC: Flat locked by "DeadLockThread 1"
| (0x41DAB100), entry count 1
| Waiting to enter:
| "DeadLockThread 0" (0x41DAAD00)
| sys_mon_t:0x00039B98 infl_mon_t: 0x00039BD8:
| java/lang/Integer@004B2290/004B229C: Flat locked by "DeadLockThread 0"
| (0x41DAAD00), entry count 1
| Waiting to enter:
| "DeadLockThread 1" (0x41DAB100)
| JVM System Monitor Dump (registered monitors):
| Thread global lock (0x00034878):
| Windows native HEAP lock lock (0x000348D0):
| NLS hash table lock (0x00034928):
| portLibrary_j9sig_async_monitor lock (0x00034980):
| Hook Interface lock (0x000349D8):
| < lines removed for brevity >
|
|=======================
|Deadlock detected !!!
|---------------------
|
| Thread "DeadLockThread 1" (0x41DAB100)
| is waiting for:
| sys_mon_t:0x00039B98 infl_mon_t: 0x00039BD8:
| java/lang/Integer@004B2290/004B229C:
| which is owned by:
| Thread "DeadLockThread 0" (0x41DAAD00)
| which is waiting for:
| sys_mon_t:0x00039B40 infl_mon_t: 0x00039B80:
| java/lang/Integer@004B22A0/004B22AC:
| which is owned by:
| Thread "DeadLockThread 1" (0x41DAB100)
Detecting Busy Hangs
If there is no deadlock between threads, consider other reasons why threads are not carrying out useful work. Usually, this state occurs for one of the following reasons:
1. Threads are in a 'wait' state waiting to be 'notified' of work to be done.
2. Threads are in explicit sleep cycles.
3. Threads are in I/O calls waiting to do work.
The first two reasons imply a fault in the Java code, either that of the application, or that of the standard class files included in the SDK.
The third reason, where threads are waiting (for instance, on sockets) for I/O, requires further investigation. Has the process at the other end of the I/O failed? Do any network problems exist?
To see how the javadump tool is used to diagnose loops, see Threads and stack trace (THREADS). If you cannot diagnose the problem from the javadump and if the process still seems to be using processor cycles, either it has entered an infinite loop or it is suffering from very bad performance. Using "ps -mp [process id] -o THREAD" allows individual threads in a particular process to be monitored to determine which threads are using the CPU time. If the process has entered an infinite loop, it is likely that a small number of threads will be using the time. For example:
$ ps -mp 43824 -o THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
wsuser 43824 51762 - A 66 60 77 * 200001 pts/4 - java ...
- - - 4021 S 0 60 1 22c4d670 c00400 - - -
- - - 11343 S 0 60 1 e6002cbc 8400400 - - -
- - - 14289 S 0 60 1 22c4d670 c00400 - - -
- - - 14379 S 0 60 1 22c4d670 c00400 - - -
...
- - - 43187 S 0 60 1 701e6114 400400 - - -
- - - 43939 R 33 76 1 20039c88 c00000 - - -
- - - 50275 S 0 60 1 22c4d670 c00400 - - -
- - - 52477 S 0 60 1 e600ccbc 8400400 - - -
...
- - - 98911 S 0 60 1 7023d46c 400400 - - -
- - - 99345 R 33 76 0 - 400000 - - -
- - - 99877 S 0 60 1 22c4d670 c00400 - - -
- - - 100661 S 0 60 1 22c4d670 c00400 - - -
- - - 102599 S 0 60 1 22c4d670 c00400 - - -
...
Those threads with the value 'R' under 'ST' are in the 'runnable' state, and therefore are able to accumulate processor time. What are these threads doing? The output from ps shows the TID (Kernel Thread ID) for each thread. This can be mapped to the Java thread ID using dbx ( AIX debugger). The output of the dbx thread command gives an output of the form of:
root@pw4 # dbx -a 43824
dbx > thread
thread state-k wchan state-u k-tid mode held scope function
$t1 wait 0xe60196bc blocked 104099 k no sys _pthread_ksleep
>$t2 run blocked 68851 k no sys _pthread_ksleep
$t3 wait 0x2015a458 running 29871 k no sys pthread_mutex_lock
...
$t50 wait running 86077 k no sys getLinkRegister
$t51 run running 43939 u no sys reverseHandle
$t52 wait running 56273 k no sys getLinkRegister
$t53 wait running 37797 k no sys getLinkRegister
$t60 wait running 4021 k no sys getLinkRegister
$t61 wait running 18791 k no sys getLinkRegister
$t62 wait running 99345 k no sys getLinkRegister
$t63 wait running 20995 k no sys getLinkRegister
By matching the TID value from ps to the k-tid value from the dbx thread command, you can see that the currently running methods in this case are reverseHandle and getLinkRegister.
Now you can use dbx to generate the C thread stack for these two threads using the dbx thread command for the corresponding dbx thread numbers ($tx). To obtain the full stack trace including Java frames, map the dbx thread number to the threads pthread_t value, which is listed by the Javadump file, and can be obtained from the ExecEnv structure for each thread using the Dump Viewer. Do this with the dbx command thread info [dbx thread number], which produces an output of the form ( in the above output $t51 is actually running: the one that uses CPU cycles, so we use 51 in the next dbx command):
dbx > thread info 51
thread state-k wchan state-u k-tid mode held scope function
$t51 run running 43939 u no sys reverseHandle
general:
pthread addr = 0x220c2dc0 size = 0x18c
vp addr = 0x22109f94 size = 0x284
thread errno = 61
start pc = 0xf04b4e64
joinable = yes
pthread_t = 3233
scheduler:
kernel =
user = 1 (other)
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x220c8018 size = 0x40000
limit = 0x22108018
sp = 0x22106930
r
$t63 wait running 20995 k no sys getLinkRegister
Showing that the TID value from ps (k-tid in dbx) corresponds to dbx thread number 51, which has a pthread_t of 3233. Looking for the pthread_t in the Javadump file, you now have a full stack trace:
"Worker#31" (TID:0x36288b10, sys_thread_t:0x220c2db8) Native Thread State:
ThreadID: 00003233 Reuse: 1 USER SUSPENDED Native Stack Data : base: 22107f80
pointer 22106390 used(7152) free(250896)
----- Monitors held -----
java.io.OutputStreamWriter@3636a930
com.ibm.servlet.engine.webapp.BufferedWriter@3636be78
com.ibm.servlet.engine.webapp.WebAppRequestDispatcher@3636c270
com.ibm.servlet.engine.srt.SRTOutputStream@36941820
com.ibm.servlet.engine.oselistener.nativeEntry.NativeServerConnection@36d84490 JNI pinning lock
----- Native stack -----
_spin_lock_global_common pthread_mutex_lock - blocked on Heap Lock
sysMonitorEnterQuicker sysMonitorEnter unpin_object unpinObj
jni_ReleaseScalarArrayElements jni_ReleaseByteArrayElements
Java_com_ibm_servlet_engine_oselistener_nativeEntry_NativeServerConnection_nativeWrite
------ Java stack ------ () prio=5
com.ibm.servlet.engine.oselistener.nativeEntry.NativeServerConnection.write(Compiled Code)
com.ibm.servlet.engine.srp.SRPConnection.write(Compiled Code)
com.ibm.servlet.engine.srt.SRTOutputStream.write(Compiled Code)
java.io.OutputStreamWriter.flushBuffer(Compiled Code)
java.io.OutputStreamWriter.flush(Compiled Code)
java.io.PrintWriter.flush(Compiled Code)
com.ibm.servlet.engine.webapp.BufferedWriter.flushChars(Compiled Code)
com.ibm.servlet.engine.webapp.BufferedWriter.write(Compiled Code)
java.io.Writer.write(Compiled Code)
java.io.PrintWriter.write(Compiled Code)
java.io.PrintWriter.write(Compiled Code)
java.io.PrintWriter.print(Compiled Code)
java.io.PrintWriter.println(Compiled Code)
pagecompile._identifycustomer_xjsp.service(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.JSPState.service(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.PageCompileServlet.doService(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.PageCompileServlet.doGet(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
And, using the full stack trace, it should be possible to identify any infinite loop that might be occurring. The above example shows the use of spin_lock_global_common, which is a busy wait on a lock, hence the use of CPU time. For more information, consult the IBM AIX documentation.
* The process has come to a complete halt because of a deadlock condition
* The process has become caught in an infinite loop
* The process is running very slowly
If the process is not taking up any CPU time, it is deadlocked. Use the "ps -fp [process id]" command to investigate whether the process is still using CPU time. The ps command is described in AIX debugging commands. For example:
root@pw4 # ps -fp 790652
UID PID PPID C STIME TTY TIME CMD
root 790652 1 0 13:54:37 pts/0 3:05 /usr/IBM/WebSphere/AppServer/java/bin/java -Declipse.se
[after 5 minutes while the app server is still being hit by requests:]
root@pw4 # ps -fp 790652
UID PID PPID C STIME TTY TIME CMD
root 790652 1 0 13:54:37 pts/0 3:05 /usr/IBM/WebSphere/AppServer/java/bin/java -Declipse.se
If the value of 'TIME' increases over the course of a few minutes, the process is still using the CPU and is not deadlocked. As the example shown above, the TIME column stayed "3:05" for the JVM process, indicating it hangs and mostly likely caused by deadlocks.
Next step is to take a java thread dump on the hanging JVM, by doing a "kill -3 [pid]", and look at the generated javacore file:
root@pw4 # kill -3 790652
root@pw4 # ls *790652*
should display a file like: javacore.20070126.122115.790652.txt
root@pw4 # vi javacore.20070126.122115.790652.txt
[and look for section similar to the following]
| LOCKS subcomponent dump routine
| ===============================
|
| Monitor pool info:
| Current total number of monitors: 2
|
|Monitor Pool Dump (flat & inflated object-monitors):
| sys_mon_t:0x00039B40 infl_mon_t: 0x00039B80:
| java/lang/Integer@004B22A0/004B22AC: Flat locked by "DeadLockThread 1"
| (0x41DAB100), entry count 1
| Waiting to enter:
| "DeadLockThread 0" (0x41DAAD00)
| sys_mon_t:0x00039B98 infl_mon_t: 0x00039BD8:
| java/lang/Integer@004B2290/004B229C: Flat locked by "DeadLockThread 0"
| (0x41DAAD00), entry count 1
| Waiting to enter:
| "DeadLockThread 1" (0x41DAB100)
| JVM System Monitor Dump (registered monitors):
| Thread global lock (0x00034878):
| Windows native HEAP lock lock (0x000348D0):
| NLS hash table lock (0x00034928):
| portLibrary_j9sig_async_monitor lock (0x00034980):
| Hook Interface lock (0x000349D8):
| < lines removed for brevity >
|
|=======================
|Deadlock detected !!!
|---------------------
|
| Thread "DeadLockThread 1" (0x41DAB100)
| is waiting for:
| sys_mon_t:0x00039B98 infl_mon_t: 0x00039BD8:
| java/lang/Integer@004B2290/004B229C:
| which is owned by:
| Thread "DeadLockThread 0" (0x41DAAD00)
| which is waiting for:
| sys_mon_t:0x00039B40 infl_mon_t: 0x00039B80:
| java/lang/Integer@004B22A0/004B22AC:
| which is owned by:
| Thread "DeadLockThread 1" (0x41DAB100)
Detecting Busy Hangs
If there is no deadlock between threads, consider other reasons why threads are not carrying out useful work. Usually, this state occurs for one of the following reasons:
1. Threads are in a 'wait' state waiting to be 'notified' of work to be done.
2. Threads are in explicit sleep cycles.
3. Threads are in I/O calls waiting to do work.
The first two reasons imply a fault in the Java code, either that of the application, or that of the standard class files included in the SDK.
The third reason, where threads are waiting (for instance, on sockets) for I/O, requires further investigation. Has the process at the other end of the I/O failed? Do any network problems exist?
To see how the javadump tool is used to diagnose loops, see Threads and stack trace (THREADS). If you cannot diagnose the problem from the javadump and if the process still seems to be using processor cycles, either it has entered an infinite loop or it is suffering from very bad performance. Using "ps -mp [process id] -o THREAD" allows individual threads in a particular process to be monitored to determine which threads are using the CPU time. If the process has entered an infinite loop, it is likely that a small number of threads will be using the time. For example:
$ ps -mp 43824 -o THREAD
USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMAND
wsuser 43824 51762 - A 66 60 77 * 200001 pts/4 - java ...
- - - 4021 S 0 60 1 22c4d670 c00400 - - -
- - - 11343 S 0 60 1 e6002cbc 8400400 - - -
- - - 14289 S 0 60 1 22c4d670 c00400 - - -
- - - 14379 S 0 60 1 22c4d670 c00400 - - -
...
- - - 43187 S 0 60 1 701e6114 400400 - - -
- - - 43939 R 33 76 1 20039c88 c00000 - - -
- - - 50275 S 0 60 1 22c4d670 c00400 - - -
- - - 52477 S 0 60 1 e600ccbc 8400400 - - -
...
- - - 98911 S 0 60 1 7023d46c 400400 - - -
- - - 99345 R 33 76 0 - 400000 - - -
- - - 99877 S 0 60 1 22c4d670 c00400 - - -
- - - 100661 S 0 60 1 22c4d670 c00400 - - -
- - - 102599 S 0 60 1 22c4d670 c00400 - - -
...
Those threads with the value 'R' under 'ST' are in the 'runnable' state, and therefore are able to accumulate processor time. What are these threads doing? The output from ps shows the TID (Kernel Thread ID) for each thread. This can be mapped to the Java thread ID using dbx ( AIX debugger). The output of the dbx thread command gives an output of the form of:
root@pw4 # dbx -a 43824
dbx > thread
thread state-k wchan state-u k-tid mode held scope function
$t1 wait 0xe60196bc blocked 104099 k no sys _pthread_ksleep
>$t2 run blocked 68851 k no sys _pthread_ksleep
$t3 wait 0x2015a458 running 29871 k no sys pthread_mutex_lock
...
$t50 wait running 86077 k no sys getLinkRegister
$t51 run running 43939 u no sys reverseHandle
$t52 wait running 56273 k no sys getLinkRegister
$t53 wait running 37797 k no sys getLinkRegister
$t60 wait running 4021 k no sys getLinkRegister
$t61 wait running 18791 k no sys getLinkRegister
$t62 wait running 99345 k no sys getLinkRegister
$t63 wait running 20995 k no sys getLinkRegister
By matching the TID value from ps to the k-tid value from the dbx thread command, you can see that the currently running methods in this case are reverseHandle and getLinkRegister.
Now you can use dbx to generate the C thread stack for these two threads using the dbx thread command for the corresponding dbx thread numbers ($tx). To obtain the full stack trace including Java frames, map the dbx thread number to the threads pthread_t value, which is listed by the Javadump file, and can be obtained from the ExecEnv structure for each thread using the Dump Viewer. Do this with the dbx command thread info [dbx thread number], which produces an output of the form ( in the above output $t51 is actually running: the one that uses CPU cycles, so we use 51 in the next dbx command):
dbx > thread info 51
thread state-k wchan state-u k-tid mode held scope function
$t51 run running 43939 u no sys reverseHandle
general:
pthread addr = 0x220c2dc0 size = 0x18c
vp addr = 0x22109f94 size = 0x284
thread errno = 61
start pc = 0xf04b4e64
joinable = yes
pthread_t = 3233
scheduler:
kernel =
user = 1 (other)
event :
event = 0x0
cancel = enabled, deferred, not pending
stack storage:
base = 0x220c8018 size = 0x40000
limit = 0x22108018
sp = 0x22106930
r
$t63 wait running 20995 k no sys getLinkRegister
Showing that the TID value from ps (k-tid in dbx) corresponds to dbx thread number 51, which has a pthread_t of 3233. Looking for the pthread_t in the Javadump file, you now have a full stack trace:
"Worker#31" (TID:0x36288b10, sys_thread_t:0x220c2db8) Native Thread State:
ThreadID: 00003233 Reuse: 1 USER SUSPENDED Native Stack Data : base: 22107f80
pointer 22106390 used(7152) free(250896)
----- Monitors held -----
java.io.OutputStreamWriter@3636a930
com.ibm.servlet.engine.webapp.BufferedWriter@3636be78
com.ibm.servlet.engine.webapp.WebAppRequestDispatcher@3636c270
com.ibm.servlet.engine.srt.SRTOutputStream@36941820
com.ibm.servlet.engine.oselistener.nativeEntry.NativeServerConnection@36d84490 JNI pinning lock
----- Native stack -----
_spin_lock_global_common pthread_mutex_lock - blocked on Heap Lock
sysMonitorEnterQuicker sysMonitorEnter unpin_object unpinObj
jni_ReleaseScalarArrayElements jni_ReleaseByteArrayElements
Java_com_ibm_servlet_engine_oselistener_nativeEntry_NativeServerConnection_nativeWrite
------ Java stack ------ () prio=5
com.ibm.servlet.engine.oselistener.nativeEntry.NativeServerConnection.write(Compiled Code)
com.ibm.servlet.engine.srp.SRPConnection.write(Compiled Code)
com.ibm.servlet.engine.srt.SRTOutputStream.write(Compiled Code)
java.io.OutputStreamWriter.flushBuffer(Compiled Code)
java.io.OutputStreamWriter.flush(Compiled Code)
java.io.PrintWriter.flush(Compiled Code)
com.ibm.servlet.engine.webapp.BufferedWriter.flushChars(Compiled Code)
com.ibm.servlet.engine.webapp.BufferedWriter.write(Compiled Code)
java.io.Writer.write(Compiled Code)
java.io.PrintWriter.write(Compiled Code)
java.io.PrintWriter.write(Compiled Code)
java.io.PrintWriter.print(Compiled Code)
java.io.PrintWriter.println(Compiled Code)
pagecompile._identifycustomer_xjsp.service(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.JSPState.service(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.PageCompileServlet.doService(Compiled Code)
com.ibm.servlet.jsp.http.pagecompile.PageCompileServlet.doGet(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
javax.servlet.http.HttpServlet.service(Compiled Code)
And, using the full stack trace, it should be possible to identify any infinite loop that might be occurring. The above example shows the use of spin_lock_global_common, which is a busy wait on a lock, hence the use of CPU time. For more information, consult the IBM AIX documentation.