An inadvertent deadlock in the application code can cause a server to hang. For example, a situation in which thread1 is waiting for resource1 and is holding a lock on resource2, while thread2 needs resource2 and is holding the lock on resource1. Neither thread can progress.
This Application Deadlock pattern should be used only after doing all the steps in the Generic Server Hang pattern. One indicator that this is an application deadlock problem is that thread dumps will show the threads are in the application methods. Several thread dumps taken a few seconds apart will show that the threads are not progressing. Troubleshooting this problem will involve reviewing the application code. There exists a thread analyzer tool at BEA dev2dev which has proven useful in analysis of the thread dumps.
Fundamentaly, this problem happens because the design and implementation of the application has introduced the possibility of deadlocks. These types of problems may only show up under heavy load. Therefore, these applications often pass through QA testing and become problems in production.
Coding problems to look for:
Hashtable
(synchronized) versus the use of HashMap
(unsynchronized) import java.util.Vector; <– Vector is a synchronized java class
Public class Employee {
Vector names = new Vector();
Employees () {
Object object = new Object();
synchronized (object) {
names.add("al");
names.add("Saganich");
}
}
synchronized String getName (int index) {
String name = (String) names.elementAt(index);
return name;
}
import java…..;
import java…..;
public class myAppMethods {
public String getName(String name) {
String lastname = getLastName(name);
return lastname;
}
public synchronized String getLastName (String name) {
do a DB Lookup <———— takes mucho time to get a last name
return lastname;
}
If the database if very slow, the server can appear to be hung because the threads will line up trying to get access to the database and all the available threads could eventually be used up.
If the first server is faster than the remote server, eventually all the threads in the execute pool will be exhausted by the server making outbound requests with fewer threads available for processing inbound responses. As the load grows, the number of outgoing requests that cannot complete their work grows while they wait for an inbound response to complete. Below is an example of a thread in the waitForDataResponseImpl.java
method of.
"ExecuteThread: ‘52′ for queue: ‘default’" daemon prio=5 tid=0×4b3e40b0 nid=0×1170 waiting on monitor [0x4c74f000..0x4c74fdbc]
at java.lang.Object.wait(Native Method)
at
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)
Top of Page
WebLogic Server cannot detect deadlocked threads. Some JVM’s are able to do so. See External Resources. There is a tool available for thread analysis as well as good information about thread dumps on BEA dev2dev.
Top of Page
If you suspect a deadlock, it is helpful to go to the site of the JVM vendor to learn if there are clues provided for you in their thread dumps. If you are using JDK 1.3.1 you can add - XX:+JavaMonitorInStack Trace
to show locking more explicitly/builtin. Here is an example of an HP JVM thread dump in which it is clearly marked that the threads’ state are waiting on monitor, and the monitor is identified. See example below.
"msg 0-941667944865" (TID:0×7b1a5ba0, sys_thread_t:0×2a4290,
state:Waiting on Monitor,
thread_t: t@108, stack_base:0×7a76e000, stack_size:0×20000,
pc: 0xc01ea178, monitor = 0×25334) prio=8
MsgThread.rest(Compiled Code)
MsgThread.run(Compiled Code)
"msg 3-941667944865" (TID:0×7b1a5a80, sys_thread_t:0×263278,
state:Waiting on Monitor,
thread_t: t@105, stack_base:0×7a95d000, stack_size:0×20000,
pc: 0xc01ea178, monitor = 0×25334) prio=8
MsgThread.rest(Compiled Code)
MsgThread.run(Compiled Code)
"msg 5-941667944864" (TID:0×7b1a5f08, sys_thread_t:0×2c42d8,
state:Waiting on Monitor,
thread_t: t@106, stack_base:0×7aa65000, stack_size:0×20000,
pc: 0xc01ea178, monitor = 0×25334) prio=8
MsgThread.rest(Compiled Code)
MsgThread.run(Compiled Code)
备注:
本文转载自:http://www.hashei.me/2009/08/serverhang_application_deadlock.html