ApplicationDeadlockException

阅读更多

参考 ： How to avoid huge transactions with CMP Entity Beans on JBoss

2014-05-23 11:06:32,475 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackException in method: 
  public abstract com.spokesoft.component.sla.SlaDetails com.spokesoft.component.sla.ejb.Sla.getSlaDetails() 
  throws java.rmi.RemoteException, causedBy:
org.jboss.util.deadlock.ApplicationDeadlockException: Application deadlock detected, 
  resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@a74da8, bean=SlaBean, id=[org:AWSUK],[id:102], 
  refs=2, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=BEMABC0RFMA3/505839, BranchQual=, localId=505839], 
  synched=Thread[http-0.0.0.0-80-29,5,jboss], timeout=5000, queue=[], 
  holder=TransactionImpl:XidImpl[FormatId=257, GlobalId=BEMABC0RFMA3/505826, BranchQual=, localId=505826], 
  waitingResource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@1db76e2, 
  bean=CalendarBean, id=[org:AWSUK],[id:102], refs=2, 
  tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=BEMABC0RFMA3/505826, BranchQual=, localId=505826], 
  synched=null, timeout=5000, 
  queue=[TXLOCK waitingTx=TransactionImpl:XidImpl[FormatId=257, GlobalId=BEMABC0RFMA3/505839, BranchQual=, 
  localId=505839] id=0 thread=Thread[http-0.0.0.0-80-21,5,jboss] queued=true], 
  waitingResourceHolder=TransactionImpl:XidImpl[FormatId=257, GlobalId=BEMABC0RFMA3/505826, BranchQual=, 
  localId=505826]
    at org.jboss.util.deadlock.DeadlockDetector.deadlockDetection(DeadlockDetector.java:69)
    at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.waitForTx(QueuedPessimisticEJBLock.java:292)
    at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java:230)
    at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.schedule(QueuedPessimisticEJBLock.java:194)
...

By default, CMP Entity Beans on JBoss are set to require a transaction. Also by default, any time you touch any session or entity bean, your request thread takes out a lock on that entire object, even if you are only reading it and not updating it. Lastly, also by default, JBoss will make sure that for any given entity, there is only one instance of that entity in memory at a time.

All of these defaults have serious implications. For one, it implies that anything other than a toy application will likely become a de-facto, single-threaded application. Imagine, for example, that you have an earthquake tracking application. Your application might have an Entity Bean called Earthquake. After getting under way with the application, you realize there are different kinds of earthquake: tectonic, volcanic, and man-made. These don’t merit having a full-on Earthquake subclass of their own, but maybe you want to model the types as a new Entity called EarthquakeType so that the application can be data-driven and new types can be added later without changing code. The vast majority (~90%) of earthquakes are tectonic, so most of what you ever display to a user will be “tectonic”.

So, you might have a web page that displays the last 40 earthquakes in descending chronological order in a table and also a count of how many different types. This could lead to innocent code like, say:

foreach (Earthquake earthquake : earthquakes){
typeSum[earthquake.getType().getId()]++;
}
The moment you call earthquake.getType() for the first earthquake in the list, you will lock the “tectonic” instance of the EarthquakeType Entity bean. This means that every other thread executing in the same JVM (if configured the default JBoss way) will most likely block (who doesn’t need to know what the earthquake type is, after all?) until this thread is done displaying its page. Even worse, if this thread is holding a lock that some other thread needs, and that other thread is holding a lock that this thread needs, then you have a deadlock. All of this in spite of the fact that actually updating an EarthquakeType is extremely rare because they are read-mostly.

A telltale sign that you are having this problem is seeing stack traces like this one:

org.jboss.util.deadlock.ApplicationDeadlockException: Application deadlock detected, resource=org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock@290df5c3, bean=

…snip…

at org.jboss.util.deadlock.DeadlockDetector.deadlockDetection(DeadlockDetector.java:69)
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.waitForTx(QueuedPessimisticEJBLock.java:292)
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java:230)

…snip.

At first, it’s tempting to fume at JBoss for having such conservative default settings. I know I did this morning as I was learning more about the details. But the fact is that they really have no choice. The application container has no idea that EarthquakeType is read-mostly. It doesn’t know if you will read it at the beginning of the request and then modify it 300 milliseconds later at the end of the request. So, it is forced to loop absolutely everything you touch into a giant transaction unless you tell it otherwise.

Now, the “telling it otherwise” is where things start to get tricky. Here, I really do think that JBoss hasn’t done us any favors. It’s a multi-step process to making sure you maximize your throughput and minimize deadlocks. If you do some steps but don’t do others, then nothing will change and you won’t know why.

So, here are the steps…

1) Mark all your read-only methods on Entity beans as such. This is where most people start, probably because it’s the kind of advice that pops to the top of Google searches. However, taking this step is a necessary but not sufficient condition for breaking down these huge transactions and their locks. You still have to follow step 2 and step 3.

To mark a method as read-only, you need to add the “true” element to its “entity” entry in jboss.xml:


Earthquake
...


getType
true

...
If you are using XDoclet, then you can accomplish the same thing by adding an annotation:

/*
*  @jboss.method-attributes  read-only="true"
*/
2) Configure the container to use a lock manager that gives a damn about read-only methods.

Even though you’ve now marked your read-only methods as read-only, the default lock manager (QueuedPessimisticEJBLock) doesn’t actually do anything useful with this information.  This fact is buried about half of the way down this lengthy page: http://docs.jboss.org/jbossas/jboss4guide/r2/html/ch5.chapter.html.

To take full advantage of your shiny, new read-only methods, you have to change the configuration for your entity beans to instead use the SimpleReadWriteEJBLock lock manager.  This is a two step process.  First, create a new container configuration that extends the default one in jboss.xml:





DeadlockAvoidingConfiguration

org.jboss.ejb.plugins.lock.SimpleReadWriteEJBLock





If you are using XDoclet, you can put the same entry in a file called “jboss-container.xml” and XDoclet will merge them together.

Next, you have to explicitly tell every entity bean to use this container configuration instead of the default one.  You do this by adding an entry to the entity entries in jboss.xml:



Earthquake

…

DeadlockAvoidingConfiguration

…

Or, again, if you are using XDoclet then add an annotation to the top of your class:

@jboss.container-configuration name = “DeadlockAvoidingConfiguration”

3) Deal with the read-only fallout

This doesn’t all come for free.  Specifically, a common (though not OO-friendly) practice when dealing with one-to-many relationships between entity EJBs is to have a getter that returns a Collection and have client code simply add and remove things from that collection.  But, once you’ve marked that getter as read only, then if you try to manipulate the returned Collection it will throw an exception.  To fix this, you have to create “adder” and “remover” methods for that relationship.

For example, let’s say we wanted to keep track of witness reports of an Earthquake.  You might add a WitnessReport class to the system and a relationship called Earthquake.getWitnessReports() that returns a Collection of WitnessReports.  When a new witness sends in a report, then you might have code that does this:

anEarthquake.getWitnessReports().add(aNewWitnessReport);

That code works fine if “getWitnessReports()” is left to its default settings, but blows up with an exception saying your CMR collection is read only if you’ve marked “getWitnessReports()” as read only.  What you need to do is add a new method to Earthquake,  like so:

public void addWitnessReport(WitnessReport record){

Collection existingReports = this.getWitnessReports();

Collection newReports = new ArrayList(existingReports.size() + 1);

newReports.addAll(existingReports);

newReports.add(record);

this.setWitnessReports(newReports);

}

When this method is invoked, the lock manager will take out a lock and remember to write the changes back to the data store.  It’s not pretty, but it does get the job done and saves a great deal of performance elsewhere in the application.

So that’s it.  Now you have a faster application that is less deadlock prone.  Enjoy the time you save by going out and having a beer right now.

p.s.  though I haven’t tested this, apparently all of the above doesn’t work in EJB 3.0 on JBoss