[JBossMQ] - Deadlock race-condition committing JBossMQ txn
by scotto
Hello.
We are getting an intermittent exception when our system processes a large number of JBossMQ messages in parallel. Whenever we get this exception, one of our Message Driven Beans becomes deadlocked, and never processes any more messages.
Our scenario is as follows (JBoss 4.2.3.GA, EJB3):
- We have 5 separate MDBs processing messages from 5 different queues simultaneously.
- Each MDB is a singleton on its queue (i.e. maxSession = 1).
- Each MDB has a large backlog of messages to process, so they are all processing at once.
- During processing of a message, each MDB calls a SLSB to do some work which updates the database. This SLSB will then in-turn call another SLSB which sends a messages to a JBossMQ topic (e.g. notifying observers that something has changed in the database).
This will work fine for a short while, but after a few seconds (maybe 50-100 messages processed) we always get the following exception:
anonymous wrote :
| 2008-10-31 11:47:50,440 563340 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (WorkManager(4)-124:) org.jboss.resource.adapter.jms.inflow.JmsServerSession@af2931 failed to commit/rollback
| org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=vieo-ws01/7090, BranchQual=, localId=7090] status=STATUS_NO_TRANSACTION; - nested throwable: (java.lang.IllegalMonitorStateExcepti
| on)
| at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:372)
| at org.jboss.tm.TxManager.commit(TxManager.java:240)
| at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:494)
| at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:248)
| at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
| at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
| at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
| at java.lang.Thread.run(Thread.java:595)
| Caused by: java.lang.IllegalMonitorStateException
| at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:125)
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1137)
| at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:431)
| at org.jboss.resource.adapter.jms.JmsManagedConnection.unlock(JmsManagedConnection.java:416)
| at org.jboss.resource.adapter.jms.JmsXAResource.prepare(JmsXAResource.java:89)
| at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.prepare(JcaXAResourceWrapper.java:93)
| at org.jboss.tm.TransactionImpl$Resource.prepare(TransactionImpl.java:2212)
| at org.jboss.tm.TransactionImpl.prepareResources(TransactionImpl.java:1660)
| at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:347)
| ... 7 more
|
At this point, the affected MDB becomes deadlocked, and fails to process any more messages. If we leave the system running, then eventually more MDBs fail with the same exception.
Notes:
1. If we only run ONE MDB, then we never get this exception! However as soon as we introduce even a second MDB, we start seeing these failures again.
2. The deadlock point seems to be when the code tries to return from the first SLSB call - i.e. execution never returns to the MDB onMessage() method context.
3. JBossMQ and the SLSB EntityManager use the same datasource (Postgres 8.2 as a local-tx-datasource).
4. This happens both if we use persistent and non-persistent messages.
5. If we mark the second nested SLSB (the one that sends a message to the JMS Topic) with TransactionAttributeType.NOT_SUPPORTED then we don't get this exception! This is not a viable solution however, as we need any messages sent during processing to be rolled back if an exception is thrown.
My guess is that this problem has something to do with the SLSB committing its transaction (at which point the nested JMS messages would also need to be sent?). As it is unpredictably intermittent and also works when single threaded, I think there must be some thread race-condition in the JMS locking mechanism.
Any help here would be greatly appreciated!
Thanks
Scott
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4185890#4185890
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4185890
17 years, 5 months
[JBoss Tools (users)] - deploying seam project
by deanhiller
anonymous wrote : We are currently porting from tomcat/MyEclipse combo which worked quite well with the first directory structure below and now we are trying to move to JBoss AS/JBoss Tools but it is not working out so well.
MAIN GOAL: get the debug button to work that starts up JBoss and runs in debug mode and attaches the debugger so I don't have to attach it myself.
We are using buildtemplate which has the following structure for every project regardless if jar, war, osgibundle or not(quite nice for changing projects and everything is in the same place still)....
project
| +--input
| +--javasrc - where the *.java files are
| +--libexclude - where *.jar files not included in distribution are
| +--libinclude - where *.jar files that need to be packaged up are
| +--webroot - where the xhtml files are
| +--WEB-INF - everything that goes in WEB-INF EXCEPT libraries and class files which are in the above directories to be consistent with all other company projects(jar, war, osgi bundle projects). Basically, this is just seam/JSF config files!!!
Since we could never get seam to work as a war in JBoss that depended on an external ejb3 jar file, we modified it to build an ear like so(which we do have working in JBoss via ant build).....
project
| +--input
| +--javasrc - where the *.java files are
| +--libexclude - where *.jar files not included in distribution are
| +--libinclude
| +-- ear - where *.jar files that go in <root> of ear file go
| +-- lib - where *.jar files that go in <root>/lib of ear go
| +-- war - where *.jar files that go in WEB-INF/lib go
| +--webroot - where the xhtml files are
| +--WEB-INF
I have an external xcore3.jar(my EJB3 jar file with entity beans) that is deployed outside the ear in the deploy directory. when I try to mount this project with JBoss Tools, I add seam facet in the properties and then I seem to have to add the JSF capabilities BUT it only asks me for one location of libraries when there seem to be 3 locations above for an ear deployment. I fill in libinclude as the directory which would include everything. Also, it builds a war file instead of an ear even though I chose ear on the seam project. Does anyone have any instructions on how to mount this with JBoss tools so I can just click the debug button?
Is there any hope of getting this to work in JBoss tools and JBoss AS or should I move back to MyEclipse/Tomcat combo which was working quite nicely. In fact, it seams we have to write way more code in JBoss with having to code up every local interface for every bean which is very annoying and we did not need to do in tomcat but that is another story.
Any ideas on how to make this move?
thanks
Dean Hiller
http://xsoftware.biz
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4185868#4185868
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4185868
17 years, 5 months