We solved this problem. Turns out it has nothing to do with JBoss Messaging and everything
to do with our code. I'm relating our experiences here in case anyone else encounters
a similar problem.
Queue clients are not only a JBoss server hosting MDB's listening to those queues, but
also freestanding programs. Because there is a fair number of these freestanding programs,
they each cache a QueueConnection. These programs are multithreaded. In short, there was a
race condition in our code which caused the QueueConnection.close() to hang. After this
occurred enough times, the JBoss session pool ran out of sessions. We observed other
strange behavior as well, including the fact that we could not undeploy applications from
JBoss listening in on queue's affected by this problem. Our initial approach was to
interrupt() one of the deadlocked threads, drop the reference, and create a new one with a
fresh QueueConnection. However, onMessage() was called immediately once again, before the
deadlock condition could resolve itself. A side-effect of this nonsense was that the JVM
filled up with orphaned threads all waiting on QueueConnection.close(). Eventually we had
OutOfMemoryE!
rrors.
The race condition never occurred in JBossMQ because it seems that the thread which calls
onMessage() in a MessageListener was the same thread as calls onException() in an
ExceptionListener. In JBoss Messaging, it appears these are different threads. Our problem
was that an onException() called while onMessage() was being processed caused a deadlock
under a particular, application-specific condition. After this, QueueConnection.close()
would wait for onMessage() to complete, which was forever.
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4033144#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...