[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1702:
--------------------------------------
I've had a look at the stack trace, here are some observations:
You basically have this situation:
You have two different JMS connection factorys (I guess you're not reusing them),
which have created (at least) one connection each.
Let's call them factory1 and factory2.
A connection failure on a connection created by factory1 occurs about the same time as a
connection created by factory2.
Thread 1 locks factory2 then calls your exceptionlistener, which then tries to close a
connection created from *factory1*. I.e. your exception listener is trying to close a
*different* connection to the one the failure has been detected on.
I suspect you have registered your same exceptionHandler instance on more than one
connection.
I notice that in BaseJMSObject you register your exceptionlistener in the connect()
method
connection.setExceptionListener(this);
but you don't unregister it from the connection anywhere. so I guess it would be
possible for it to register on multiple connections, so you can end up with 2 different
threads trying to close the same connection.
I'm guessing you could workaround the issue by calling
connection.setExceptionListener(null) in your disconnect() method (and this is only a
guess).
We could add some code in JBossConnection to make sure exceptionListeners never get called
after a connection is closed.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL:
https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira