[JBoss JIRA] Created: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
7:01 p.m.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
I have a client application which has a single producer and receiver. These are created as
singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:35 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1702:
--------------------------------------
Can you post the code to the following methods:
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
So we can see what is happening in your reconnect logic?
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:40 a.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Adrian Woodhead updated JBMESSAGING-1702:
-----------------------------------------
Attachment: BaseJMSObject.java
QueueingProducer.java
source code for BaseJMSObject.java and QueueingProducer.java attached.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:22 a.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1702:
----------------------------------------------
I tried shutting down a client which hadn't ever been connected to a crashed server
(https://jira.jboss.org/jira/browse/JBMESSAGING-1699) and it shut down cleanly. So I'm
thinking this issue might be related to 1699 and the onException handler being called
getting it into a deadlock state. I'll keep an eye on it and let you know when I find
out more.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:19 p.m.
New subject: [JBoss JIRA] Updated: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox updated JBMESSAGING-1702:
---------------------------------
Fix Version/s: 2.0.0
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:25 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1702:
--------------------------------------
I've had a look at the stack trace, here are some observations:
You basically have this situation:
You have two different JMS connection factorys (I guess you're not reusing them),
which have created (at least) one connection each.
Let's call them factory1 and factory2.
A connection failure on a connection created by factory1 occurs about the same time as a
connection created by factory2.
Thread 1 locks factory2 then calls your exceptionlistener, which then tries to close a
connection created from *factory1*. I.e. your exception listener is trying to close a
*different* connection to the one the failure has been detected on.
I suspect you have registered your same exceptionHandler instance on more than one
connection.
I notice that in BaseJMSObject you register your exceptionlistener in the connect()
method
connection.setExceptionListener(this);
but you don't unregister it from the connection anywhere. so I guess it would be
possible for it to register on multiple connections, so you can end up with 2 different
threads trying to close the same connection.
I'm guessing you could workaround the issue by calling
connection.setExceptionListener(null) in your disconnect() method (and this is only a
guess).
We could add some code in JBossConnection to make sure exceptionListeners never get called
after a connection is closed.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
2:44 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1702:
--------------------------------------
I can replicate this deadlock in a simple test which registers the same exceptionlistener
on two different connections obtained from two different connection factorys. Will
investigate further.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:10 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1702:
----------------------------------------------
In our situation we have one connection factory (looked up via JNDI) instance and then one
Producer instance and one Receiver instance (both extend JMSObject). These 2 instances use
the same connection factory instance to get connections. When they start up they call
connect() which gets the connection and store it as a member variable. They then register
the exception handler on this connection. This is the same connection as the one that is
used in onException to disconnect from. So unless I am missing something its the same
connection being used by each JMSObject throughout.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:27 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1702:
--------------------------------------
The two different conection factory instances can be seen as follows:
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
**** this is the failoverLock in the ConnectionManagerImpl class
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
And:
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
*** This is also the failoverLock but in a different instance of ConnectionManagerImpl
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Note that the ids for monitors highlighted above are different - this means you have 2
distinct ConnectionManagerImpl instances.
Each ConnectionManagerImpl instance is maintained by a ConnectionFactory so you must have
(at least) two connection factorys instance.
I don't know how you've got 2, but you've definitely got 2.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:42 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1702:
----------------------------------------------
I did a jmap -histo dump of one of our running clients and you are right, we have two
connection factories:
341: 2 448 org.jboss.messaging.core.client.impl.ConnectionManagerImpl
We get the factories from a singleton JNDIJMSConext (of which we definitely only have
one):
951: 1 32 fm.last.jms.jbm.JNDIJMSContext
Each object calls
@Override
public ConnectionFactory getConnectionFactory(String factoryName) throws NamingException
{
return (ConnectionFactory) initialContext.lookup(factoryName);
}
to get a connection factory. This method is called twice (once for the receiver, once for
the producer) so I guess the lookup is returning two references (possibly to the same
connection factory found via JNDI?) Either way, I don't see why having two connection
factories should be a problem. I actually quite like the idea of the producer and the
receiver having different instances, that way they aren't related at all.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:55 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox commented on JBMESSAGING-1702:
--------------------------------------
Of course, every time you do a JNDI lookup a distinct object instance will be returned, so
this makes sense now.
There's nothing necessarily wrong with having multiple connection factorys but you
should be aware that connection multiplexing happens at a per connection factory level.
Each connection factory maintains a pool of underlying TCP connections, the default max
pool size is 8 (maxConnections parameter).
When you create JMS connections on a connection factory each JMS connection will use one
of those underlying TCP connections.
If you say create 1000 JMS connections using the same connection factory you will still
only have 8 TCP connections - each JMS connection uses one of the 8, so TCP connections
are shared by the JMS connections.
If you create a new connection factory for each connection then the multiplexing will be
lost and you might end up with thousands of TCP connections (depending on how many
connection factorys you create), which is not desirable.
In your case you are creating two connection factorys and registering the same exception
listener on both factorys. When a failure occurs simulataneously on two connections
obtained from the different factories it causes the deadlock, which i have replicated in a
simple test.
To workaround the issue you could just use one connection factory as opposed to 2, but I
will try and fix it so a deadlock is avoided even in the case of 2 connection factorys.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:14 p.m.
New subject: [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Adrian Woodhead commented on JBMESSAGING-1702:
----------------------------------------------
I understand all your comments except this one:
"In your case you are creating two connection factorys and registering the same
exception listener on both factorys. When a failure occurs simulataneously on two
connections obtained from the different factories it causes the deadlock, which i have
replicated in a simple test. "
I am creating 2 connection factories, one in the producer, one in the receiver, in each of
these I then get a connection from its own connection factory and do this:
connection.setExceptionListener(this);
where in one case "this" is the receiver and in the other case "this"
is the producer. So that would be an exception listener per factory. I see the receiver
performing the onException() logic sometimes at a different time from the producer and
they are totally different object instances so I don't see why this would be seen as
one exception listener?
As to your other comments - I think we're OK, the receiver creates one connection on
startup and uses this to receive messages. The Producer does the same and its connection
is used to send all messages (they are queued up and sent once per second). So I'm
actually only using 2 connections so there shouldn't be any issues related to the
pools, multiplexing etc.
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:11 p.m.
New subject: [JBoss JIRA] Closed: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()
[
https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.ji...
]
Tim Fox closed JBMESSAGING-1702.
--------------------------------
Resolution: Done
I believe I have fixed this by
a) Executing exception listeners on a different thread
b) Ensuring no more than one thread calls the exception listener concurrently
deadlock on application shutdown calling connection.close()
-----------------------------------------------------------
Key: JBMESSAGING-1702
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 2.0.0.beta4
Reporter: Adrian Woodhead
Assignee: Tim Fox
Fix For: 2.0.0
Attachments: BaseJMSObject.java, QueueingProducer.java
I have a client application which has a single producer and receiver. These are created
as singletons by a Spring application context. I define a shutdown method in my producer
which in turn calls connection.close(). This shutdown method is called by Spring when the
application is shutdown. This used to work fine when my Producer used ActiveMQ, however
with JBM it leads to deadlock and my application no longer shuts down cleanly. I have
included the deadlock stack trace from jstack below. To me what seems to be going wrong is
that while my shutdown method is calling connection.close() two other threads have called
my onException handler (which calls connection.close() too). So it seems as if these 3
threads are now fighting over trying to close the connection. Am I doing something wrong
here or should this work? When I do the shutdown I don't see any output from
onException (the first thing I do is log a message) so is it possible that these threads
are left over from some earlier onException() call? If so, then why are they blocked? I
guess a simple solution could be to not call close() when my application shuts down but
I'd prefer to do a clean close if possible.
Deadlock stack:
Found one Java-level deadlock:
=============================
"Thread-8":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
"Thread-179 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a
java.lang.Object),
which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
"Thread-138 (group:JBM-client-global-threads-1304545579)":
waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a
org.jboss.messaging.jms.client.JBossConnection),
which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
Java stack information for the threads listed above:
===================================================
"Thread-8":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
at
org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
at
org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
at
org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
at
org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
"Thread-179 (group:JBM-client-global-threads-1304545579)":
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at
org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6b4530> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a
org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a
org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
- locked <0x00007f451b6af088> (a java.lang.Object)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at
org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at
org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5440
days inactive
5441
days old
12 comments
2 participants
participants (2)
-
Adrian Woodhead (JIRA) -
Tim Fox (JIRA)