[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1702) deadlock on application shutdown calling connection.close()

Tim Fox (JIRA) jira-events at lists.jboss.org
Thu Aug 6 09:27:29 EDT 2009 

    [ https://jira.jboss.org/jira/browse/JBMESSAGING-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12479249#action_12479249 ] 

Tim Fox commented on JBMESSAGING-1702:
--------------------------------------

The two different conection factory instances can be seen as follows:

"Thread-179 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
- locked <0x00007f451b6af078> (a java.lang.Object)
at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6b0780> (a org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)

**** this is the failoverLock in the ConnectionManagerImpl class
- locked <0x00007f451b6b4530> (a java.lang.Object)

at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)


And:

"Thread-138 (group:JBM-client-global-threads-1304545579)":
at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
- waiting to lock <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
- locked <0x00007f451b6a8f10> (a org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)

*** This is also the failoverLock but in a different instance of ConnectionManagerImpl

- locked <0x00007f451b6af088> (a java.lang.Object)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)


at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619) 

Note that the ids for monitors highlighted above are different - this means you have 2 distinct ConnectionManagerImpl instances.

Each ConnectionManagerImpl instance is maintained by a ConnectionFactory so you must have (at least) two connection factorys instance.

I don't know how you've got 2, but you've definitely got 2.

 


> deadlock on application shutdown calling connection.close()
> -----------------------------------------------------------
>
>                 Key: JBMESSAGING-1702
>                 URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1702
>             Project: JBoss Messaging
>          Issue Type: Bug
>    Affects Versions: 2.0.0.beta4
>            Reporter: Adrian Woodhead
>            Assignee: Tim Fox
>             Fix For: 2.0.0
>
>         Attachments: BaseJMSObject.java, QueueingProducer.java
>
>
> I have a client application which has a single producer and receiver. These are created as singletons by a Spring application context. I define a shutdown method in my producer which in turn calls connection.close(). This shutdown method is called by Spring when the application is shutdown. This used to work fine when my Producer used ActiveMQ, however with JBM it leads to deadlock and my application no longer shuts down cleanly. I have included the deadlock stack trace from jstack below. To me what seems to be going wrong is that while my shutdown method is calling connection.close() two other threads have called my onException handler (which calls connection.close() too). So it seems as if these 3 threads are now fighting over trying to close the connection. Am I doing something wrong here or should this work? When I do the shutdown I don't see any output from onException (the first thing I do is log a message) so is it possible that these threads are left over from some earlier onException() call? If so, then why are they blocked? I guess a simple solution could be to not call close() when my application shuts down but I'd prefer to do a clean close if possible.
> Deadlock stack:
> Found one Java-level deadlock:
> =============================
> "Thread-8":
>   waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a org.jboss.messaging.jms.client.JBossConnection),
>   which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
> "Thread-179 (group:JBM-client-global-threads-1304545579)":
>   waiting to lock monitor 0x00000000006c4220 (object 0x00007f451b6af088, a java.lang.Object),
>   which is held by "Thread-138 (group:JBM-client-global-threads-1304545579)"
> "Thread-138 (group:JBM-client-global-threads-1304545579)":
>   waiting to lock monitor 0x00000000006c4370 (object 0x00007f451b6a8f30, a org.jboss.messaging.jms.client.JBossConnection),
>   which is held by "Thread-179 (group:JBM-client-global-threads-1304545579)"
> Java stack information for the threads listed above:
> ===================================================
> "Thread-8":
> 	at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
> 	- waiting to lock <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
> 	at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
> 	at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
> 	at fm.last.jms.QueueingProducer.shutdown(QueueingProducer.java:93)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:208)
> 	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:165)
> 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487)
> 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:462)
> 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:430)
> 	- locked <0x00007f451b55f560> (a java.util.LinkedHashMap)
> 	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:853)
> 	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:831)
> 	at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:764)
> "Thread-179 (group:JBM-client-global-threads-1304545579)":
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:436)
> 	- waiting to lock <0x00007f451b6af088> (a java.lang.Object)
> 	- locked <0x00007f451b6af078> (a java.lang.Object)
> 	at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1262)
> 	at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:686)
> 	at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:278)
> 	at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
> 	- locked <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
> 	at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
> 	at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
> 	at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
> 	at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
> 	at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
> 	- locked <0x00007f451b6b0780> (a org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
> 	- locked <0x00007f451b6b4530> (a java.lang.Object)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> 	at java.lang.Thread.run(Thread.java:619)
> "Thread-138 (group:JBM-client-global-threads-1304545579)":
> 	at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:250)
> 	- waiting to lock <0x00007f451b6a8f30> (a org.jboss.messaging.jms.client.JBossConnection)
> 	at fm.last.jms.BaseJMSObject.disconnect(BaseJMSObject.java:108)
> 	at fm.last.jms.QueueingProducer.disconnect(QueueingProducer.java:86)
> 	at fm.last.jms.BaseJMSObject.reconnect(BaseJMSObject.java:117)
> 	at fm.last.jms.BaseJMSObject.onException(BaseJMSObject.java:141)
> 	at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
> 	- locked <0x00007f451b6a8f10> (a org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failoverOrReconnect(ConnectionManagerImpl.java:660)
> 	- locked <0x00007f451b6af088> (a java.lang.Object)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.handleConnectionFailure(ConnectionManagerImpl.java:506)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.access$600(ConnectionManagerImpl.java:80)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener.connectionFailed(ConnectionManagerImpl.java:1194)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
> 	at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
> 	at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction$1.run(ConnectionManagerImpl.java:1150)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> 	at java.lang.Thread.run(Thread.java:619)
> Found 1 deadlock.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

More information about the jboss-jira mailing list