[jboss-jira] [JBoss JIRA] (JBMESSAGING-1955) Deadlock on failover with client lease

Doug Grove (JIRA) issues at jboss.org
Sun Sep 14 12:00:05 EDT 2014 

     [ https://issues.jboss.org/browse/JBMESSAGING-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Grove updated JBMESSAGING-1955:
------------------------------------
    Description: 
JBoss is configured as an 8 node cluster.  This cluster is for JMS messaging only.  Other JBoss instances host MDBs.  When failover occurs in the 8 node cluster, a deadlock is seen on the JBoss instances that host the MDBs.

{code}
Found one Java-level deadlock:
=============================
"Thread-177":
  waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object),
  which is held by "Thread-130"
"Thread-130":
  waiting to lock monitor 0x08aea1a4 (object 0x81dde148, a java.lang.Object),
  which is held by "Timer-18"
"Timer-18":
  waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object),
  which is held by "Thread-130"

Java stack information for the threads listed above:
===================================================
"Thread-177":
	at org.jboss.remoting.Client.removeConnectionListener(Client.java:599)
	- waiting to lock <0x7c3dfdb0> (a java.lang.Object)
	at org.jboss.jms.client.remoting.JMSRemotingConnection.removeConnectionListener(JMSRemotingConnection.java:565)
	- locked <0x7f589740> (a org.jboss.jms.client.remoting.JMSRemotingConnection)
	at org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:183)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)
	at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardownConnection(JmsActivation.java:635)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardown(JmsActivation.java:376)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.handleFailure(JmsActivation.java:275)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.onException(JmsActivation.java:312)
	at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)
	- locked <0x7f320410> (a java.lang.Object)
	- locked <0x7f5899a0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener)
	at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524)
"Thread-130":
	at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:507)
	- waiting to lock <0x81dde148> (a java.lang.Object)
	at org.jboss.remoting.Client.setupClientLease(Client.java:2056)
	- locked <0x7c3dfdb0> (a java.lang.Object)
	at org.jboss.remoting.Client.connect(Client.java:1918)
	at org.jboss.remoting.Client.connect(Client.java:737)
	at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:175)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
	at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
	at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_25018031.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_25018031.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
	at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
	at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129)
	at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62)
	at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:84)
	- locked <0x813598d0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener)
	at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524)
"Timer-18":
	at org.jboss.remoting.Client.notifyListeners(Client.java:1873)
	- waiting to lock <0x7c3dfdb0> (a java.lang.Object)
	at org.jboss.remoting.LeasePinger.stopPing(LeasePinger.java:134)
	at org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:434)
	- locked <0x81dde148> (a java.lang.Object)
	at org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask.run(ConnectionValidator.java:1081)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)

Found 1 deadlock.
{code}


  was:
JBoss is configured as an 8 node cluster.  This cluster is for JMS messaging only.  Other JBoss instances host MDBs.  When failover occurs in the 8 node cluster, a deadlock is seen on the JBoss instances that host the MDBs.

{quote}
Found one Java-level deadlock:
=============================
"Thread-177":
  waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object),
  which is held by "Thread-130"
"Thread-130":
  waiting to lock monitor 0x08aea1a4 (object 0x81dde148, a java.lang.Object),
  which is held by "Timer-18"
"Timer-18":
  waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object),
  which is held by "Thread-130"

Java stack information for the threads listed above:
===================================================
"Thread-177":
	at org.jboss.remoting.Client.removeConnectionListener(Client.java:599)
	- waiting to lock <0x7c3dfdb0> (a java.lang.Object)
	at org.jboss.jms.client.remoting.JMSRemotingConnection.removeConnectionListener(JMSRemotingConnection.java:565)
	- locked <0x7f589740> (a org.jboss.jms.client.remoting.JMSRemotingConnection)
	at org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:183)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)
	at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardownConnection(JmsActivation.java:635)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardown(JmsActivation.java:376)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.handleFailure(JmsActivation.java:275)
	at org.jboss.resource.adapter.jms.inflow.JmsActivation.onException(JmsActivation.java:312)
	at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)
	- locked <0x7f320410> (a java.lang.Object)
	- locked <0x7f5899a0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener)
	at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524)
"Thread-130":
	at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:507)
	- waiting to lock <0x81dde148> (a java.lang.Object)
	at org.jboss.remoting.Client.setupClientLease(Client.java:2056)
	- locked <0x7c3dfdb0> (a java.lang.Object)
	at org.jboss.remoting.Client.connect(Client.java:1918)
	at org.jboss.remoting.Client.connect(Client.java:737)
	at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:175)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
	at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
	at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_25018031.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_25018031.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
	at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
	at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129)
	at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62)
	at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:84)
	- locked <0x813598d0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener)
	at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524)
"Timer-18":
	at org.jboss.remoting.Client.notifyListeners(Client.java:1873)
	- waiting to lock <0x7c3dfdb0> (a java.lang.Object)
	at org.jboss.remoting.LeasePinger.stopPing(LeasePinger.java:134)
	at org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:434)
	- locked <0x81dde148> (a java.lang.Object)
	at org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask.run(ConnectionValidator.java:1081)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)

Found 1 deadlock.
{quote}




> Deadlock on failover with client lease
> --------------------------------------
>
>                 Key: JBMESSAGING-1955
>                 URL: https://issues.jboss.org/browse/JBMESSAGING-1955
>             Project: JBoss Messaging
>          Issue Type: Feature Request
>          Components: Messaging Core Distributed Support
>    Affects Versions: 1.4.8.SP5
>         Environment: JBoss EAP 5.1.2
>            Reporter: Doug Grove
>
> JBoss is configured as an 8 node cluster.  This cluster is for JMS messaging only.  Other JBoss instances host MDBs.  When failover occurs in the 8 node cluster, a deadlock is seen on the JBoss instances that host the MDBs.
> {code}
> Found one Java-level deadlock:
> =============================
> "Thread-177":
>   waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object),
>   which is held by "Thread-130"
> "Thread-130":
>   waiting to lock monitor 0x08aea1a4 (object 0x81dde148, a java.lang.Object),
>   which is held by "Timer-18"
> "Timer-18":
>   waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object),
>   which is held by "Thread-130"
> Java stack information for the threads listed above:
> ===================================================
> "Thread-177":
> 	at org.jboss.remoting.Client.removeConnectionListener(Client.java:599)
> 	- waiting to lock <0x7c3dfdb0> (a java.lang.Object)
> 	at org.jboss.jms.client.remoting.JMSRemotingConnection.removeConnectionListener(JMSRemotingConnection.java:565)
> 	- locked <0x7f589740> (a org.jboss.jms.client.remoting.JMSRemotingConnection)
> 	at org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:183)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
> 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
> 	at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
> 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
> 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
> 	at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
> 	at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
> 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
> 	at org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)
> 	at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)
> 	at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardownConnection(JmsActivation.java:635)
> 	at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardown(JmsActivation.java:376)
> 	at org.jboss.resource.adapter.jms.inflow.JmsActivation.handleFailure(JmsActivation.java:275)
> 	at org.jboss.resource.adapter.jms.inflow.JmsActivation.onException(JmsActivation.java:312)
> 	at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120)
> 	- locked <0x7f320410> (a java.lang.Object)
> 	- locked <0x7f5899a0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener)
> 	at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524)
> "Thread-130":
> 	at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:507)
> 	- waiting to lock <0x81dde148> (a java.lang.Object)
> 	at org.jboss.remoting.Client.setupClientLease(Client.java:2056)
> 	- locked <0x7c3dfdb0> (a java.lang.Object)
> 	at org.jboss.remoting.Client.connect(Client.java:1918)
> 	at org.jboss.remoting.Client.connect(Client.java:737)
> 	at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368)
> 	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:175)
> 	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
> 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
> 	at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
> 	at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_25018031.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_25018031.java)
> 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
> 	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
> 	at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
> 	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
> 	at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
> 	at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129)
> 	at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62)
> 	at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:84)
> 	- locked <0x813598d0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener)
> 	at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524)
> "Timer-18":
> 	at org.jboss.remoting.Client.notifyListeners(Client.java:1873)
> 	- waiting to lock <0x7c3dfdb0> (a java.lang.Object)
> 	at org.jboss.remoting.LeasePinger.stopPing(LeasePinger.java:134)
> 	at org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:434)
> 	- locked <0x81dde148> (a java.lang.Object)
> 	at org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask.run(ConnectionValidator.java:1081)
> 	at java.util.TimerThread.mainLoop(Timer.java:512)
> 	at java.util.TimerThread.run(Timer.java:462)
> Found 1 deadlock.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.1#6329)

More information about the jboss-jira mailing list