[jboss-jira] [JBoss JIRA] (JBMESSAGING-1955) Deadlock on failover with client lease

Deadlock on failover with client lease -------------------------------------- Key: JBMESSAGING-1955 URL: https://issues.jboss.org/browse/JBMESSAGING-1955 Project: JBoss Messaging Issue Type: Feature Request Components: Messaging Core Distributed Support Affects Versions: 1.4.8.SP5 Environment: JBoss EAP 5.1.2 Reporter: Doug Grove Assignee: Doug Grove JBoss is configured as an 8 node cluster. This cluster is for JMS messaging only. Other JBoss instances host MDBs. When failover occurs in the 8 node cluster, a deadlock is seen on the JBoss instances that host the MDBs. {code} Found one Java-level deadlock: ============================= "Thread-177": waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object), which is held by "Thread-130" "Thread-130": waiting to lock monitor 0x08aea1a4 (object 0x81dde148, a java.lang.Object), which is held by "Timer-18" "Timer-18": waiting to lock monitor 0x4c561e18 (object 0x7c3dfdb0, a java.lang.Object), which is held by "Thread-130" Java stack information for the threads listed above: =================================================== "Thread-177": at org.jboss.remoting.Client.removeConnectionListener(Client.java:599) - waiting to lock <0x7c3dfdb0> (a java.lang.Object) at org.jboss.jms.client.remoting.JMSRemotingConnection.removeConnectionListener(JMSRemotingConnection.java:565) - locked <0x7f589740> (a org.jboss.jms.client.remoting.JMSRemotingConnection) at org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:183) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java) at org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132) at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardownConnection(JmsActivation.java:635) at org.jboss.resource.adapter.jms.inflow.JmsActivation.teardown(JmsActivation.java:376) at org.jboss.resource.adapter.jms.inflow.JmsActivation.handleFailure(JmsActivation.java:275) at org.jboss.resource.adapter.jms.inflow.JmsActivation.onException(JmsActivation.java:312) at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:120) - locked <0x7f320410> (a java.lang.Object) - locked <0x7f5899a0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener) at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524) "Thread-130": at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:507) - waiting to lock <0x81dde148> (a java.lang.Object) at org.jboss.remoting.Client.setupClientLease(Client.java:2056) - locked <0x7c3dfdb0> (a java.lang.Object) at org.jboss.remoting.Client.connect(Client.java:1918) at org.jboss.remoting.Client.connect(Client.java:737) at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:175) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_25018031.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_25018031.java) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java) at org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java) at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129) at org.jboss.jms.client.container.ConnectionFailureListener.handleConnectionException(ConnectionFailureListener.java:62) at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:84) - locked <0x813598d0> (a org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener) at org.jboss.remoting.ConnectionValidator$3.run(ConnectionValidator.java:524) "Timer-18": at org.jboss.remoting.Client.notifyListeners(Client.java:1873) - waiting to lock <0x7c3dfdb0> (a java.lang.Object) at org.jboss.remoting.LeasePinger.stopPing(LeasePinger.java:134) at org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:434) - locked <0x81dde148> (a java.lang.Object) at org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask.run(ConnectionValidator.java:1081) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Found 1 deadlock. {code}