[jboss-jira] [JBoss JIRA] Resolved: (JBMESSAGING-1825) Deadlock in client Failover

Deadlock in client Failover --------------------------- Key: JBMESSAGING-1825 URL: https://issues.jboss.org/browse/JBMESSAGING-1825 Project: JBoss Messaging Issue Type: Bug Components: JMS Clustering Affects Versions: 1.4.0.SP3.CP10, 1.4.7.GA Reporter: Yong Hao Gao Assignee: Yong Hao Gao Fix For: 1.4.0.SP3.CP12, 1.4.8.GA Client failover can be triggered by two possible ways. One is by the Remoting ConnectionValidator when a ping timeout is detected. The other is by an remote invocation on the failing connection, like createConsumer(). This two ways happened independently and concurrently. A chance of deadlock exists when both happens at the same time, here is the details: 1. A client's connection to a server is broken when it is trying to create a consumer. 2. The FailoverValveInterceptor intercepts this remoting call, finding that the call cannot go through the connection, and then initiates the failover process, in which 3. It closes the valve (preventing other's from entering). Then 4. after a new connection delegate has been created, it tries to set the JMS exception listener (if any), using the following code: if(remotingConnection.getConnectionListener() != null && remotingConnection.getConnectionListener().getJMSExceptionListener() != null) { log.trace("Adding Exception Listener to new connection"); newDelegate.setExceptionListener(remotingConnection.getConnectionListener().getJMSExceptionListener()); } Where the getConnectionListener() is a synchronized method of ConsolidatedRemotingConnectionListener. So the first thing in this call is grab the lock. Meanwhile at the other side, 5. a ping times out, it notifies the JBM listener ConsolidatedRemotingConnectionListener by calling handleConnectionException(Throwable throwable, Client client) method (which is a synchronous method), in which it calls FailoverCommandCenter.failureDetected() in which it also tries to close the valve. As the valve has been closed in step 3, it will wait for the lock to be releases. However, while it is waiting, I has already locked the ConsolidatedRemotingConnectionListener (as it is inside the synchronous method handleConnectionException(Throwable throwable, Client client)). Thus One party closed the value and tries to grab a lock, the other owned the lock and waits on the valve -- A deadlock heppened.