[jboss-jira] [JBoss JIRA] Created: (JBMESSAGING-1825) Deadlock in client Failover

[Yong Hao Gao (JIRA)]

Deadlock in client Failover
---------------------------

                 Key: JBMESSAGING-1825
                 URL: https://jira.jboss.org/browse/JBMESSAGING-1825
             Project: JBoss Messaging
          Issue Type: Bug
          Components: JMS Clustering
    Affects Versions: 1.4.7.GA, 1.4.0.SP3.CP10
            Reporter: Yong Hao Gao
            Assignee: Yong Hao Gao
             Fix For: 1.4.0.SP3.CP11, 1.4.8.GA


Client failover can be triggered by two possible ways. One is by the Remoting ConnectionValidator when a ping timeout is detected. The other is by an remote invocation on the failing connection, like createConsumer().

This two ways happened independently and concurrently. A chance of deadlock exists when both happens at the same time, here is the details:

1. A client's connection to a server is broken when it is trying to create a consumer.
2. The FailoverValveInterceptor intercepts this remoting call, finding that the call cannot go through the connection, and then initiates the failover process, in which
3. It closes the valve (preventing other's from entering).  Then 
4. after a new connection delegate has been created, it tries to set the JMS exception listener (if any), using the following code:

            if(remotingConnection.getConnectionListener() != null &&
                    remotingConnection.getConnectionListener().getJMSExceptionListener() != null)
            {
               log.trace("Adding Exception Listener to new connection");
               newDelegate.setExceptionListener(remotingConnection.getConnectionListener().getJMSExceptionListener());
            }
 
Where the getConnectionListener() is a synchronized method of ConsolidatedRemotingConnectionListener. So the first thing in this call is grab the lock. Meanwhile at the other side,

5. a ping times out, it notifies the JBM listener ConsolidatedRemotingConnectionListener by calling handleConnectionException(Throwable throwable, Client client) method (which is a synchronous method), in which it calls FailoverCommandCenter.failureDetected() in which it also tries to close the valve. As the valve has been closed in step 3, it will wait for the lock to be releases. However, while it is waiting, I has already locked the ConsolidatedRemotingConnectionListener
(as it is inside the synchronous method handleConnectionException(Throwable throwable, Client client)). Thus

One party closed the value and tries to grab a lock, the other owned the lock and waits on the valve -- A deadlock heppened.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira