[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1895) Deadlock between failover and transaction rolling back

Deadlock between failover and transaction rolling back ------------------------------------------------------ Key: JBMESSAGING-1895 URL: https://issues.jboss.org/browse/JBMESSAGING-1895 Project: JBoss Messaging Issue Type: Bug Components: JMS Clustering Affects Versions: 1.4.0.SP3.CP14, 1.4.8.SP2 Reporter: Yong Hao Gao Assignee: Yong Hao Gao Fix For: 1.4.0.SP3.CP15, 1.4.8.SP3 To get this dead lock, failover must happen in strict order of the following events: first, one thread detects the failure and performs failover. It closes the FailoverValve2 object in order to let all other threads to block until it finishes failover. Then the thread goes on to create a new connection and then synchronizes all the states with the new connection. When synchronizing session state, it needs to grab a 'failoverLock' (See SessionState.synchronizeWith() method). This failoverLock was introduced recently to fix one of the duplicate message issues found by you. On the other hand, another thread tries to commit a two-phase transaction but fails at preparing stage. So it goes on to roll back the transaction, which grabs the 'failoverLock' before the first thread gets it. Then because the FailoverValve2 has been closed by the first thread, then this thread has to wait on the FailoverValve2 until the first thread finishes the failover job. So the condition is: thread 1 closes the valve and wait for failoverLock, which is held by thread 2. Thread 2 is waiting on the valve which is closed and held by thread 1. The deadlock happens. There is one more condition needed, i. e. when thread 2 is rolling back the transaction, the message handler (MDB) must have been closed already. This will force the message to be rolled back to the server, which is intercepted by the FailoverValve2, causing the thread wait on the valve. If by the roll back time the MDB still there, the deadlock won't happen.