[jboss-jira] [JBoss JIRA] Commented: (JBMESSAGING-1895) Deadlock between failover and transaction rolling back

[Yong Hao Gao (JIRA)]

    [ https://issues.jboss.org/browse/JBMESSAGING-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625144#comment-12625144 ] 

Yong Hao Gao commented on JBMESSAGING-1895:
-------------------------------------------

To fix the problem:

Release the 'failoverLock' if the MDB has already been closed (removed) during rolling back. That will make thread 2 not wait for the valve and therefore avoid the deadlock.

> Deadlock between failover and transaction rolling back
> ------------------------------------------------------
>
>                 Key: JBMESSAGING-1895
>                 URL: https://issues.jboss.org/browse/JBMESSAGING-1895
>             Project: JBoss Messaging
>          Issue Type: Bug
>          Components: JMS Clustering
>    Affects Versions: 1.4.0.SP3.CP14, 1.4.8.SP2
>            Reporter: Yong Hao Gao
>            Assignee: Yong Hao Gao
>             Fix For: 1.4.0.SP3.CP15, 1.4.8.SP3
>
>
> To get this dead lock, failover must happen in strict order of the following events:
> first, one thread detects the failure and performs failover. It closes the FailoverValve2 object in order to let all other threads to block until it finishes failover.
> Then the thread goes on to create a new connection and then synchronizes all the states with the new connection. When synchronizing session state, it needs to grab a 'failoverLock' (See SessionState.synchronizeWith() method). This failoverLock was introduced recently to fix one of the duplicate message issues found by you.
> On the other hand, another thread tries to commit a two-phase transaction but fails at preparing stage. So it goes on to roll back the transaction, which grabs the 'failoverLock' before the first thread gets it. Then because the FailoverValve2 has been closed by the first thread, then this thread has to wait on the FailoverValve2
> until the first thread finishes the failover job.
> So the condition is: thread 1 closes the valve and wait for failoverLock, which is held by thread 2. Thread 2 is waiting on the valve which is closed and held by thread 1. The deadlock happens.
> There is one more condition needed, i. e. when thread 2 is rolling back the transaction, the message handler (MDB) must have been closed already. This will force the message to be rolled back to the server, which is intercepted by the FailoverValve2, causing the thread wait on the valve. If by the roll back time the MDB still there, the deadlock won't happen.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira