[jbossts-issues] [JBoss JIRA] (JBTM-1350) Deadlock in LockManager

Michael Musgrove (JIRA) jira-events at lists.jboss.org
Wed Mar 20 07:23:43 EDT 2013


    [ https://issues.jboss.org/browse/JBTM-1350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762046#comment-12762046 ] 

Michael Musgrove edited comment on JBTM-1350 at 3/20/13 7:22 AM:
-----------------------------------------------------------------

The same problem occurs when the commit request comes in via the call com.arjuna.ats.internal.jts.orbspecific.interposition.resources.arjuna.ServerNestedAction.commit_subtransaction (I am attaching the stacktraces)

With this route no transaction is associated with the current thread and the deadlock is:

* Thread 1: LockManager propagate is called (via commit) which takes two locks
 Lock1: associated action or StateManager::mutex
 Lock2: LockManager::locksHeldLockObject


Thread 2: LockManager setlock is called (via a remote client invoking a get operation on HammerObject which extends LockManager)
 Lock1: associated action or StateManager::mutex
 Lock2: LockManager::locksHeldLockObject
 Lock3: StateManager::mutex

So if Thread 1 does not have the action associated with the thread but Thread 2 does then we have a classic de
adlock. This happens when Thread 1 comes in via ServerNestedAction.commit_subtransaction. Rather than fixing i
t be associated the transaction with the thread the better solution as follows: 

- force LockManager propagate to take the StateManager::mutex

This nearly works but now we have a lock ordering problem when using intrinsic locks on StateManager (synchronized method calls). 

Thread 1: StateManager::forgetAction
 Lock1: intrinsic lock on HammerObject (synchronized boolean forgetAction)
 Lock2: StateManager::mutex

Thread 2: StateManager::activate (via LockManager setlock) is called (via a remote client invoking a get operation on HammerObject which extends LockManager)
 Lock1: associated action or StateManager::mutex
 Lock2: LockManager::locksHeldLockObject
 Lock3: StateManager::mutex
 Lock4: intrinsic lock on HammerObject (synchronized boolean activate)

So the two threads are taking the intrinsic lock on HammerObject and StateManager::mutex is different orders causing another deadlock.

The solution for this problem is to make sure that the intrinsic lock on HammerObject is taken in the correct order (ie in LockManager::propagate and LockManager::setlock). This appears to resolve the JIRA.

                
      was (Author: mmusgrov):
    The same problem occurs when the commit request comes in via the call
com.arjuna.ats.internal.jts.orbspecific.interposition.resources.arjuna.ServerNestedAction.commit_subtransaction (I am attaching the stacktraces)

With this route no transaction is associated with the current thread and the deadlock is:

Thread 1: LockManager propagate is called (via commit) which takes two locks
 Lock1: associated action or StateManager::mutex
 Lock2: LockManager::locksHeldLockObject


Thread 2: LockManager setlock is called (via a remote client invoking a get operation on HammerObject which extends LockManager)
 Lock1: associated action or StateManager::mutex
 Lock2: LockManager::locksHeldLockObject
 Lock3: StateManager::mutex

So if Thread 1 does not have the action associated with the thread but Thread 2 does then we have a classic de
adlock. This happens when Thread 1 comes in via ServerNestedAction.commit_subtransaction. Rather than fixing i
t be associated the transaction with the thread the better solution as follows: 

- force LockManager propagate to take the StateManager::mutex

This nearly works but now we have a lock ordering problem when using intrinsic locks on StateManager (synchronized method calls). 

Thread 1: StateManager::forgetAction
 Lock1: intrinsic lock on HammerObject (synchronized boolean forgetAction)
 Lock2: StateManager::mutex

Thread 2: StateManager::activate (via LockManager setlock) is called (via a remote client invoking a get operation on HammerObject which extends LockManager)
 Lock1: associated action or StateManager::mutex
 Lock2: LockManager::locksHeldLockObject
 Lock3: StateManager::mutex
 Lock4: intrinsic lock on HammerObject (synchronized boolean activate)

So the two threads are taking the intrinsic lock on HammerObject and StateManager::mutex is different orders causing another deadlock.

The solution for this problem is to make sure that the intrinsic lock on HammerObject is taken in the correct order (ie in LockManager::propagate and LockManager::setlock). This appears to resolve the JIRA.

                  
> Deadlock in LockManager
> -----------------------
>
>                 Key: JBTM-1350
>                 URL: https://issues.jboss.org/browse/JBTM-1350
>             Project: JBoss Transaction Manager
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Transaction Core
>    Affects Versions: 5.0.0.M1, 4.17.2
>            Reporter: Michael Musgrove
>            Assignee: Michael Musgrove
>             Fix For: 4.17.3, 5.0.0.M2
>
>         Attachments: deadlock_via_intrinsic_lock, deadlock_via_ServerNestedAction, jstack.16785
>
>
> A deadlock can occur whilst calling LockManager.setlock if another thread tries to commit a transaction that has the same lock manager as a participant. The attached java stack dump, jstack.16785, (from test com.hp.mwtests.ts.jts.remote.hammer.DistributedHammer2) shows an example. It shows two threads interacting with a remote HammerObject:
> - Thread 1 updates the remote object;
> - Thread 2 commits a transaction that has the same HammerObject instance as a participant;
> Thread 1 calls setlock on HammerObject which synchronizes on BasicAction.Current() and LockManager.locksHeldLockObject and then activates the object (which triggers an object load from the object store). The activate call tries to lock StateManager.mutex and this is where thread 1 deadlocks.
> Meanwhile a commit request is issued which results in Thread 2 running at the same time thread 1 is calling activate. The commit asks HammerObject to prepare and commit. The participant commit asks HammerObject to release any locks it has held (LockManager.releaseAll). This call first tries to lock BasicAction.Current() which is null during commit time (since commit disassociates the transaction from the thread before committing the participants). Instead it locks StateManager.mutex which succeeds. Then it tries to lock LockManager.locksHeldLockObject and that is where Thread 2 deadlocks.
> Note that the problem arises because Thread 2 gets null when it calls BasicAction.Current() and instead locks StateManager.mutex. If BasicAction.Current() was not null thread 2 would try to lock it but would not obtain the lock until thread 1 had completed the activate call and released BasicAction.Current() thus allowing thread 2 to continue as normal.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


More information about the jbossts-issues mailing list