[jbosscache-issues] [JBoss JIRA] (JBCACHE-1620) Race condition when automatically creating parent nodes during locking fails to get lock

[Dennis Reed (JIRA)]

    [ https://issues.jboss.org/browse/JBCACHE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709551#comment-12709551 ] 

Dennis Reed commented on JBCACHE-1620:
--------------------------------------

One failure scenario this can cause:

Several HTTP requests are processed at the same time as soon as the node is started.

Thread1 wants to write /JSESSION/ib_localhost/foo
Thread2 wants to write /JSESSION/ib_localhost/bar

Thread1 tries to lock /JSESSION/ib_localhost/foo
Thread2 tries to lock /JSESSION/ib_localhost/bar

The lock call is handled by JBoss Cache's PessimisticNodeBasedLockManager
It loops through trying to get locks on /, /JSESSION, /JSESSION/ib_localhost, and /JSESSION/ib_localhost/xxx.

If the node already exists, it tries to get a read lock (except for /JSESSION/ib_localhost/xxx where it gets a write lock).
If the node doesn't exist, it creates it and gets a write lock.

The bug/race condition is in PessimisticUnversionedNode#addChildAndAcquireLock
If two threads try to create the same node at the exact same time, one thread creates it and gets a write lock.  
The other thread returns the previously created node and DOES NOT LOCK IT.

So:
At startup, only the node / exists in the cache.

Thread1 gets a read lock on /
Thread2 gets a read lock on /

Thread1 sees that /JSESSION does not exist, and calls addChildAndAcquireLock
Thread2 sees that /JSESSION does not exist, and calls addChildAndAcquireLock

Thread1 creates the node and gets a write lock
Thread2 returns the node created by Thread1 without a lock

Thread1 sees that /JSESSION/ib_localhost does not exist, and calls addChildAndAcquireLock
Thread2 sees that /JSESSION/ib_localhost does not exist, and calls addChildAndAcquireLock

Thread2 creates the node and gets a write lock
Thread1 returns the node created by Thread2 without a lock

At this point the locks are:
Thread1:
    / read
    /JSESSION write
    /JSESSION/ib_localhost none
Thread2:
    / read
    /JSESSION none
    /JSESSION/ib_localhost write

Both threads then create their /JSESSION/ib_localhost/xxx and populate the data.
Both threads then make another cache call that need the same locks.

Thread1 already has a read lock on /, and a write lock on /JSESSION.  It wants a read lock on /JSESSION/ib_localhost but has to wait on Thread2's write lock
Thread2 already has a read lock on /.  It wants a read lock on /JSESSION but has to wait on Thread1's write lock.

DEADLOCK
                
> Race condition when automatically creating parent nodes during locking fails to get lock
> ----------------------------------------------------------------------------------------
>
>                 Key: JBCACHE-1620
>                 URL: https://issues.jboss.org/browse/JBCACHE-1620
>             Project: JBoss Cache
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Locking
>    Affects Versions: 3.2.9.GA
>            Reporter: Dennis Reed
>            Assignee: Dennis Reed
>
> Bug/race condition in PessimisticUnversionedNode#addChildAndAcquireLock.
> If two threads try to create the same node at the exact same time, one thread creates it and gets a write lock.  
> The other thread does not create a node, but returns the previously created node and does not lock it.
> The only caller PessimisticNodeBasedLockManager assumes the node was both created and locked after the call returns.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira