[jboss-jira] [JBoss JIRA] Updated: (JBCACHE-814) Concurrency issue with eviction

[Brian Stansberry (JIRA)]

     [ http://jira.jboss.com/jira/browse/JBCACHE-814?page=all ]

Brian Stansberry updated JBCACHE-814:
-------------------------------------

        Summary: Concurrency issue with eviction  (was: Concurrency issue with passivation)
    Description: 
If the eviction thread attempts to evict a node roughly concurrently with a get() call, it is quite easy for the get() to return null even though the evicted node is stored in a cache loader with data.

Tests org.jboss.cache.passivation.ConcurrentPassivationTest and org.jboss.cache.eviction.ConcurrentEvictionTest show the problem.

I believe the problem results from having the PessimisticLockInterceptor in the chain after the Activation/CacheLoader and Passivation/CacheStore interceptors. Potential cause is a race to the PessimisticLockInterceptor between the eviction thread and an application thread doing a get().

1) Eviction thread wins race, acquires a WL on a node.
2) Application thread is trailing in the race but proceeds through the cache loading interceptors w/o loading data, because the desired node is still in the cache.
3) Application thread blocks waiting for eviction thread's WL to be released.
4) Eviction thread removes the node from the tree structure.
5) Eviction thread releases the WL.  Application thread acquires a RL (it already had a ref to the node).
6) Application thread gets to the cache _get method, which can't find the node in the tree structure, and thus returns null.

Another variation on the same theme:

1) Eviction thread wins race, acquires a WL on a node.
2) Application thread is trailing in the race but proceeds through the cache loading interceptors w/o loading data, because the desired node is still in the cache.
3) Eviction thread removes the node from the tree structure.
4) Application thread gets to PessimisticLockInterceptor, which can't find the node. No need to acquire lock on non-existent node, so thread proceeds.
5) Application thread gets to the cache _get method, which can't find the node in the tree structure, and thus returns null.

  was:
If the eviction thread attempts to passivate a node roughly concurrently with a get() call, it is quite easy for the get() to return null even though the passivated node has data.

Test org.jboss.cache.passivation.ConcurrentPassivationTest shows the problem.

I believe the problem results from having the PessimisticLockInterceptor in the chain after the Activation and Passivation interceptors. Potential cause is a race to the PessimisticLockInterceptor between the eviction thread and an application thread doing a get().

1) Eviction thread wins race, acquires a WL on a node.
2) Application thread is trailing in the race but proceeds through the Activation/Passivation interceptors w/o loading data, because the desired node is still in the cache.
3) Application thread blocks waiting for eviction thread's WL to be released.
4) Eviction thread removes the node from the tree structure.
5) Eviction thread releases the WL.  Application thread acquires a RL (it already had a ref to the node).
6) Application thread gets to the cache _get method, which can't find the node in the tree structure, and thus returns null.

Another variation on the same theme:

1) Eviction thread wins race, acquires a WL on a node.
2) Application thread is trailing in the race but proceeds through the Activation/Passivation interceptors w/o loading data, because the desired node is still in the cache.
3) Eviction thread removes the node from the tree structure.
4) Application thread gets to PessimisticLockInterceptor, which can't find the node. No need to acquire lock on non-existent node, so thread proceeds.
5) Application thread gets to the cache _get method, which can't find the node in the tree structure, and thus returns null.


Updated to reflect that the issue exists whether or not passivation is used.

> Concurrency issue with eviction
> -------------------------------
>
>                 Key: JBCACHE-814
>                 URL: http://jira.jboss.com/jira/browse/JBCACHE-814
>             Project: JBoss Cache
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>          Components: Cache loaders
>    Affects Versions: 1.4.0.SP1
>            Reporter: Brian Stansberry
>         Assigned To: Manik Surtani
>
> If the eviction thread attempts to evict a node roughly concurrently with a get() call, it is quite easy for the get() to return null even though the evicted node is stored in a cache loader with data.
> Tests org.jboss.cache.passivation.ConcurrentPassivationTest and org.jboss.cache.eviction.ConcurrentEvictionTest show the problem.
> I believe the problem results from having the PessimisticLockInterceptor in the chain after the Activation/CacheLoader and Passivation/CacheStore interceptors. Potential cause is a race to the PessimisticLockInterceptor between the eviction thread and an application thread doing a get().
> 1) Eviction thread wins race, acquires a WL on a node.
> 2) Application thread is trailing in the race but proceeds through the cache loading interceptors w/o loading data, because the desired node is still in the cache.
> 3) Application thread blocks waiting for eviction thread's WL to be released.
> 4) Eviction thread removes the node from the tree structure.
> 5) Eviction thread releases the WL.  Application thread acquires a RL (it already had a ref to the node).
> 6) Application thread gets to the cache _get method, which can't find the node in the tree structure, and thus returns null.
> Another variation on the same theme:
> 1) Eviction thread wins race, acquires a WL on a node.
> 2) Application thread is trailing in the race but proceeds through the cache loading interceptors w/o loading data, because the desired node is still in the cache.
> 3) Eviction thread removes the node from the tree structure.
> 4) Application thread gets to PessimisticLockInterceptor, which can't find the node. No need to acquire lock on non-existent node, so thread proceeds.
> 5) Application thread gets to the cache _get method, which can't find the node in the tree structure, and thus returns null.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira