[jboss-jira] [JBoss JIRA] Commented: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks

[Owen Taylor (JIRA)]

    [ http://jira.jboss.com/jira/browse/JBCACHE-794?page=comments#action_12344968 ] 
            
Owen Taylor commented on JBCACHE-794:
-------------------------------------

In terms of 1), right now, I'm skeptical that a system that hits the limit is in a "functional" state:

 - You get many, many warnings before then (from the 98% full check)
 - System performance will be awful, since all tree cache accessing threads will block until the eviction timer wakes up
 - You'll get this bug

But I suppose if this bug was fixed, that situation might be OK for a system that experienced a unexpected burst
of cache activity much greater than the normal load.

Anyways, I don't have a unit test case, and I'm not quite sure how to build one. Something like the following
perhaps:

 * Configure a cache with a LRU eviction policy and maxNodes of 1000
 * Continuously
   - Open a transaction
   - Write values to 1010 nodes
   - Close the transction

If that loop runs quickly enough:

 - The eviction queue will fill (producing a ton of warnings at 98% full)
 - The writing (main) thread will block
 - After a few seconds, the eviction thread will run, try try to evict some values from locked nodes, and deadlock, throwing a TimeoutException
 - The aggregate node-writing throughput after this happens won't be the expected
      200,000 every 5 seconds
   (the maximum rate we can clear the eviction queue), but will instead be about:
     1 every 15 seconds
   (the rate the lock times out)

There may be some subtle reasons why that doesn't quite work, but it should be close. 

> Eviction Queue hard limit can cause deadlocks
> ---------------------------------------------
>
>                 Key: JBCACHE-794
>                 URL: http://jira.jboss.com/jira/browse/JBCACHE-794
>             Project: JBoss Cache
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>    Affects Versions: 1.4.0.SP1
>            Reporter: Owen Taylor
>         Assigned To: Manik Surtani
>             Fix For: 2.0.0, 1.4.0.SP2
>
>
> if the eviction queue fills up, you can get the situation where:
>  - The eviction queue processing task calls evict() which needs to get a write lock on a node
>  - A different thread has a read lock on that node and is blocking to put something into the eviction queue
> Once this starts happening the evication queue thread is procesing events *very* slowly because
> it repeatedly stalls while the lock times out, so the queue will stay full.
> Without digging into the code too much, possible fixes that occur to me include:
>  1) Don't have a hard limit on the size of the eviction queue, and just warn when it gets too big
>  2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a node is
>       locked when we try to evict it, it probably shouldn't be evicted anyways.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira