2:27 p.m.
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Owen Taylor
Assigned To: Manik Surtani
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:01 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=comments#action_12344894 ]
Manik Surtani commented on JBCACHE-794:
---------------------------------------
1 is unfeasible as it can get very, very big in some cases. There is another feature
request somewhere (JBCACHE-684) to make this limit configurable.
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 2.0.0, 1.4.0.SP2
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:01 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=comments#action_12344895 ]
Manik Surtani commented on JBCACHE-794:
---------------------------------------
2 makes a lot more sense though - do you have a unit test for this scenario?
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 2.0.0, 1.4.0.SP2
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:01 a.m.
New subject: [JBoss JIRA] Updated: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=all ]
Manik Surtani updated JBCACHE-794:
----------------------------------
Fix Version/s: 1.4.0.SP2
2.0.0
Affects Version/s: 1.4.0.SP1
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 2.0.0, 1.4.0.SP2
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:45 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=comments#action_12344968 ]
Owen Taylor commented on JBCACHE-794:
-------------------------------------
In terms of 1), right now, I'm skeptical that a system that hits the limit is in a
"functional" state:
- You get many, many warnings before then (from the 98% full check)
- System performance will be awful, since all tree cache accessing threads will block
until the eviction timer wakes up
- You'll get this bug
But I suppose if this bug was fixed, that situation might be OK for a system that
experienced a unexpected burst
of cache activity much greater than the normal load.
Anyways, I don't have a unit test case, and I'm not quite sure how to build one.
Something like the following
perhaps:
* Configure a cache with a LRU eviction policy and maxNodes of 1000
* Continuously
- Open a transaction
- Write values to 1010 nodes
- Close the transction
If that loop runs quickly enough:
- The eviction queue will fill (producing a ton of warnings at 98% full)
- The writing (main) thread will block
- After a few seconds, the eviction thread will run, try try to evict some values from
locked nodes, and deadlock, throwing a TimeoutException
- The aggregate node-writing throughput after this happens won't be the expected
200,000 every 5 seconds
(the maximum rate we can clear the eviction queue), but will instead be about:
1 every 15 seconds
(the rate the lock times out)
There may be some subtle reasons why that doesn't quite work, but it should be close.
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 2.0.0, 1.4.0.SP2
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:30 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=comments#action_12345074 ]
Manik Surtani commented on JBCACHE-794:
---------------------------------------
Playing around with unit tests for this; I don't see any timeout exceptions or
deadlocks as you suggest but I do see the main thread (or any writer thread for that
matter) blocking until the eviction thread can start to clear stuff up.
I can see how this is a problem, particularly if the eviction thread does get a timeout
exception, as it then puts the eviction request *back* on the queue. This should not
happen since the node obviously should no longer be evicted.
Looking at the test some more, trying to recreate the deadlock scenario you speak of.
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 2.0.0, 1.4.0.SP2
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:04 p.m.
New subject: [JBoss JIRA] Updated: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=all ]
Manik Surtani updated JBCACHE-794:
----------------------------------
JBoss Forum Reference:
http://jboss.org/index.html?module=bb&op=viewtopic&p=3979697
Complexity: High
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 1.4.0.SP2, 2.0.0
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:37 a.m.
New subject: [JBoss JIRA] Commented: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=comments#action_12345907 ]
Manik Surtani commented on JBCACHE-794:
---------------------------------------
Ok, decision made to set lock acquisition timeout to 0 for the eviction thread.
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 1.4.0.SP2, 2.0.0
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:17 a.m.
New subject: [JBoss JIRA] Closed: (JBCACHE-794) Eviction Queue hard limit can cause deadlocks
[ http://jira.jboss.com/jira/browse/JBCACHE-794?page=all ]
Manik Surtani closed JBCACHE-794.
---------------------------------
Resolution: Done
Eviction Queue hard limit can cause deadlocks
---------------------------------------------
Key: JBCACHE-794
URL: http://jira.jboss.com/jira/browse/JBCACHE-794
Project: JBoss Cache
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 1.4.0.SP1
Reporter: Owen Taylor
Assigned To: Manik Surtani
Fix For: 1.4.1.GA, 2.0.0.GA
if the eviction queue fills up, you can get the situation where:
- The eviction queue processing task calls evict() which needs to get a write lock on a
node
- A different thread has a read lock on that node and is blocking to put something into
the eviction queue
Once this starts happening the evication queue thread is procesing events *very* slowly
because
it repeatedly stalls while the lock times out, so the queue will stay full.
Without digging into the code too much, possible fixes that occur to me include:
1) Don't have a hard limit on the size of the eviction queue, and just warn when it
gets too big
2) Use a timeout of 0 when trying to get the write lock to evict a node, after all, if a
node is
locked when we try to evict it, it probably shouldn't be evicted anyways.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7140
days inactive
7166
days old
8 comments
2 participants
participants (2)
-
Manik Surtani (JIRA) -
Owen Taylor (JIRA)