8:53 a.m.
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/jira/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Andrew Dinn
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:55 a.m.
New subject: [JBoss JIRA] Updated: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/jira/browse/JBTM-599?page=com.atlassian.jira.plugi...
]
Andrew Dinn updated JBTM-599:
-----------------------------
Component/s: Transaction Core
Affects Version/s: 4.7.0
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/jira/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:05 a.m.
New subject: [JBoss JIRA] Commented: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/jira/browse/JBTM-599?page=com.atlassian.jira.plugi...
]
Andrew Dinn commented on JBTM-599:
----------------------------------
I looked further into this and managed to isolate the problem. Each of the test threads
does an add then a delete of an object store record. I used a byteman script to allow 50
threads to proceed and then stall the next 50 threads for a few seconds at the call to
add.Work This caused the stall to happen every time I ran the test.
The remove operations alter the wait condition state because they increase the number of
pending remove operations. This can cause the condition test made in addWork to see the
cache as full. However, the removing threads do not test the condition and notify the
AsyncCache thread when the state changes to full.
So, the AsyncStore thread can deplete the queue of all pendiing adds and removes and then
go into a wait. If this is followed by enough removes with no adds then the cache fills
and further adds are stalled. This leaves them stalled until the AsyncStore thread wakes
up.
The fix is to test the wait condition during remove and, if the cache is full notify the
AsyncStore thread.
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/jira/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:46 p.m.
New subject: [JBoss JIRA] Work started: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/browse/JBTM-599?page=com.atlassian.jira.plugin.sys...
]
Work on JBTM-599 started by Mark Little.
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
Assignee: Mark Little
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:46 p.m.
New subject: [JBoss JIRA] Assigned: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/browse/JBTM-599?page=com.atlassian.jira.plugin.sys...
]
Mark Little reassigned JBTM-599:
--------------------------------
Assignee: Mark Little
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
Assignee: Mark Little
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:27 a.m.
New subject: [JBoss JIRA] Updated: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/browse/JBTM-599?page=com.atlassian.jira.plugin.sys...
]
Mark Little updated JBTM-599:
-----------------------------
Fix Version/s: 4.14.0
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
Assignee: Mark Little
Fix For: 4.14.0
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:43 p.m.
New subject: [JBoss JIRA] Closed: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/browse/JBTM-599?page=com.atlassian.jira.plugin.sys...
]
Mark Little closed JBTM-599.
----------------------------
Resolution: Done
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
Assignee: Mark Little
Fix For: 4.14.0
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:37 p.m.
New subject: [JBoss JIRA] Commented: (JBTM-599) Synchronization problem in CacheStore
[
https://jira.jboss.org/browse/JBTM-599?page=com.atlassian.jira.plugin.sys...
]
Misty Stanley-Jones commented on JBTM-599:
------------------------------------------
Release Note:
Cause: <methodname>remove</methodname> operations increase the number of
pending <methodname>remove</methodname> operations. They do not notify the
AsyncCache thread when the state of the cache changes to full.
Consequence: The AsyncStore thread can deplete the qeuue of all pending adds and removes,
and then go into a wait. If this is followed by enough removes with no adds, the cache
fills and further adds are stalled until the AsyncStore thread wakes up.
Fix: Test the wait condition during remove, and if the cache is full, notify the
AsyncStore thread.
Result: The AsyncCache does not stall, and clears the cache properly.
Synchronization problem in CacheStore
-------------------------------------
Key: JBTM-599
URL: https://jira.jboss.org/browse/JBTM-599
Project: JBoss Transaction Manager
Issue Type: Bug
Security Level: Public(Everyone can see)
Components: Transaction Core
Affects Versions: 4.7.0
Reporter: Andrew Dinn
Assignee: Mark Little
Fix For: 4.14.0
This bug manifests occasionally when running the CachedTest. The AsyncStore thread
suspends inside its run method on a (120 sec) timed wait because the cache is not full.
When I managed to catch this case in the debugger I found that there were many writer
threads (~30) suspended inside addWork in an untimed wait on the overflow lock. So, these
writing threads make no progress for 120 seconds. The AsyncStore thread does notify the
object on which the writers are waiting but there is clearly a window where the writers
can go to sleep while the cache is full and not get notified when the AsyncStore thread
has emptied it.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
4885
days inactive
5377
days old
jbossts-issues@lists.jboss.org
7 comments
3 participants
participants (3)
-
Andrew Dinn (JIRA) -
Mark Little (JIRA)
-
Misty Stanley-Jones (JIRA)