6:12 p.m.
Alexey Makhmutov created AS7-3343:
-------------------------------------
Summary: Concurrency issues in ReferenceCountingEntityCache
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: jaikiran pai
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance in
cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:12 p.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Alexey Makhmutov updated AS7-3343:
----------------------------------
Attachment: cacheProblemTC.zip
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: jaikiran pai
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:24 p.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Alexey Makhmutov commented on AS7-3343:
---------------------------------------
In my view, this problem looks pretty critical. It completely blocked testing of
application migration (from AS 5.1) -- every time we've put concurrent workload,
we've always ended with lot of mentioned errors.
As workaround, we've switched entity cache from 'ReferenceCountingEntityCache'
to 'TransactionLocalEntityCache', however it's not possible to perform such
change via configuration as for now, so it requrs AS code change and rebuilding of EJB3
subsystem in order to enable this workaround.
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: jaikiran pai
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:21 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Stuart Douglas commented on AS7-3343:
-------------------------------------
Can you test this with upstream? I think there have been some entity bean concurrency
fixes since CR1b.
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: jaikiran pai
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:21 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Stuart Douglas reassigned AS7-3343:
-----------------------------------
Assignee: Stuart Douglas (was: jaikiran pai)
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:16 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Alexey Makhmutov commented on AS7-3343:
---------------------------------------
The test case was acutally assembled on 7.1.Final-SNAPSHOT. I've just pulled latest
changes and rebuilt server -- same behavior is observed (i.e. errors).
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:03 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Stuart Douglas commented on AS7-3343:
-------------------------------------
Would you be able to test with
https://github.com/stuartwdouglas/jboss-as/tree/entityBeanSync and let me know if this
resolves your issues?
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:06 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Alexey Makhmutov commented on AS7-3343:
---------------------------------------
Stuart, as I understand this covers both this issue and AS7-3355? These changes look
really were promising. :) It's just what I'd like to see -- no finalizers,
configurable pooling and cache selection.
Here is the results from synthetic cases:
* First, I wasn't able to reproduce the problem with errors during concurrent access
on synthetic cases and smoe
* Then, I've used the test application from AS7-3355 to test these changes under
concurrent workload with different configurations (using the same environment as for the
baseline) -- with and without pooling, using pessimistic and optimistic locking. Here is
the summary:
|| Configuration || Throughput ||
| JBoss AS 5.1 | 133 req/s |
| JBoss AS 7.1 -- without fixes | 66 req/s |
| JBoss AS 7.1 -- with fixes, no pool, pessimistic locking | 107 req/s |
| JBoss AS 7.1 -- with fixes, pool enabled, pessimistic locking | 142.5 req/s |
| JBoss AS 7.1 -- with fixes, no pool, optimistic locking | 134.3 req/s |
| JBoss AS 7.1 -- with fixes, pool enabled, pessimistic locking | 142.5 req/s |
So, definitely this fix makes a significant performance improvement for the Entity
handling.
I'm making more real-life testing right now, so probably I'll be back shortly with
more observations.
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:06 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Alexey Makhmutov edited comment on AS7-3343 at 1/20/12 8:04 AM:
----------------------------------------------------------------
Stuart, as I understand this covers both this issue and AS7-3355? These changes look
really were promising. :) It's just what I'd like to see -- no finalizers,
configurable pooling and cache selection.
Here is the results from synthetic cases:
* First, I wasn't able to reproduce the problem with errors during concurrent access
on synthetic cases.
* Then, I've used the test application from AS7-3355 to test these changes under
concurrent workload with different configurations (using the same environment as for the
baseline) -- with and without pooling, using pessimistic and optimistic locking. Here is
the summary:
|| Configuration || Throughput ||
| JBoss AS 5.1 | 133 req/s |
| JBoss AS 7.1 -- without fixes | 66 req/s |
| JBoss AS 7.1 -- with fixes, no pool, pessimistic locking | 107 req/s |
| JBoss AS 7.1 -- with fixes, pool enabled, pessimistic locking | 142.5 req/s |
| JBoss AS 7.1 -- with fixes, no pool, optimistic locking | 134.3 req/s |
| JBoss AS 7.1 -- with fixes, pool enabled, pessimistic locking | 142.5 req/s |
So, definitely this fix makes a significant performance improvement for the Entity
handling.
I'm making more real-life testing right now, so probably I'll be back shortly with
more observations.
was (Author: Lerm):
Stuart, as I understand this covers both this issue and AS7-3355? These changes look
really were promising. :) It's just what I'd like to see -- no finalizers,
configurable pooling and cache selection.
Here is the results from synthetic cases:
* First, I wasn't able to reproduce the problem with errors during concurrent access
on synthetic cases and smoe
* Then, I've used the test application from AS7-3355 to test these changes under
concurrent workload with different configurations (using the same environment as for the
baseline) -- with and without pooling, using pessimistic and optimistic locking. Here is
the summary:
|| Configuration || Throughput ||
| JBoss AS 5.1 | 133 req/s |
| JBoss AS 7.1 -- without fixes | 66 req/s |
| JBoss AS 7.1 -- with fixes, no pool, pessimistic locking | 107 req/s |
| JBoss AS 7.1 -- with fixes, pool enabled, pessimistic locking | 142.5 req/s |
| JBoss AS 7.1 -- with fixes, no pool, optimistic locking | 134.3 req/s |
| JBoss AS 7.1 -- with fixes, pool enabled, pessimistic locking | 142.5 req/s |
So, definitely this fix makes a significant performance improvement for the Entity
handling.
I'm making more real-life testing right now, so probably I'll be back shortly with
more observations.
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:23 a.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Alexey Makhmutov commented on AS7-3343:
---------------------------------------
I've done more testing -- it seems, that locking works now as expected for my cases
(both pessimistic and optimistic modes).
So, yes -- it seems, that this fix has resolved observed problems.
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:01 p.m.
[
https://issues.jboss.org/browse/AS7-3343?page=com.atlassian.jira.plugin.s...
]
Stuart Douglas resolved AS7-3343.
---------------------------------
Fix Version/s: 7.1.0.Final
Resolution: Done
I just pushed the final patch for this that removes the gap between get() and reference(),
so this should now be resolved
Concurrency issues in ReferenceCountingEntityCache
--------------------------------------------------
Key: AS7-3343
URL: https://issues.jboss.org/browse/AS7-3343
Project: Application Server 7
Issue Type: Bug
Components: EJB
Affects Versions: 7.1.0.CR1b
Reporter: Alexey Makhmutov
Assignee: Stuart Douglas
Fix For: 7.1.0.Final
Attachments: cacheProblemTC.zip
While running multithreaded workload against AS 7.1Beta/CR1 for application with Entity
EJB 2.x we've faced a lot of following errors:
* Instance for PK [XXX] already registerd.
* Instance [YYY] not found in cache
It seems, that these errors are caused by synchronization issues in
ReferenceCountingEntityCache:
* If two threads are trying to access the cache and there is no ready instance for
particular PK in the cache, then both threads are trying to get some instance from the
pool and put it into the cache -- as result, the second thread can get 'already
registered' exception. We were able to compose a minimal test case application which
reproduces this kind of problem (see below).
* There is a gap in time between the call to Associate interceptor (which puts instance
in cache - via 'get' method) and Synchronization interceptor (which calls
'reference' method). During this time instance is not referenced, so it seems it
can be removed by some other thread which finish its invocation at the same time. Probably
this is the root cause of 'not found in cache' error, however it's hard to
create a syntetic test case for it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4640
days inactive
4645
days old
10 comments
2 participants
participants (2)
-
Alexey Makhmutov (JIRA) -
Stuart Douglas (JIRA)