We started seeing test failure in PassivationTestCase.testPassivationMaxSize() which has passivation max-size=1 and repeated called to two separate beans...
4:18 a.m.
We started to see what looks like a JPA extended persistence context
related error. [1] is the server.log that shows the exception (see the
last one near the bottom) that shouldn't be happening on WildFly master.
Also, there are some marshalling errors that I didn't see on brontes
(I'm wondering if there is a concurrency error between the bean
invocation and passivation/activation when Hibernate throws the
"java.lang.IllegalStateException: Cannot serialize a session while
connected" error during marshalling as if bean is active).
I am able to recreate the failure locally with a modification to the
PassivationTestCase.testPassivationMaxSize() [2] to repeatedly
alternative between calls to remote1 + remote2 beans.
I don't have this nailed down to the actual cause but it seems like a
race condition between passivation/activation and bean invocation (imo).
Scott
[1] https://www.dropbox.com/s/277pwvxv53dp8vk/server.zip contains the
results from more than one test run. If you look at the server.log, you
probably should go to the end and see the last "javax.ejb.EJBException:
WFLYJPA0030: Found extended persistence context in SFSB invocation call
stack but that cannot be used" error
[2] unit test change to loop repeatedly until failure occurs
https://github.com/scottmarlow/wildfly/tree/passivationxpcissue
11:52 a.m.
For time beeing I have muted and assigned this test failure on brontes to
you Scott.
This way people wont get bogus PR test failures while you are working on a
fix.
--
tomaz
On Thu, Jul 31, 2014 at 4:18 AM, Scott Marlow <smarlow(a)redhat.com> wrote:
We started to see what looks like a JPA extended persistence context
related error. [1] is the server.log that shows the exception (see the
last one near the bottom) that shouldn't be happening on WildFly master.
Also, there are some marshalling errors that I didn't see on brontes
(I'm wondering if there is a concurrency error between the bean
invocation and passivation/activation when Hibernate throws the
"java.lang.IllegalStateException: Cannot serialize a session while
connected" error during marshalling as if bean is active).
I am able to recreate the failure locally with a modification to the
PassivationTestCase.testPassivationMaxSize() [2] to repeatedly
alternative between calls to remote1 + remote2 beans.
I don't have this nailed down to the actual cause but it seems like a
race condition between passivation/activation and bean invocation (imo).
Scott
[1] https://www.dropbox.com/s/277pwvxv53dp8vk/server.zip contains the
results from more than one test run. If you look at the server.log, you
probably should go to the end and see the last "javax.ejb.EJBException:
WFLYJPA0030: Found extended persistence context in SFSB invocation call
stack but that cannot be used" error
[2] unit test change to loop repeatedly until failure occurs
https://github.com/scottmarlow/wildfly/tree/passivationxpcissue
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
5:22 p.m.
Just to keep everyone in the loop, it seems like configuring the
passivation max-size to be less than "one + number of nested beans",
could be the cause. We are currently configuring the passivation cache
max-size to be one, which may lead to passivation of the nested bean.
In the PassivationTestCase test, each top level session bean has a
nested bean. The top level bean shares the extended persistence context
with the nested bean.
If users hit this same failure, they should ensure that max-size is at
least equal to "1 + number of nested beans referenced from top level bean".
We will continue to discuss this case on IRC or email and keep you all
posted.
Scott
On 07/31/2014 05:52 AM, Tomaž Cerar wrote:
For time beeing I have muted and assigned this test failure on
brontes
to you Scott.
This way people wont get bogus PR test failures while you are working on
a fix.
--
tomaz
On Thu, Jul 31, 2014 at 4:18 AM, Scott Marlow <smarlow(a)redhat.com
<mailto:smarlow@redhat.com>> wrote:
We started to see what looks like a JPA extended persistence context
related error. [1] is the server.log that shows the exception (see the
last one near the bottom) that shouldn't be happening on WildFly master.
Also, there are some marshalling errors that I didn't see on brontes
(I'm wondering if there is a concurrency error between the bean
invocation and passivation/activation when Hibernate throws the
"java.lang.IllegalStateException: Cannot serialize a session while
connected" error during marshalling as if bean is active).
I am able to recreate the failure locally with a modification to the
PassivationTestCase.testPassivationMaxSize() [2] to repeatedly
alternative between calls to remote1 + remote2 beans.
I don't have this nailed down to the actual cause but it seems like a
race condition between passivation/activation and bean invocation (imo).
Scott
[1] https://www.dropbox.com/s/277pwvxv53dp8vk/server.zip contains the
results from more than one test run. If you look at the server.log, you
probably should go to the end and see the last "javax.ejb.EJBException:
WFLYJPA0030: Found extended persistence context in SFSB invocation call
stack but that cannot be used" error
[2] unit test change to loop repeatedly until failure occurs
https://github.com/scottmarlow/wildfly/tree/passivationxpcissue
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
10:06 p.m.
From talking with Paul about this situation, it sounds like we
currently do allow direct passivation of the nested stateful beans,
without passivating the top level bean that might reference it.
Up to now, I thought that we always passivated the top level stateful
bean and all referenced beans, as a group. We actually do passivate the
entire group (top level + nested beans), when the top level bean is
passivated but we also allow the nested beans to passivated in an
isolated fashion.
My question at this point is, could we switch to only passivating top
level beans? I'm not sure if we know the difference between top-level
and nested beans, which might make this difficult or impossible.
On 07/31/2014 11:22 AM, Scott Marlow wrote:
Just to keep everyone in the loop, it seems like configuring the
passivation max-size to be less than "one + number of nested beans",
could be the cause. We are currently configuring the passivation cache
max-size to be one, which may lead to passivation of the nested bean.
In the PassivationTestCase test, each top level session bean has a
nested bean. The top level bean shares the extended persistence context
with the nested bean.
If users hit this same failure, they should ensure that max-size is at
least equal to "1 + number of nested beans referenced from top level bean".
We will continue to discuss this case on IRC or email and keep you all
posted.
Scott
On 07/31/2014 05:52 AM, Tomaž Cerar wrote:
> For time beeing I have muted and assigned this test failure on brontes
> to you Scott.
>
> This way people wont get bogus PR test failures while you are working on
> a fix.
>
> --
> tomaz
>
>
> On Thu, Jul 31, 2014 at 4:18 AM, Scott Marlow <smarlow(a)redhat.com
> <mailto:smarlow@redhat.com>> wrote:
>
> We started to see what looks like a JPA extended persistence context
> related error. [1] is the server.log that shows the exception (see the
> last one near the bottom) that shouldn't be happening on WildFly master.
> Also, there are some marshalling errors that I didn't see on brontes
> (I'm wondering if there is a concurrency error between the bean
> invocation and passivation/activation when Hibernate throws the
> "java.lang.IllegalStateException: Cannot serialize a session while
> connected" error during marshalling as if bean is active).
>
> I am able to recreate the failure locally with a modification to the
> PassivationTestCase.testPassivationMaxSize() [2] to repeatedly
> alternative between calls to remote1 + remote2 beans.
>
> I don't have this nailed down to the actual cause but it seems like a
> race condition between passivation/activation and bean invocation (imo).
>
> Scott
>
> [1] https://www.dropbox.com/s/277pwvxv53dp8vk/server.zip contains the
> results from more than one test run. If you look at the server.log, you
> probably should go to the end and see the last "javax.ejb.EJBException:
> WFLYJPA0030: Found extended persistence context in SFSB invocation call
> stack but that cannot be used" error
>
> [2] unit test change to loop repeatedly until failure occurs
> https://github.com/scottmarlow/wildfly/tree/passivationxpcissue
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
11:02 p.m.
----- Original Message -----
From: "Scott Marlow" <smarlow(a)redhat.com>
To: wildfly-dev(a)lists.jboss.org
Sent: Thursday, July 31, 2014 4:06:24 PM
Subject: Re: [wildfly-dev] We started seeing test failure in
PassivationTestCase.testPassivationMaxSize() which has
passivation max-size=1 and repeated called to two separate beans...
From talking with Paul about this situation, it sounds like we
currently do allow direct passivation of the nested stateful beans,
without passivating the top level bean that might reference it.
Sorry if I wasn't clear, but that's not the case.
All nested beans are always passivated together - they have to be to preserve references
to the same objects (e.g. the XPC).
Up to now, I thought that we always passivated the top level
stateful
bean and all referenced beans, as a group. We actually do passivate the
entire group (top level + nested beans), when the top level bean is
passivated but we also allow the nested beans to passivated in an
isolated fashion.
That's not quite true either. A bean forever belongs to the group into which it was
1st created.
So, even if a nested bean is invoked directly (rather than via the parent bean) - it would
still passivate as a group along with its other associated beans.
My question at this point is, could we switch to only passivating
top
level beans? I'm not sure if we know the difference between top-level
and nested beans, which might make this difficult or impossible.
The issue, I think, is when passivation of the group is triggered within the context of
the invocation that uses the bean.
The reason this is happening in this test is because the passivation threshold is lower
than the depth of your nested beans.
When did this test start failing? I suspect it might be related to this commit:
https://github.com/wildfly/wildfly/commit/555b8a42249527674448b880b95b334...
If so, I should have a fix for this shortly.
Let's also keep in mind that this isn't really a real-world scenario, as nobody in
their right mind would ever set their max-size of the SFSB cache to be so small.
So, this is more about getting the test to pass than to fix a functional issue.
On 07/31/2014 11:22 AM, Scott Marlow wrote:
> Just to keep everyone in the loop, it seems like configuring the
> passivation max-size to be less than "one + number of nested beans",
> could be the cause. We are currently configuring the passivation cache
> max-size to be one, which may lead to passivation of the nested bean.
>
> In the PassivationTestCase test, each top level session bean has a
> nested bean. The top level bean shares the extended persistence context
> with the nested bean.
>
> If users hit this same failure, they should ensure that max-size is at
> least equal to "1 + number of nested beans referenced from top level
bean".
>
> We will continue to discuss this case on IRC or email and keep you all
> posted.
>
> Scott
>
> On 07/31/2014 05:52 AM, Tomaž Cerar wrote:
>> For time beeing I have muted and assigned this test failure on brontes
>> to you Scott.
>>
>> This way people wont get bogus PR test failures while you are working on
>> a fix.
>>
>> --
>> tomaz
>>
>>
>> On Thu, Jul 31, 2014 at 4:18 AM, Scott Marlow <smarlow(a)redhat.com
>> <mailto:smarlow@redhat.com>> wrote:
>>
>> We started to see what looks like a JPA extended persistence context
>> related error. [1] is the server.log that shows the exception (see
>> the
>> last one near the bottom) that shouldn't be happening on WildFly
>> master.
>> Also, there are some marshalling errors that I didn't see on
>> brontes
>> (I'm wondering if there is a concurrency error between the bean
>> invocation and passivation/activation when Hibernate throws the
>> "java.lang.IllegalStateException: Cannot serialize a session while
>> connected" error during marshalling as if bean is active).
>>
>> I am able to recreate the failure locally with a modification to the
>> PassivationTestCase.testPassivationMaxSize() [2] to repeatedly
>> alternative between calls to remote1 + remote2 beans.
>>
>> I don't have this nailed down to the actual cause but it seems like a
>> race condition between passivation/activation and bean invocation
>> (imo).
>>
>> Scott
>>
>> [1] https://www.dropbox.com/s/277pwvxv53dp8vk/server.zip contains the
>> results from more than one test run. If you look at the server.log,
>> you
>> probably should go to the end and see the last
>> "javax.ejb.EJBException:
>> WFLYJPA0030: Found extended persistence context in SFSB invocation
>> call
>> stack but that cannot be used" error
>>
>> [2] unit test change to loop repeatedly until failure occurs
>> https://github.com/scottmarlow/wildfly/tree/passivationxpcissue
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>>
>>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
1:13 a.m.
On 07/31/2014 05:02 PM, Paul Ferraro wrote:
When did this test start failing? I suspect it might be related to
this commit:
https://github.com/wildfly/wildfly/commit/555b8a42249527674448b880b95b334...
If so, I should have a fix for this shortly.
The first failure on brontes was at 07/30/2014 02:03 AM (Brno time?).
It looks like the above commit was merged around 07/29/2014 11:18 PM
(EST timezone).
It could be the above commit, I'm not sure.
I tried running the same test against
4a765b5d7dee4d28950805c113b0e623fdb017b2 ( WFLY-3486 Make default
session timeout configurable in the Undertow subsystem) but got a
"Unable to acquire lock after [15 seconds] on key
[5f1c2614-2ed8-4b70-951e-f30c9699c8d8] for requestor
[GlobalTransaction:<null>:105:local]! Lock held by
[GlobalTransaction:<null>:106:local]
" error and other exceptions as well. Not sure why.
Let's also keep in mind that this isn't really a real-world scenario, as nobody
in their right mind would ever set their max-size of the SFSB cache to be so small.
So, this is more about getting the test to pass than to fix a functional issue.
1:40 a.m.
On 7/31/14, 6:13 PM, Scott Marlow wrote:
On 07/31/2014 05:02 PM, Paul Ferraro wrote:
> When did this test start failing? I suspect it might be related to this commit:
> https://github.com/wildfly/wildfly/commit/555b8a42249527674448b880b95b334...
> If so, I should have a fix for this shortly.
The first failure on brontes was at 07/30/2014 02:03 AM (Brno time?).
It looks like the above commit was merged around 07/29/2014 11:18 PM
(EST timezone).
It could be the above commit, I'm not sure.
For any test run, click on the "Changes" link and you can often see a
lot of the recent history of the branch.
In this 2:03 AM run, the commit in question is at the top:
http://brontes.lab.eng.brq.redhat.com/viewLog.html?buildId=18974&buil...
Seems I shouldn't have merged it.
Some story about a boy shepherd and a wolf comes to mind...
I tried running the same test against
4a765b5d7dee4d28950805c113b0e623fdb017b2 ( WFLY-3486 Make default
session timeout configurable in the Undertow subsystem) but got a
"Unable to acquire lock after [15 seconds] on key
[5f1c2614-2ed8-4b70-951e-f30c9699c8d8] for requestor
[GlobalTransaction:<null>:105:local]! Lock held by
[GlobalTransaction:<null>:106:local]
" error and other exceptions as well. Not sure why.
>
> Let's also keep in mind that this isn't really a real-world scenario, as
nobody in their right mind would ever set their max-size of the SFSB cache to be so
small.
>
> So, this is more about getting the test to pass than to fix a functional issue.
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Brian Stansberry
Senior Principal Software Engineer
JBoss by Red Hat
3550
days inactive
3550
days old
6 comments
4 participants
participants (4)
-
Brian Stansberry -
Paul Ferraro -
Scott Marlow -
Tomaž Cerar