Now that I know what to look for I can reproduce it using jboss-cl 2.0
(rev 100736). I'm almost certain 2.0.8 exhibits the same problem.
The BaseClassLoader never has a connection to rt.jar. So in essence it
can't load anything from there. During normal operations the whole class
loader and delegates will load up a class properly, but once it goes
into shutdown mode it can only load 'locally' defined classes.
It then turns into a race whether the class loader gets shutdown first
or the invoker manages to write out the response. With jboss-cl 2.2.0
undeploy has become too fast. ;-)
Carlo
On 02/09/2010 11:17 AM, Jaikiran Pai wrote:
Hmm, poof part in the logs is misleading then :) I mean it should not
have been a NCDFE on classes from rt.jar. And then again, the
difference in behavior between the jbcl versions.
-Jaikiran
Carlo de Wolf wrote:
> This should have nothing to do at all with jboss-cl. In essence it's
> the following scenario that fails:
>
> - request comes in
> - undeploy request comes in
> - container blocks undeploy
> - response goes out the container
> - container unblocks and undeploys
> - response tries to get out of the invoker => poof
>
> Carlo
>
> On 02/06/2010 06:40 PM, Brian Stansberry wrote:
>> Thanks, Jaikiran. I'm forwarding this to the jboss-dev list for
>> greater visibility.
>>
>> The EJB3 team is seeing consistent failures where a server thread is
>> unable to load classes from rt.jar. These occur in a test where a
>> remote client makes requests against a service that is concurrently
>> being undeployed. Failure is quite similar to what is seen at
>>
https://jira.jboss.org/jira/browse/JBAS-7688 and previously
>> mentioned on this list.[1]
>>
>> I've elevated JBAS-7688 to Blocker; we'll have to get comfortable
>> early next week that this isn't due to the jboss-cl 2.2.0.Alpha1
>> upgrade, fix jboss-cl 2.2.0, or roll back AS trunk to 2.0.8.GA.
>>
>> [1]
>>
http://lists.jboss.org/pipermail/jboss-development/2010-February/015693.html
>>
>>
>> On 02/06/2010 09:24 AM, Jaikiran Pai wrote:
>>> I switched one of the jobs (which i have been using for testing) on
>>> Mike
>>> to 2.0.8.GA of jbcl and haven't seen this hang in those. However,
>>> the
>>> other job which is using 2.2.0.Alpha1 is consistently hanging with the
>>> same exception being thrown. I have tried to reproduce this
>>> locally, but
>>> haven't been able to do so (i.e. the test runs fine with
>>> 2.2.0.Alpha1 on
>>> my local setup).
>>>
>>> The exception stacktrace (as Carlo posted earlier) seems to be
>>> originating from a remoting thread which is doing some marshalling and
>>> running into CNFE (all these CNFE classes so far have been the ones
>>> residing in rt.jar). I took a thread dump of the server, at the time
>>> when Mike was hung during this testcase - couldn't find anything
>>> useful
>>> in there. Perhaps, i should be taking a thread dump of the JVM on the
>>> client (the JUnit testcase), to see what it is waiting on. At the same
>>> time, David mentions[1] that:
>>>
>>> "I hypothesize that the classloader is (sometimes) being destroyed
>>> before the thread gets going, causing it to be unable to load anything
>>> at all."
>>>
>>> I'll see if i can figure out:
>>>
>>> 1) Whether that statement applies to this testcase too
>>> 2) Why this happens specifically in this testcase
>>>
>>>
>>>
>>> [1]
>>>
http://lists.jboss.org/pipermail/jboss-development/2010-February/015694.html
>>>
>>>
>>>
>>> regards,
>>> -Jaikiran
>>> Brian Stansberry wrote:
>>>> On 02/04/2010 12:18 PM, Jaikiran Pai wrote:
>>>>> Mike is running into this again
>>>>>
http://mike.lab.bos.redhat.com:8380/hudson/job/EJB3_1-1.0.3-JBoss-AS-6.x-...
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The
https://jira.jboss.org/jira/browse/EJBTHREE-1982 has linked
>>>>> JBKERNEL-86 as one of the problems. JBKERNEL-86 is marked as
>>>>> fixed in
>>>>> 2.2.0.alpha-4 of MC kernel. The current AS trunk has
>>>>> 2.2.0-alpha-5. But
>>>>> we are still seeing the same hang. So i guess there's more to
it.
>>>>> Looking at the logs (attached) i don't see an explicit OOM, but i
>>>>> do see
>>>>> signs of it (like CNFE on a com.sun.* class after numerous deploys).
>>>> On the face of it that sounds like a different problem. Occam's
razor
>>>> say's it's not, but a CNFE seems like a different beast than a
memory
>>>> leak.
>>>>
>>>> Possibly the CNFE is due to the jboss-cl upgrade to 2.2.0.Alpha1? See
>>>>
https://jira.jboss.org/jira/browse/JBAS-7688 for another issue where
>>>> after that change classes in rt.jar weren't available.
>>>>
>>>> The AS should run fine w/ it's jboss-cl moved back to 2.0.8.GA, so
>>>> this theory is testable.
>
> _______________________________________________
> jboss-development mailing list
> jboss-development(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/jboss-development