[jboss-dev] Possible blocker with jboss-cl 2.2.0.Alpha1 [WAS]: Hudson Jobs Hanging

[Carlo de Wolf]

Now that I know what to look for I can reproduce it using jboss-cl 2.0 
(rev 100736). I'm almost certain 2.0.8 exhibits the same problem.

The BaseClassLoader never has a connection to rt.jar. So in essence it 
can't load anything from there. During normal operations the whole class 
loader and delegates will load up a class properly, but once it goes 
into shutdown mode it can only load 'locally' defined classes.

It then turns into a race whether the class loader gets shutdown first 
or the invoker manages to write out the response. With jboss-cl 2.2.0 
undeploy has become too fast. ;-)

Carlo

On 02/09/2010 11:17 AM, Jaikiran Pai wrote:
> Hmm, poof part in the logs is misleading then :) I mean it should not 
> have been a NCDFE on classes from rt.jar. And then again, the 
> difference in behavior between the jbcl versions.
>
> -Jaikiran
> Carlo de Wolf wrote:
>> This should have nothing to do at all with jboss-cl. In essence it's 
>> the following scenario that fails:
>>
>> - request comes in
>> - undeploy request comes in
>> - container blocks undeploy
>> - response goes out the container
>> - container unblocks and undeploys
>> - response tries to get out of the invoker => poof
>>
>> Carlo
>>
>> On 02/06/2010 06:40 PM, Brian Stansberry wrote:
>>> Thanks, Jaikiran. I'm forwarding this to the jboss-dev list for 
>>> greater visibility.
>>>
>>> The EJB3 team is seeing consistent failures where a server thread is 
>>> unable to load classes from rt.jar. These occur in a test where a 
>>> remote client makes requests against a service that is concurrently 
>>> being undeployed. Failure is quite similar to what is seen at 
>>> https://jira.jboss.org/jira/browse/JBAS-7688 and previously 
>>> mentioned on this list.[1]
>>>
>>> I've elevated JBAS-7688 to Blocker; we'll have to get comfortable 
>>> early next week that this isn't due to the jboss-cl 2.2.0.Alpha1 
>>> upgrade, fix jboss-cl 2.2.0, or roll back AS trunk to 2.0.8.GA.
>>>
>>> [1] 
>>> http://lists.jboss.org/pipermail/jboss-development/2010-February/015693.html 
>>>
>>>
>>> On 02/06/2010 09:24 AM, Jaikiran Pai wrote:
>>>> I switched one of the jobs (which i have been using for testing) on 
>>>> Mike
>>>>   to 2.0.8.GA of jbcl and haven't seen this hang in those. However, 
>>>> the
>>>> other job which is using 2.2.0.Alpha1 is consistently hanging with the
>>>> same exception being thrown. I have tried to reproduce this 
>>>> locally, but
>>>> haven't been able to do so (i.e. the test runs fine with 
>>>> 2.2.0.Alpha1 on
>>>> my local setup).
>>>>
>>>> The exception stacktrace (as Carlo posted earlier) seems to be
>>>> originating from a remoting thread which is doing some marshalling and
>>>> running into CNFE (all these CNFE classes so far have been the ones
>>>> residing in rt.jar). I took a thread dump of the server, at the time
>>>> when Mike was hung during this testcase - couldn't find anything 
>>>> useful
>>>> in there. Perhaps, i should be taking a thread dump of the JVM on the
>>>> client (the JUnit testcase), to see what it is waiting on. At the same
>>>> time, David mentions[1] that:
>>>>
>>>> "I hypothesize that the classloader is (sometimes) being destroyed
>>>> before the thread gets going, causing it to be unable to load anything
>>>> at all."
>>>>
>>>> I'll see if i can figure out:
>>>>
>>>> 1) Whether that statement applies to this testcase too
>>>> 2) Why this happens specifically in this testcase
>>>>
>>>>
>>>>
>>>> [1]
>>>> http://lists.jboss.org/pipermail/jboss-development/2010-February/015694.html 
>>>>
>>>>
>>>>
>>>> regards,
>>>> -Jaikiran
>>>> Brian Stansberry wrote:
>>>>> On 02/04/2010 12:18 PM, Jaikiran Pai wrote:
>>>>>> Mike is running into this again
>>>>>> http://mike.lab.bos.redhat.com:8380/hudson/job/EJB3_1-1.0.3-JBoss-AS-6.x-tests-ejb3/3/console 
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The https://jira.jboss.org/jira/browse/EJBTHREE-1982 has linked
>>>>>> JBKERNEL-86 as one of the problems. JBKERNEL-86 is marked as 
>>>>>> fixed in
>>>>>> 2.2.0.alpha-4 of MC kernel. The current AS trunk has 
>>>>>> 2.2.0-alpha-5. But
>>>>>> we are still seeing the same hang. So i guess there's more to it.
>>>>>> Looking at the logs (attached) i don't see an explicit OOM, but i 
>>>>>> do see
>>>>>> signs of it (like CNFE on a com.sun.* class after numerous deploys).
>>>>> On the face of it that sounds like a different problem. Occam's razor
>>>>> say's it's not, but a CNFE seems like a different beast than a memory
>>>>> leak.
>>>>>
>>>>> Possibly the CNFE is due to the jboss-cl upgrade to 2.2.0.Alpha1? See
>>>>> https://jira.jboss.org/jira/browse/JBAS-7688 for another issue where
>>>>> after that change classes in rt.jar weren't available.
>>>>>
>>>>> The AS should run fine w/ it's jboss-cl moved back to 2.0.8.GA, so
>>>>> this theory is testable.
>>
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
>