[jboss-dev] Possible blocker with jboss-cl 2.2.0.Alpha1 [WAS]: Hudson Jobs Hanging

[Brian Stansberry]

Thanks, Jaikiran. I'm forwarding this to the jboss-dev list for greater 
visibility.

The EJB3 team is seeing consistent failures where a server thread is 
unable to load classes from rt.jar. These occur in a test where a remote 
client makes requests against a service that is concurrently being 
undeployed. Failure is quite similar to what is seen at 
https://jira.jboss.org/jira/browse/JBAS-7688 and previously mentioned on 
this list.[1]

I've elevated JBAS-7688 to Blocker; we'll have to get comfortable early 
next week that this isn't due to the jboss-cl 2.2.0.Alpha1 upgrade, fix 
jboss-cl 2.2.0, or roll back AS trunk to 2.0.8.GA.

[1] 
http://lists.jboss.org/pipermail/jboss-development/2010-February/015693.html

On 02/06/2010 09:24 AM, Jaikiran Pai wrote:
> I switched one of the jobs (which i have been using for testing) on Mike
>   to 2.0.8.GA of jbcl and haven't seen this hang in those. However, the
> other job which is using 2.2.0.Alpha1 is consistently hanging with the
> same exception being thrown. I have tried to reproduce this locally, but
> haven't been able to do so (i.e. the test runs fine with 2.2.0.Alpha1 on
> my local setup).
>
> The exception stacktrace (as Carlo posted earlier) seems to be
> originating from a remoting thread which is doing some marshalling and
> running into CNFE (all these CNFE classes so far have been the ones
> residing in rt.jar). I took a thread dump of the server, at the time
> when Mike was hung during this testcase - couldn't find anything useful
> in there. Perhaps, i should be taking a thread dump of the JVM on the
> client (the JUnit testcase), to see what it is waiting on. At the same
> time, David mentions[1] that:
>
> "I hypothesize that the classloader is (sometimes) being destroyed
> before the thread gets going, causing it to be unable to load anything
> at all."
>
> I'll see if i can figure out:
>
> 1) Whether that statement applies to this testcase too
> 2) Why this happens specifically in this testcase
>
>
>
> [1]
> http://lists.jboss.org/pipermail/jboss-development/2010-February/015694.html
>
>
> regards,
> -Jaikiran
> Brian Stansberry wrote:
>> On 02/04/2010 12:18 PM, Jaikiran Pai wrote:
>>> Mike is running into this again
>>> http://mike.lab.bos.redhat.com:8380/hudson/job/EJB3_1-1.0.3-JBoss-AS-6.x-tests-ejb3/3/console
>>>
>>>
>>>
>>> The https://jira.jboss.org/jira/browse/EJBTHREE-1982 has linked
>>> JBKERNEL-86 as one of the problems. JBKERNEL-86 is marked as fixed in
>>> 2.2.0.alpha-4 of MC kernel. The current AS trunk has 2.2.0-alpha-5. But
>>> we are still seeing the same hang. So i guess there's more to it.
>>> Looking at the logs (attached) i don't see an explicit OOM, but i do see
>>> signs of it (like CNFE on a com.sun.* class after numerous deploys).
>>
>> On the face of it that sounds like a different problem. Occam's razor
>> say's it's not, but a CNFE seems like a different beast than a memory
>> leak.
>>
>> Possibly the CNFE is due to the jboss-cl upgrade to 2.2.0.Alpha1? See
>> https://jira.jboss.org/jira/browse/JBAS-7688 for another issue where
>> after that change classes in rt.jar weren't available.
>>
>> The AS should run fine w/ it's jboss-cl moved back to 2.0.8.GA, so
>> this theory is testable.
>


-- 
Brian Stansberry
Lead, AS Clustering
JBoss by Red Hat