[JBoss-dev] RE: Fixing the testsuite for JRockit

[Dimitris Andreadis]

I started looking into some Jrockit failures and I realized that the
execution order of Junit tests inside a class *MAY BE THE REVERSE* when
running with Jrockit, compared to using sun's JVM.

This boils down to Class.getMethods()/getDeclaredMethods() returnings
methods in an undefined order:
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Class.html#getMethods(
)

Other people have stumbled on this before and the verdict seems to be
that tests should be re-written to work in any order:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=142528
http://mail-archive.objectweb.org/howl/2005-11/msg00058.html

I'm certain there are places in our testsuite that really depend on the
execution order, in fact this could explain the majority of jrockit
failures, so unless we isolate and conditionally execute those tests, we
need to fix them!


> -----Original Message-----
> From: Dimitris Andreadis 
> Sent: Thursday, September 07, 2006 5:05 PM
> To: JBoss.org development list; QA
> Subject: Fixing the testsuite for JRockit
> 
> An important task for the jbas codebase is to fix the 
> testsuite when run under the jrockit JVM. This is initially 
> for the 4.x branch but the fixes should apply verbatim to jboss head.
> 
> The starting point is here (see linked tasks & subtasks)
> http://jira.jboss.com/jira/browse/JBAS-3598
> http://cruisecontrol.jboss.com/cc/buildresults/jboss-4.0-testsuite-1.4
> 
> At any time there are anywhere between 54-58 failures, so if 
> something is in your area please have a go. We'll be 
> assigning tasks to people, soon.
> 
> Same thing for the testsuite when run under sun-jdk5 (around 
> 10 failures)
> http://cruisecontrol.jboss.com/cc/buildresults/jboss-4.0-tests
> uite-sun-1.5
> http://jira.jboss.com/jira/browse/JBAS-3611
> 
> There are also a lot of transient failures in parts of the 
> testsuite that are more sensitive to timing/cpu issues (e.g. 
> JMS), so subsequent runs often produce different results. 
> This may also have to do with testsuite runs that interfere 
> with each other.
> 
> If we can (a) improve those parts of the testsuite and (b) 
> make sure testsuite runs are more isolated from each other, 
> that would be an improvement.
> 
> (I see already a related task from Adrian: 
> http://jira.jboss.com/jira/browse/JBAS-3221)
> 
> Thanks
> /Dimitris
>