[jboss-dev] AS testsuites

[Bill Burke]

The productivity of teams like EJB 3.0 has been severely compromised for 
over a year because projects.  60% of the reason I didn't want to work 
on EJB 3 anymore was because I would not commit anything for a week or 
two and come back to find that half of my unit tests were broken.  This 
has gotten worse and worse as projects have split off from jboss-head.

There is ZERO peer pressure for breaking the build.  Nobody cares. 
Since the testsuites *ALWAYS* fail, nobody is paying attention to 
regressions except for the individual projects where it fails.  You even 
have cases where people comment out failing tests!  We just *cannot* do 
the refactorings that the majority of teams want to do without a stable 
testsuite.

This has to be fixed immediately.  Since there is ZERO peer pressure, 
there needs to be consequences for breaking the build or regressing.  I 
propose the following:

1. Calculate a baseline of passed vs. failed tests
2. Tag HEAD
3. If there is any testsuite regression or build breakage, freeze SVN 
until the build or regressions are fixed.
4. If build or regressions are fixed within 24 hours.  Rollback to 
previous tag
5. If no regressions or breakage, recalculate baseline and tag head
6. GOTO 3

Bill

Dimitris Andreadis wrote:
> Given the multitude of component updates and our carelessness, it's a 
> rare thing to see jboss AS testsuites run at 100%, A lot of stuff is 
> just checked in without testing it locally, and worse, the CC runs that 
> show failures after the check-ins are ignored.
> 
> The good new is we finally got there for 4.2.x. (minus 
> occasional/transient timing failures), so let's keep it that way for 
> that branch, and help fix the other ones, especially HEAD.
> 
> On a broken testsuite it's very "convenient" to just ignore failures, 
> since "somebody else must have done it".
> 
> Cheers
> /Dimitris
> 
> ===========================
> 
> View results here ->
> http://cruisecontrol.jboss.com/cc/buildresults/jboss-4.2-testsuite-sun-1.5?log=log20070620223106Lbuild.48 
> 
> 
> BUILD COMPLETE -  build.48
> Date of build: 06/20/2007 22:31:06
> Time to build: 165 minutes 8 seconds
> 
>  Unit Tests: (4087)  Total Errors and Failures: (0)
> All Tests Passed
> _______________________________________________
> jboss-development mailing list
> jboss-development at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-development

-- 
Bill Burke
JBoss, a division of Red Hat Inc.