[jboss-dev] AS testsuites

[Andrig T Miller]

To strengthen your argument, I had 385 build messages in my e-mail and
here is the breakdown:

79 of the 385 or 20.5% were outright build failures
231 of the 385, or 60% were build completed with test suite failures
52 of the 385, or 13.5% were build that successfully completed
11 of the 385, or 3% were builds that timed out (probably infrastructure
related)
12 of the 385, or 3% were builds that were fixed

Over 80% of all builds across all projects failed or had testsuite
failures.

Andy

On Thu, 2007-06-21 at 11:39 -0400, Bill Burke wrote:

> The productivity of teams like EJB 3.0 has been severely compromised for 
> over a year because projects.  60% of the reason I didn't want to work 
> on EJB 3 anymore was because I would not commit anything for a week or 
> two and come back to find that half of my unit tests were broken.  This 
> has gotten worse and worse as projects have split off from jboss-head.
> 
> There is ZERO peer pressure for breaking the build.  Nobody cares. 
> Since the testsuites *ALWAYS* fail, nobody is paying attention to 
> regressions except for the individual projects where it fails.  You even 
> have cases where people comment out failing tests!  We just *cannot* do 
> the refactorings that the majority of teams want to do without a stable 
> testsuite.
> 
> This has to be fixed immediately.  Since there is ZERO peer pressure, 
> there needs to be consequences for breaking the build or regressing.  I 
> propose the following:
> 
> 1. Calculate a baseline of passed vs. failed tests
> 2. Tag HEAD
> 3. If there is any testsuite regression or build breakage, freeze SVN 
> until the build or regressions are fixed.
> 4. If build or regressions are fixed within 24 hours.  Rollback to 
> previous tag
> 5. If no regressions or breakage, recalculate baseline and tag head
> 6. GOTO 3
> 
> Bill
> 
> Dimitris Andreadis wrote:
> > Given the multitude of component updates and our carelessness, it's a 
> > rare thing to see jboss AS testsuites run at 100%, A lot of stuff is 
> > just checked in without testing it locally, and worse, the CC runs that 
> > show failures after the check-ins are ignored.
> > 
> > The good new is we finally got there for 4.2.x. (minus 
> > occasional/transient timing failures), so let's keep it that way for 
> > that branch, and help fix the other ones, especially HEAD.
> > 
> > On a broken testsuite it's very "convenient" to just ignore failures, 
> > since "somebody else must have done it".
> > 
> > Cheers
> > /Dimitris
> > 
> > ===========================
> > 
> > View results here ->
> > http://cruisecontrol.jboss.com/cc/buildresults/jboss-4.2-testsuite-sun-1.5?log=log20070620223106Lbuild.48 
> > 
> > 
> > BUILD COMPLETE -  build.48
> > Date of build: 06/20/2007 22:31:06
> > Time to build: 165 minutes 8 seconds
> > 
> >  Unit Tests: (4087)  Total Errors and Failures: (0)
> > All Tests Passed
> > _______________________________________________
> > jboss-development mailing list
> > jboss-development at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/jboss-development
> 

Andrig (Andy) Miller
VP of Engineering
JBoss, a division of Red Hat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-development/attachments/20070621/7141670c/attachment.html