[jboss-dev] AS testsuites

[Tim Fox]

One related issue:

Our project's CC run takes a minimum of 3 hours to complete, and we 
repeat that against each database we support: MySQL, PostgreSQL, MSSQL, 
Sybase, Oracle, i.e. 5 combinations.

The cc machine is so overstretched, that it takes about 8 days (!!) to 
complete all of them.

This basically makes the whole concept of continuous build irrelevant, 
so until QA gets more machines we pretty much ignore those results, and 
just make sure we get clean runs before releases.

Andrig T Miller wrote:
> To strengthen your argument, I had 385 build messages in my e-mail and 
> here is the breakdown:
>
> 79 of the 385 or 20.5% were outright build failures
> 231 of the 385, or 60% were build completed with test suite failures
> 52 of the 385, or 13.5% were build that successfully completed
> 11 of the 385, or 3% were builds that timed out (probably 
> infrastructure related)
> 12 of the 385, or 3% were builds that were fixed
>
> Over 80% of all builds across all projects failed or had testsuite 
> failures.
>
> Andy
>
> On Thu, 2007-06-21 at 11:39 -0400, Bill Burke wrote:
>> The productivity of teams like EJB 3.0 has been severely compromised for 
>> over a year because projects.  60% of the reason I didn't want to work 
>> on EJB 3 anymore was because I would not commit anything for a week or 
>> two and come back to find that half of my unit tests were broken.  This 
>> has gotten worse and worse as projects have split off from jboss-head.
>>
>> There is ZERO peer pressure for breaking the build.  Nobody cares. 
>> Since the testsuites *ALWAYS* fail, nobody is paying attention to 
>> regressions except for the individual projects where it fails.  You even 
>> have cases where people comment out failing tests!  We just *cannot* do 
>> the refactorings that the majority of teams want to do without a stable 
>> testsuite.
>>
>> This has to be fixed immediately.  Since there is ZERO peer pressure, 
>> there needs to be consequences for breaking the build or regressing.  I 
>> propose the following:
>>
>> 1. Calculate a baseline of passed vs. failed tests
>> 2. Tag HEAD
>> 3. If there is any testsuite regression or build breakage, freeze SVN 
>> until the build or regressions are fixed.
>> 4. If build or regressions are fixed within 24 hours.  Rollback to 
>> previous tag
>> 5. If no regressions or breakage, recalculate baseline and tag head
>> 6. GOTO 3
>>
>> Bill
>>
>> Dimitris Andreadis wrote:
>> > Given the multitude of component updates and our carelessness, it's a 
>> > rare thing to see jboss AS testsuites run at 100%, A lot of stuff is 
>> > just checked in without testing it locally, and worse, the CC runs that 
>> > show failures after the check-ins are ignored.
>> > 
>> > The good new is we finally got there for 4.2.x. (minus 
>> > occasional/transient timing failures), so let's keep it that way for 
>> > that branch, and help fix the other ones, especially HEAD.
>> > 
>> > On a broken testsuite it's very "convenient" to just ignore failures, 
>> > since "somebody else must have done it".
>> > 
>> > Cheers
>> > /Dimitris
>> > 
>> > ===========================
>> > 
>> > View results here ->
>> > http://cruisecontrol.jboss.com/cc/buildresults/jboss-4.2-testsuite-sun-1.5?log=log20070620223106Lbuild.48 
>> > 
>> > 
>> > BUILD COMPLETE -  build.48
>> > Date of build: 06/20/2007 22:31:06
>> > Time to build: 165 minutes 8 seconds
>> > 
>> >  Unit Tests: (4087)  Total Errors and Failures: (0)
>> > All Tests Passed
>> > _______________________________________________
>> > jboss-development mailing list
>> > jboss-development at lists.jboss.org <mailto:jboss-development at lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/jboss-development
>>
>>     
> Andrig (Andy) Miller
> VP of Engineering
> JBoss, a division of Red Hat
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> jboss-development mailing list
> jboss-development at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-development
>