[jboss-dev] AS testsuites

[Ovidiu Feodorov]

Outsource

Tim Fox wrote:
> One related issue:
>
> Our project's CC run takes a minimum of 3 hours to complete, and we 
> repeat that against each database we support: MySQL, PostgreSQL, 
> MSSQL, Sybase, Oracle, i.e. 5 combinations.
>
> The cc machine is so overstretched, that it takes about 8 days (!!) to 
> complete all of them.
>
> This basically makes the whole concept of continuous build irrelevant, 
> so until QA gets more machines we pretty much ignore those results, 
> and just make sure we get clean runs before releases.
>
> Andrig T Miller wrote:
>> To strengthen your argument, I had 385 build messages in my e-mail 
>> and here is the breakdown:
>>
>> 79 of the 385 or 20.5% were outright build failures
>> 231 of the 385, or 60% were build completed with test suite failures
>> 52 of the 385, or 13.5% were build that successfully completed
>> 11 of the 385, or 3% were builds that timed out (probably 
>> infrastructure related)
>> 12 of the 385, or 3% were builds that were fixed
>>
>> Over 80% of all builds across all projects failed or had testsuite 
>> failures.
>>
>> Andy
>>
>> On Thu, 2007-06-21 at 11:39 -0400, Bill Burke wrote:
>>> The productivity of teams like EJB 3.0 has been severely compromised 
>>> for over a year because projects.  60% of the reason I didn't want 
>>> to work on EJB 3 anymore was because I would not commit anything for 
>>> a week or two and come back to find that half of my unit tests were 
>>> broken.  This has gotten worse and worse as projects have split off 
>>> from jboss-head.
>>>
>>> There is ZERO peer pressure for breaking the build.  Nobody cares. 
>>> Since the testsuites *ALWAYS* fail, nobody is paying attention to 
>>> regressions except for the individual projects where it fails.  You 
>>> even have cases where people comment out failing tests!  We just 
>>> *cannot* do the refactorings that the majority of teams want to do 
>>> without a stable testsuite.
>>>
>>> This has to be fixed immediately.  Since there is ZERO peer 
>>> pressure, there needs to be consequences for breaking the build or 
>>> regressing.  I propose the following:
>>>
>>> 1. Calculate a baseline of passed vs. failed tests
>>> 2. Tag HEAD
>>> 3. If there is any testsuite regression or build breakage, freeze 
>>> SVN until the build or regressions are fixed.
>>> 4. If build or regressions are fixed within 24 hours.  Rollback to 
>>> previous tag
>>> 5. If no regressions or breakage, recalculate baseline and tag head
>>> 6. GOTO 3
>>>
>>> Bill
>>>
>>> Dimitris Andreadis wrote:
>>> > Given the multitude of component updates and our carelessness, 
>>> it's a > rare thing to see jboss AS testsuites run at 100%, A lot of 
>>> stuff is > just checked in without testing it locally, and worse, 
>>> the CC runs that > show failures after the check-ins are ignored.
>>> > > The good new is we finally got there for 4.2.x. (minus > 
>>> occasional/transient timing failures), so let's keep it that way for 
>>> > that branch, and help fix the other ones, especially HEAD.
>>> > > On a broken testsuite it's very "convenient" to just ignore 
>>> failures, > since "somebody else must have done it".
>>> > > Cheers
>>> > /Dimitris
>>> > > ===========================
>>> > > View results here ->
>>> > 
>>> http://cruisecontrol.jboss.com/cc/buildresults/jboss-4.2-testsuite-sun-1.5?log=log20070620223106Lbuild.48 
>>> > > > BUILD COMPLETE -  build.48
>>> > Date of build: 06/20/2007 22:31:06
>>> > Time to build: 165 minutes 8 seconds
>>> > >  Unit Tests: (4087)  Total Errors and Failures: (0)
>>> > All Tests Passed
>>> > _______________________________________________
>>> > jboss-development mailing list
>>> > jboss-development at lists.jboss.org 
>>> <mailto:jboss-development at lists.jboss.org>
>>> > https://lists.jboss.org/mailman/listinfo/jboss-development
>>>
>>>     
>> Andrig (Andy) Miller
>> VP of Engineering
>> JBoss, a division of Red Hat
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> jboss-development mailing list
>> jboss-development at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-development
>>   
>
> _______________________________________________
> jboss-development mailing list
> jboss-development at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-development