[jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

[Jason T. Greene]

On 5/7/12 5:19 AM, Darran Lofthouse wrote:
> On 05/07/2012 02:34 AM, Brian Stansberry wrote:
>> Nice!
>>
>> However, if things have been committed since you pushed and it's not a
>> big imposition, it's nice if you rebase to trigger a retest. It's a more
>> accurate test, and if there are any rebase issues you'll find out early.
>
> Failures are so frequent now that the need for a rebase seems to happen
> quite rarely for me.
>
> One thing that would be nice would be if we could somehow capture which
> tests regularly fail so those tests can either be fixed or replaced.

I am tired of them as well. We fixed most of them, but the problem we 
run into is that some of the intermittent failures are actually critical 
tests. For example the clustering tests for failover can fail 
intermittently, but we can't just disable them.

For 7.2 I want to eliminate them. I'm thinking maybe we need a new rule, 
if any test fails intermittently, it will be disabled and the author 
gets a blocker jira to fix it. What do you guys think, too draconian?



-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat