[jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

[Brian Stansberry]

On 5/7/12 9:25 AM, Jason T. Greene wrote:
> On 5/7/12 5:19 AM, Darran Lofthouse wrote:
>> On 05/07/2012 02:34 AM, Brian Stansberry wrote:
>>> Nice!
>>>
>>> However, if things have been committed since you pushed and it's not a
>>> big imposition, it's nice if you rebase to trigger a retest. It's a more
>>> accurate test, and if there are any rebase issues you'll find out early.
>>
>> Failures are so frequent now that the need for a rebase seems to happen
>> quite rarely for me.
>>
>> One thing that would be nice would be if we could somehow capture which
>> tests regularly fail so those tests can either be fixed or replaced.
>
> I am tired of them as well. We fixed most of them, but the problem we
> run into is that some of the intermittent failures are actually critical
> tests. For example the clustering tests for failover can fail
> intermittently, but we can't just disable them.
>
> For 7.2 I want to eliminate them. I'm thinking maybe we need a new rule,
> if any test fails intermittently, it will be disabled and the author
> gets a blocker jira to fix it. What do you guys think, too draconian?
>

Make it Critical.

You're always having to fight people who assign "Blocker" to stuff that 
wouldn't actually block a release. So better IMHO is to use Critical 
unless you feel not getting the test sorted would actually block a release.


-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat