[jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

[Jason T. Greene]

On 5/7/12 9:33 AM, Brian Stansberry wrote:
> On 5/7/12 9:25 AM, Jason T. Greene wrote:
>> On 5/7/12 5:19 AM, Darran Lofthouse wrote:
>>> On 05/07/2012 02:34 AM, Brian Stansberry wrote:
>>>> Nice!
>>>>
>>>> However, if things have been committed since you pushed and it's not a
>>>> big imposition, it's nice if you rebase to trigger a retest. It's a more
>>>> accurate test, and if there are any rebase issues you'll find out early.
>>>
>>> Failures are so frequent now that the need for a rebase seems to happen
>>> quite rarely for me.
>>>
>>> One thing that would be nice would be if we could somehow capture which
>>> tests regularly fail so those tests can either be fixed or replaced.
>>
>> I am tired of them as well. We fixed most of them, but the problem we
>> run into is that some of the intermittent failures are actually critical
>> tests. For example the clustering tests for failover can fail
>> intermittently, but we can't just disable them.
>>
>> For 7.2 I want to eliminate them. I'm thinking maybe we need a new rule,
>> if any test fails intermittently, it will be disabled and the author
>> gets a blocker jira to fix it. What do you guys think, too draconian?
>>
>
> Make it Critical.
>
> You're always having to fight people who assign "Blocker" to stuff that
> wouldn't actually block a release. So better IMHO is to use Critical
> unless you feel not getting the test sorted would actually block a release.

Well as an example, can we release with the clustering failover tests 
disabled? I guess maybe if there is a manual run or something.

-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat