[jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

[Brian Stansberry]

On 5/7/12 9:39 AM, Jason T. Greene wrote:
> On 5/7/12 9:33 AM, Brian Stansberry wrote:
>> On 5/7/12 9:25 AM, Jason T. Greene wrote:
>>> On 5/7/12 5:19 AM, Darran Lofthouse wrote:
>>>> On 05/07/2012 02:34 AM, Brian Stansberry wrote:
>>>>> Nice!
>>>>>
>>>>> However, if things have been committed since you pushed and it's not a
>>>>> big imposition, it's nice if you rebase to trigger a retest. It's a
>>>>> more
>>>>> accurate test, and if there are any rebase issues you'll find out
>>>>> early.
>>>>
>>>> Failures are so frequent now that the need for a rebase seems to happen
>>>> quite rarely for me.
>>>>
>>>> One thing that would be nice would be if we could somehow capture which
>>>> tests regularly fail so those tests can either be fixed or replaced.
>>>
>>> I am tired of them as well. We fixed most of them, but the problem we
>>> run into is that some of the intermittent failures are actually critical
>>> tests. For example the clustering tests for failover can fail
>>> intermittently, but we can't just disable them.
>>>
>>> For 7.2 I want to eliminate them. I'm thinking maybe we need a new rule,
>>> if any test fails intermittently, it will be disabled and the author
>>> gets a blocker jira to fix it. What do you guys think, too draconian?
>>>
>>
>> Make it Critical.
>>
>> You're always having to fight people who assign "Blocker" to stuff that
>> wouldn't actually block a release. So better IMHO is to use Critical
>> unless you feel not getting the test sorted would actually block a
>> release.
>
> Well as an example, can we release with the clustering failover tests
> disabled? I guess maybe if there is a manual run or something.
>

Sure, it's a case by case thing, and in some cases maybe it is a Blocker.

I'm just trying to reiterate to folks reading this list that Blocker 
means we'd seriously consider not releasing. It doesn't mean *anything* 
else.

-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat