[jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

[Radoslav Husar]

On 05/07/2012 04:39 PM, Jason T. Greene wrote:
> On 5/7/12 9:33 AM, Brian Stansberry wrote:
>> On 5/7/12 9:25 AM, Jason T. Greene wrote:
>>> On 5/7/12 5:19 AM, Darran Lofthouse wrote:
>>>> On 05/07/2012 02:34 AM, Brian Stansberry wrote:
>>>>> Nice!
>>>>>
>>>>> However, if things have been committed since you pushed and it's not a
>>>>> big imposition, it's nice if you rebase to trigger a retest. It's a more
>>>>> accurate test, and if there are any rebase issues you'll find out early.
>>>>
>>>> Failures are so frequent now that the need for a rebase seems to happen
>>>> quite rarely for me.
>>>>
>>>> One thing that would be nice would be if we could somehow capture which
>>>> tests regularly fail so those tests can either be fixed or replaced.
>>>
>>> I am tired of them as well. We fixed most of them, but the problem we
>>> run into is that some of the intermittent failures are actually critical
>>> tests. For example the clustering tests for failover can fail
>>> intermittently, but we can't just disable them.
>>>
>>> For 7.2 I want to eliminate them. I'm thinking maybe we need a new rule,
>>> if any test fails intermittently, it will be disabled and the author
>>> gets a blocker jira to fix it. What do you guys think, too draconian?
>>>
>>
>> Make it Critical.
>>
>> You're always having to fight people who assign "Blocker" to stuff that
>> wouldn't actually block a release. So better IMHO is to use Critical
>> unless you feel not getting the test sorted would actually block a release.
>
> Well as an example, can we release with the clustering failover tests
> disabled? I guess maybe if there is a manual run or something.

I agree it's a problem.

Maybe could we run the default cluster tests on sync replication?

Because these particular ones are all positive tests and I don't 
remember there would be justification apart from using UDP and ASYNC 
replication that these tests could fail intermittently.

I have seen some test failures in the past were indeed regular failures, 
its just "luck" that they didn't fail that often...

Rado