[jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

[Jason T. Greene]

On 5/7/12 10:32 AM, Radoslav Husar wrote:
> On 05/07/2012 04:39 PM, Jason T. Greene wrote:
>> On 5/7/12 9:33 AM, Brian Stansberry wrote:
>>> On 5/7/12 9:25 AM, Jason T. Greene wrote:
>>>> On 5/7/12 5:19 AM, Darran Lofthouse wrote:
>>>>> On 05/07/2012 02:34 AM, Brian Stansberry wrote:
>>>>>> Nice!
>>>>>>
>>>>>> However, if things have been committed since you pushed and it's
>>>>>> not a
>>>>>> big imposition, it's nice if you rebase to trigger a retest. It's
>>>>>> a more
>>>>>> accurate test, and if there are any rebase issues you'll find out
>>>>>> early.
>>>>>
>>>>> Failures are so frequent now that the need for a rebase seems to
>>>>> happen
>>>>> quite rarely for me.
>>>>>
>>>>> One thing that would be nice would be if we could somehow capture
>>>>> which
>>>>> tests regularly fail so those tests can either be fixed or replaced.
>>>>
>>>> I am tired of them as well. We fixed most of them, but the problem we
>>>> run into is that some of the intermittent failures are actually
>>>> critical
>>>> tests. For example the clustering tests for failover can fail
>>>> intermittently, but we can't just disable them.
>>>>
>>>> For 7.2 I want to eliminate them. I'm thinking maybe we need a new
>>>> rule,
>>>> if any test fails intermittently, it will be disabled and the author
>>>> gets a blocker jira to fix it. What do you guys think, too draconian?
>>>>
>>>
>>> Make it Critical.
>>>
>>> You're always having to fight people who assign "Blocker" to stuff that
>>> wouldn't actually block a release. So better IMHO is to use Critical
>>> unless you feel not getting the test sorted would actually block a
>>> release.
>>
>> Well as an example, can we release with the clustering failover tests
>> disabled? I guess maybe if there is a manual run or something.
>
> I agree it's a problem.
>
> Maybe could we run the default cluster tests on sync replication?

Yes we should move all of these to sync, and remove all Thread.sleep() etc.

>
> Because these particular ones are all positive tests and I don't
> remember there would be justification apart from using UDP and ASYNC
> replication that these tests could fail intermittently.
>
> I have seen some test failures in the past were indeed regular failures,
> its just "luck" that they didn't fail that often...
>
> Rado


-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat