Re: [jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

On 05/07/2012 04:39 PM, Jason T. Greene wrote: > On 5/7/12 9:33 AM, Brian Stansberry wrote: >> On 5/7/12 9:25 AM, Jason T. Greene wrote: >>> On 5/7/12 5:19 AM, Darran Lofthouse wrote: >>>> On 05/07/2012 02:34 AM, Brian Stansberry wrote: >>>>> Nice! >>>>> >>>>> However, if things have been committed since you pushed and it's >>>>> not a >>>>> big imposition, it's nice if you rebase to trigger a retest. It's >>>>> a more >>>>> accurate test, and if there are any rebase issues you'll find out >>>>> early. >>>> >>>> Failures are so frequent now that the need for a rebase seems to >>>> happen >>>> quite rarely for me. >>>> >>>> One thing that would be nice would be if we could somehow capture >>>> which >>>> tests regularly fail so those tests can either be fixed or replaced. >>> >>> I am tired of them as well. We fixed most of them, but the problem we >>> run into is that some of the intermittent failures are actually >>> critical >>> tests. For example the clustering tests for failover can fail >>> intermittently, but we can't just disable them. >>> >>> For 7.2 I want to eliminate them. I'm thinking maybe we need a new >>> rule, >>> if any test fails intermittently, it will be disabled and the author >>> gets a blocker jira to fix it. What do you guys think, too draconian? >>> >> >> Make it Critical. >> >> You're always having to fight people who assign "Blocker" to stuff that >> wouldn't actually block a release. So better IMHO is to use Critical >> unless you feel not getting the test sorted would actually block a >> release. > > Well as an example, can we release with the clustering failover tests > disabled? I guess maybe if there is a manual run or something. I agree it's a problem. Maybe could we run the default cluster tests on sync replication?

Because these particular ones are all positive tests and I don't remember there would be justification apart from using UDP and ASYNC replication that these tests could fail intermittently. I have seen some test failures in the past were indeed regular failures, its just "luck" that they didn't fail that often... Rado