Re: [jboss-as7-dev] Intermittent Tests (Re: Pull Request Retesting Ability)

On 5/7/12 9:33 AM, Brian Stansberry wrote: > On 5/7/12 9:25 AM, Jason T. Greene wrote: >> On 5/7/12 5:19 AM, Darran Lofthouse wrote: >>> On 05/07/2012 02:34 AM, Brian Stansberry wrote: >>>> Nice! >>>> >>>> However, if things have been committed since you pushed and it's not a >>>> big imposition, it's nice if you rebase to trigger a retest. It's a >>>> more >>>> accurate test, and if there are any rebase issues you'll find out >>>> early. >>> >>> Failures are so frequent now that the need for a rebase seems to happen >>> quite rarely for me. >>> >>> One thing that would be nice would be if we could somehow capture which >>> tests regularly fail so those tests can either be fixed or replaced. >> >> I am tired of them as well. We fixed most of them, but the problem we >> run into is that some of the intermittent failures are actually critical >> tests. For example the clustering tests for failover can fail >> intermittently, but we can't just disable them. >> >> For 7.2 I want to eliminate them. I'm thinking maybe we need a new rule, >> if any test fails intermittently, it will be disabled and the author >> gets a blocker jira to fix it. What do you guys think, too draconian? >> > > Make it Critical. > > You're always having to fight people who assign "Blocker" to stuff that > wouldn't actually block a release. So better IMHO is to use Critical > unless you feel not getting the test sorted would actually block a > release. Well as an example, can we release with the clustering failover tests disabled? I guess maybe if there is a manual run or something.