[jboss-as7-dev] IMPORTANT - New Policy Proposal - No More Intermittent Test Failures

[Carlo de Wolf]

+1 on the policy.

On 07/09/2012 08:16 PM, Jason T. Greene wrote:
> We always have the problem of having a set of tests which fail one out
> of 10 runs, but we leave the test around hoping one day someone will fix
> it. The problem is no one does, and it makes regression catching hard.

And with a couple of such tests in a big code base you'll never get a 
blue run.
> Right now people that submit pull requests have to scan through test
> results and ask around to figure out if they broke something or not.
>
> So I propose a new policy. Any test which intermittently fails will be
> ignored and a JIRA opened to the author for up to a month. If that test
> is not passing in one month time, it will be removed from the codebase.

What we had in the ejb3 testsuite was a mechanism which we could control 
via an xml file: 
http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/projects/ejb3/trunk/testsuite/src/test/resources/known-issues.xml?revision=100056&view=markup
That way no code needs to be changed and you can easily do a 
'known-issues' test run.

-1 on the removal. There is no incentive for the author to fix the test 
at all and we should be careful with what we lose.
I would say: the author gets one month to fix it up, after that SET is 
going to fix it up. During that time no contributions are honored from 
that author.

>
> The biggest problem with this policy is that we might completely lose
> coverage. A number of the clustering tests for example fail
> intermittently, and if we removed them we would have no other coverage.
> So for special cases like clustering, I am thinking of relocating them
> to a different test run called "broken-clustering", or something like
> that. This run would only be monitored by those working on clustering,
> and would not be included in the main "all tests" run.
>
> Any other ideas?
>

At the end of the day, you are working the left side of the field and 
you don't want to get jammed by bits coming from the right side. This is 
why I think smaller integration suites (bringing together 2 or 3 
components / techs) would make more sense instead of a large big bang 
suite. In essence you're already on that path by separating a subset of 
the clustering tests.

Carlo