[jboss-as7-dev] IMPORTANT - New Policy Proposal - No More Intermittent Test Failures

[Thomas Diesler]

I agree with part one of the policy - @Ignore the failing tests and 
create a jira.

The silent removal of a test if nothing happens is problematic. Instead 
I propose to subsequently increase priority on the issue. The person how 
is supposed to make the next progress step is supposed to work on 
blocking issues (i.e. the failing test) first before anything else would 
get pulled. The issue can be closed as "won't fix" if there is agreement 
(documented in the issue) that we don't loose significant test coverage 
for a functional area.

On 07/09/2012 08:16 PM, Jason T. Greene wrote:
> We always have the problem of having a set of tests which fail one out
> of 10 runs, but we leave the test around hoping one day someone will fix
> it. The problem is no one does, and it makes regression catching hard.
> Right now people that submit pull requests have to scan through test
> results and ask around to figure out if they broke something or not.
>
> So I propose a new policy. Any test which intermittently fails will be
> ignored and a JIRA opened to the author for up to a month. If that test
> is not passing in one month time, it will be removed from the codebase.
>
> The biggest problem with this policy is that we might completely lose
> coverage. A number of the clustering tests for example fail
> intermittently, and if we removed them we would have no other coverage.
> So for special cases like clustering, I am thinking of relocating them
> to a different test run called "broken-clustering", or something like
> that. This run would only be monitored by those working on clustering,
> and would not be included in the main "all tests" run.
>
> Any other ideas?
>

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx