[wildfly-dev] IMPORTANT - Intermittent Test Ban Redux

[Jason Greene]

Hi Everyone,

Almost 2 years ago we agreed to disable all intermittent tests[1]; however, this is still a major problem today, and is still a major hassle for everyone that submits a PR (“retest this please” is now a common part of our process).

I think it’s finally time to enforce this policy, and disallow all tests that lead to even occasional intermittent failures, even if that leads to loss of coverage. The most fundamental need of our test suite, is to catch regressions, and right now it’s not really serving that purpose.

So I propose that after 8.1.0.Final is shipped that we immediately disable all tests with a failing history, and create a JIRA issue for the component author to fix them or the bug that is producing the failure. If after 3 weeks, a JIRA issue has for this has made no progress, I propose we completely remove the test from our codebase. 

In addition, I propose that any repaired test be shown to be intermittent free using a custom CI job (Tomaz can set them) for several runs before accepting them back in.

Thanks!

[1] http://wildfly-development.1055759.n5.nabble.com/IMPORTANT-New-Policy-Proposal-No-More-Intermittent-Test-Failures-td5709984.html#a5709993
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat