On Wed, Apr 23, 2014 at 3:21 PM, Jason Greene <jason.greene(a)redhat.com>
Almost 2 years ago we agreed to disable all intermittent tests;
however, this is still a major problem today, and is still a major
hassle for everyone that submits a PR (“retest this please” is
now a common part of our process).
I think it’s finally time to enforce this policy, and disallow all
tests that lead to even occasional intermittent failures, even if
that leads to loss of coverage. The most fundamental need of our test
suite, is to catch regressions, and right now it’s not really
serving that purpose.
So I propose that after 8.1.0.Final is shipped that we immediately
disable all tests with a failing history, and create a JIRA issue for
the component author to fix them or the bug that is producing the
failure. If after 3 weeks, a JIRA issue has for this has made no
progress, I propose we completely remove the test from our codebase.
In addition, I propose that any repaired test be shown to be
intermittent free using a custom CI job (Tomaz can set them) for
several runs before accepting them back in.
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
wildfly-dev mailing list