[wildfly-dev] The CI Inquisition!

[Tomaž Cerar]

I have unmuted all muted tests on CI.

so if you start seeing more test failures, PR with @Ignore as Jason said
should be sent.


--
tomaz

On Fri, Jul 17, 2015 at 6:26 PM, Jason Greene <jason.greene at redhat.com>
wrote:

> Hello Everyone!
>
> Over the last few years we have attempted to get a grip on the
> intermittent test failures we see in CI almost every run, and have
> attempted various policies which were poorly enforced. This was mainly due
> to an understanding of tight deadline, but also a concern that strict
> enforcement would damage our test coverage. I would find myself holding out
> in hope that over time we would see these issues decrease.
>
> Unfortunately that hope was misplaced, and the simple truth is that
> coverage is already harmed, because no one pays attention to the
> intermittent fails, since they are treated as noise and not a true defect.
> I think we have no other option but to @Ignore any and every intermittent
> test, no matter the number or area they appear in.
>
> So unless someone can propose another workable strategy, intermittent
> tests should be disabled immediately by anyone that is impacted by them
> (reviewers, contributors, whoever). To do this, a simple short independent
> PR should be sent in, and a note should be left in a comment to the
> component lead of the area tested (using github’s super cool @user syntax).
> The component lead can then act on the message and choose whether to open
> and assign a JIRA to have it fixed, to ignore it, to delete it, to rewrite,
> etc. The intention is that the process of disabling an intermittent test
> should be as minimal effort as possible.
>
> In order for a known intermittent test to be reenabled, it needs to be
> shown that it is no longer intermittent by completing many full testsuite
> runs successfully. Anyone that needs it can request a custom CI job run for
> this purpose.
>
> Finally, if you are a saint and decide to fix an intermittent failing
> test, but have trouble reproducing it, feel free to raise the topic in the
> WildFly HipChat and we can come up with a solution like custom debug test
> jobs that can get you the info you need.
>
> Thoughts?
>
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/wildfly-dev/attachments/20150720/df294761/attachment.html