On 30 May 2012, at 18:06, Manik Surtani wrote:
On 30 May 2012, at 17:22, Dan Berindei wrote:
>
> Yet there are still lots of tests that have both group="manual" and
> enabled="false", with descriptions like "Disabled until we can
> configure Surefire to skip manual tests". You can't blame me for
> thinking the comment is still valid :)
Find out who added this and educate them. :-)
>
> So now I have a counter-suggestion:
> 1. Add a 'flaky-' prefix to the group name of tests that fail randomly
> (or always!) instead of disabling them.
> 2. Always run tests in parallel, forget about sequential runs as they
> take way too long. Many of the changes we'd have to make so that tests
> take less time sequentially will make it easier for those tests to
> fail when they run in parallel.
> 3. Configure a separate build in jenkins to run flaky tests as well,
> let everyone else run only non-flaky ones.
> 4. Create a JIRA for fixing flaky tests in general, create separate
> JIRAs only where the fix changes production code.
> 5. Profit!
+1, that's essentially what I have been getting at.
+1, this sounds very
reasonable.