[infinispan-dev] Let's stop with pull requests

[Dan Berindei]

On Wed, May 30, 2012 at 5:19 PM, Manik Surtani <manik at jboss.org> wrote:
>
> On 30 May 2012, at 14:58, Sanne Grinovero wrote:
>
>
> I see two issues with your plan, though:
>
> 1. Buildhive is limited to 15 mins, and a reviewer wouldn't
>
> necessarily wait for 2 hours to integrate a pull request anyway. So
>
> the sequential build would be limited to Jenkins runs.
>
> 2. How do we select which tests run where? I remember we had to
>
> disable tests precisely because configuring test groups in
>
> testng/surefire didn't work.
>
>
> I wouldn't oppose a good plan just because of some minor technical
> difficulties. We can solve those in many ways.
> # improve Arquillian
>
>
> Yup.
>

That would be nice, but not vital as long as we have a solution to
exclude Arquillian tests from manual/buildhive runs.

> # make-your-own BuildHive by using same integration / have them lift
> limitations. (i.e. trust Galder and give him some time..)
>
>
> Yup.  Galder is working with CloudBees to lift this restriction.  We could
> also always ping our friends at OpenShift.
>

Still, it doesn't make sense to wait 2 hours every time after every
pull request update to see that it's fine in buildhive. So I imagine
buildhive will continue running tests in parallel.

> # fix/workaround TestNG / switch to JUnit / make your own
>

At some point it's simpler to just fix the tests...

>
> Test groups do work, just overriding them on the command line was flaky.
>  The correct approach is to use Maven profiles.  Solution: fix Maven POMs.
>

Hmm, I see our POMs already have different profiles setting the test
groups, and maven really doesn't run manual tests by default. Nice!

Yet there are still lots of tests that have both group="manual" and
enabled="false", with descriptions like "Disabled until we can
configure Surefire to skip manual tests". You can't blame me for
thinking the comment is still valid :)

So now I have a counter-suggestion:
1. Add a 'flaky-' prefix to the group name of tests that fail randomly
(or always!) instead of disabling them.
2. Always run tests in parallel, forget about sequential runs as they
take way too long. Many of the changes we'd have to make so that tests
take less time sequentially will make it easier for those tests to
fail when they run in parallel.
3. Configure a separate build in jenkins to run flaky tests as well,
let everyone else run only non-flaky ones.
4. Create a JIRA for fixing flaky tests in general, create separate
JIRAs only where the fix changes production code.
5. Profit!