On Wed, May 30, 2012 at 5:19 PM, Manik Surtani <manik(a)jboss.org> wrote:
On 30 May 2012, at 14:58, Sanne Grinovero wrote:
I see two issues with your plan, though:
1. Buildhive is limited to 15 mins, and a reviewer wouldn't
necessarily wait for 2 hours to integrate a pull request anyway. So
the sequential build would be limited to Jenkins runs.
2. How do we select which tests run where? I remember we had to
disable tests precisely because configuring test groups in
testng/surefire didn't work.
I wouldn't oppose a good plan just because of some minor technical
difficulties. We can solve those in many ways.
# improve Arquillian
Yup.
That would be nice, but not vital as long as we have a solution to
exclude Arquillian tests from manual/buildhive runs.
# make-your-own BuildHive by using same integration / have them lift
limitations. (i.e. trust Galder and give him some time..)
Yup. Galder is working with CloudBees to lift this restriction. We could
also always ping our friends at OpenShift.
Still, it doesn't make sense to wait 2 hours every time after every
pull request update to see that it's fine in buildhive. So I imagine
buildhive will continue running tests in parallel.
# fix/workaround TestNG / switch to JUnit / make your own
At some point it's simpler to just fix the tests...
Test groups do work, just overriding them on the command line was flaky.
The correct approach is to use Maven profiles. Solution: fix Maven POMs.
Hmm, I see our POMs already have different profiles setting the test
groups, and maven really doesn't run manual tests by default. Nice!
Yet there are still lots of tests that have both group="manual" and
enabled="false", with descriptions like "Disabled until we can
configure Surefire to skip manual tests". You can't blame me for
thinking the comment is still valid :)
So now I have a counter-suggestion:
1. Add a 'flaky-' prefix to the group name of tests that fail randomly
(or always!) instead of disabling them.
2. Always run tests in parallel, forget about sequential runs as they
take way too long. Many of the changes we'd have to make so that tests
take less time sequentially will make it easier for those tests to
fail when they run in parallel.
3. Configure a separate build in jenkins to run flaky tests as well,
let everyone else run only non-flaky ones.
4. Create a JIRA for fixing flaky tests in general, create separate
JIRAs only where the fix changes production code.
5. Profit!