[infinispan-dev] Let's stop with pull requests

[Galder Zamarreño]

IMO, test groups + maven integration to run this  = PITA

On May 30, 2012, at 6:22 PM, Dan Berindei wrote:

> On Wed, May 30, 2012 at 5:19 PM, Manik Surtani <manik at jboss.org> wrote:
>> 
>> On 30 May 2012, at 14:58, Sanne Grinovero wrote:
>> 
>> 
>> I see two issues with your plan, though:
>> 
>> 1. Buildhive is limited to 15 mins, and a reviewer wouldn't
>> 
>> necessarily wait for 2 hours to integrate a pull request anyway. So
>> 
>> the sequential build would be limited to Jenkins runs.
>> 
>> 2. How do we select which tests run where? I remember we had to
>> 
>> disable tests precisely because configuring test groups in
>> 
>> testng/surefire didn't work.
>> 
>> 
>> I wouldn't oppose a good plan just because of some minor technical
>> difficulties. We can solve those in many ways.
>> # improve Arquillian
>> 
>> 
>> Yup.
>> 
> 
> That would be nice, but not vital as long as we have a solution to
> exclude Arquillian tests from manual/buildhive runs.
> 
>> # make-your-own BuildHive by using same integration / have them lift
>> limitations. (i.e. trust Galder and give him some time..)
>> 
>> 
>> Yup.  Galder is working with CloudBees to lift this restriction.  We could
>> also always ping our friends at OpenShift.
>> 
> 
> Still, it doesn't make sense to wait 2 hours every time after every
> pull request update to see that it's fine in buildhive. So I imagine
> buildhive will continue running tests in parallel.
> 
>> # fix/workaround TestNG / switch to JUnit / make your own
>> 
> 
> At some point it's simpler to just fix the tests...
> 
>> 
>> Test groups do work, just overriding them on the command line was flaky.
>>  The correct approach is to use Maven profiles.  Solution: fix Maven POMs.
>> 
> 
> Hmm, I see our POMs already have different profiles setting the test
> groups, and maven really doesn't run manual tests by default. Nice!
> 
> Yet there are still lots of tests that have both group="manual" and
> enabled="false", with descriptions like "Disabled until we can
> configure Surefire to skip manual tests". You can't blame me for
> thinking the comment is still valid :)
> 
> So now I have a counter-suggestion:
> 1. Add a 'flaky-' prefix to the group name of tests that fail randomly
> (or always!) instead of disabling them.
> 2. Always run tests in parallel, forget about sequential runs as they
> take way too long. Many of the changes we'd have to make so that tests
> take less time sequentially will make it easier for those tests to
> fail when they run in parallel.
> 3. Configure a separate build in jenkins to run flaky tests as well,
> let everyone else run only non-flaky ones.
> 4. Create a JIRA for fixing flaky tests in general, create separate
> JIRAs only where the fix changes production code.
> 5. Profit!
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache