On 30 May 2012 13:36, Manik Surtani <manik(a)jboss.org> wrote:
Well, one problem leads to another, as you are well aware. Flaky
parallel suite leads careless commits. Yes, we should fix what is broken at the moment
but that is not enough since it will get this way again unless we have a stable suite that
can be used to ensure quality moving fwd. The options are:
1. We always use the sequential suite.
2. We identify tests that are reliable with the parallel suite (should be at least 90%
of the tests IMO) and use this, leaving the sequential suite to BuildHive.
Ah, now I understand. You want to have some tests executed only when
run sequentially.. that looks like a nice compromise, cheers!
that will also speed up our test runs, as not all tests are executed;
we could exclude also some of the slower ones even if they could work
fine in parallel.
Sanne
My pref is #2 but that requires identifying the tests that flake out with the parallel
suite.
On 30 May 2012, at 13:25, Sanne Grinovero wrote:
> It's not really about "which" test mode we should use, as long as
> people check their commits and have any test that gives enough
> confidence, which is still not the case even during this discussion:
>
> last build Infinispan-master-JDK6-tcp-NON_PARALLEL : 123 failures / +95 failures
>
> We need to use non-parallel if that's the only choice we currently
> have, until parallel is fixed too. I agree that's time consuming, but
> that's exactly why I'm worried about the fact we leave this go this
> far. Also it doesn't really take 2 hours anymore, and slow tests can
> be speed up in many ways.. yes that's an effort, but it will pay off.
>
> Cheers,
> Sanne
>
>
>
> On 30 May 2012 13:00, Dan Berindei <dan.berindei(a)gmail.com> wrote:
>> On Tue, May 29, 2012 at 1:48 PM, Manik Surtani <manik(a)jboss.org> wrote:
>>> I pretty much agree with this; and here's a bit of history.
>>>
>>> For the large part we have had a stable test suite, but the occasional
unpredictability in the suite came in when we introduced the parallel test runner, to
allow us to run the (core) suite in under 5 minutes - a suite which otherwise took over 2
hours when run sequentially.
>>>
>>> We could revert back to just using the sequential test runner if people
prefer that - it makes the suite run more predictably and hence easier to debug and
maintain - but the drawback is, well, it takes 2 hours to run.
>>>
>>> Perhaps we should use the parallel suite as a "smoke test", but in
the event of any failures, revert to a run using the sequential suite?
>>>
>>
>> -1, a smoke test should be something that is not only faster but
>> always passes, so we could run that on each pull req. Getting a FAIL
>> from buildhive on each pull request would get tiring real quick.
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev