On Thu, May 31, 2012 at 12:40 PM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
On 31 May 2012 10:33, Galder ZamarreƱo <galder(a)redhat.com>
wrote:
>
> On May 29, 2012, at 12:48 PM, Manik Surtani wrote:
>
>> I pretty much agree with this; and here's a bit of history.
>>
>> For the large part we have had a stable test suite, but the occasional
unpredictability in the suite came in when we introduced the parallel test runner, to
allow us to run the (core) suite in under 5 minutes - a suite which otherwise took over 2
hours when run sequentially.
>>
>> We could revert back to just using the sequential test runner if people prefer
that - it makes the suite run more predictably and hence easier to debug and maintain -
but the drawback is, well, it takes 2 hours to run.
>>
>> Perhaps we should use the parallel suite as a "smoke test", but in the
event of any failures, revert to a run using the sequential suite?
>
> I did some thinking on this and here's my view:
>
> There're tests that sometimes are sensitive to thread scheduling. We all have
pretty powerful machines and often we won't see these issues, but when we go in CI, we
might see them.
>
> What happens is that CI often uses machines that are less powerful, and if running a
paralell testsuite in less powerful machines, these thread scheduling errors can come up
more often.
>
> One way to solve this would be for individuals to run the testsuite in paralell, and
when we go in CI, run it sequentially.
>
> This is what JDG is doing and trust me, it can highlight different issues in our test
suite (seen it already), but at least, the thread scheduling issues are less common.
>
> Thoughts?
I also thought it was a matter of "powerful", but the opposite way ;-)
Tests fail very often to me, much worse than on CI, and I always ended
up blaming the more powerful environment I'm running them in.
FYI I'm unable to complete any test run on master, testsuite just
hangs. When it succeeds, I always had more failures than what I see on
CI: even while CI has been occasionally happy (like in February), I
never had a fully stable build, even when running it on a single
thread.
Sorry if that wasn't clear, maybe I should complain about it more often :-P
Seriously, I just think we can design the tests better.
Do complain more often :)
I got used to running only the core test suite all the time - I rely
on the Jenkins results to check that I haven't broken everything else.
@Dan
sure you can reconfigure the CDI pom.xml to override the surefire settings.
Adi checked at the CDI pom.xml and surefire actually *was* configured
to run tests sequentially.
I guess the setting was just ignored for a a few builds - now it's
back to running in one thread and there are no CDI tests failing in
the last build.
Cheers
Dan