[infinispan-dev] Let's stop with pull requests

Thu May 31 05:33:46 EDT 2012

On May 29, 2012, at 12:48 PM, Manik Surtani wrote:

> I pretty much agree with this; and here's a bit of history.  
> 
> For the large part we have had a stable test suite, but the occasional unpredictability in the suite came in when we introduced the parallel test runner, to allow us to run the (core) suite in under 5 minutes - a suite which otherwise took over 2 hours when run sequentially.  
> 
> We could revert back to just using the sequential test runner if people prefer that - it makes the suite run more predictably and hence easier to debug and maintain - but the drawback is, well, it takes 2 hours to run.  
> 
> Perhaps we should use the parallel suite as a "smoke test", but in the event of any failures, revert to a run using the sequential suite?

I did some thinking on this and here's my view:

There're tests that sometimes are sensitive to thread scheduling. We all have pretty powerful machines and often we won't see these issues, but when we go in CI, we might see them.

What happens is that CI often uses machines that are less powerful, and if running a paralell testsuite in less powerful machines, these thread scheduling errors can come up more often.

One way to solve this would be for individuals to run the testsuite in paralell, and when we go in CI, run it sequentially.

This is what JDG is doing and trust me, it can highlight different issues in our test suite (seen it already), but at least, the thread scheduling issues are less common.

Thoughts?

> 
> On 29 May 2012, at 11:02, Sanne Grinovero wrote:
> 
>> On 29 May 2012 10:24, Galder Zamarreño <galder at redhat.com> wrote:
>>> BuildHive might help somewhat (when it fully works...) but the real problem testsuites will continue to be there:
>>> 
>>> 1. Some tests fail randomly, due to concurrency issues or any kind.
>>> 2. Some tests fail due to differences in environments (CI vs your own machine)
>>> 
>>> The problem with this tests will continue as long as people are individually tasked to tackle these failures. I'm yet to see people proactively fixing this stuff, without any external intervention. As long as this is like this, we'll always be reacting to things.
>> 
>> +1, that's my point. I've been spending much time on the testsuite,
>> not only in my areas of know-how, and proactively improving things.
>> Still, if when I leave the project for a month and when I come back
>> the situation is worse, I feel like I'm fighting windmills. Worst is,
>> I feel I'm fighting alone and others don't care. Therefore I highly
>> appreciated your help last week Galder, it's not about the couple
>> of tests, it's the feeling of someone helping and caring for it.
>> 
>>> 
>>> Either *we all* behave like Sanne suggests, or we carry on like we've done so far.
>>> 
>>> And then thing is that there's a balance to be found between chasing down this annoying failures, and moving the project forward with new functionality, helping paying customers, helping community, fixing bugs that affect functionality, evangelise….etc.
>> 
>> I think we all want that, but there is a disagreement on how we think
>> we should act to reach the common goals. Of course YMMV, but in my
>> mileage the feeling is that the energy spent around failing tests and
>> working around broken functionality far out weights my
>> time in developing new improvements.
>> 
>> My new Lucene CacheLoader is an example; the bill for it:
>> 
>> - 3 hours coding it
>> - 4 hours figuring out problems with configuring it - problems which
>> were already highlighted by existing tests
>> - 4 hours fixing tests and debugging core - yea I'm slow at it as I'm
>> not familiar with it
>> [desisted from fixing it all]
>> - 1 hour working around all core issues by hacking hardcoded configuration
>> - 2 hours testing it: it works for me (but can't contribute it back as
>> configuration is hacked) and performance is great: much better than
>> using our other CacheLoaders.
>> 
>> Advantage: ZERO as I can't deliver the value. More especially, I just
>> had a couple of hours to make this and I'm long overspilled the
>> timebox (which started in my last weekend anyway), so since I have now
>> to work on more urgent things, this situation will likely lead to a
>> couple of months delay in delivering the feature.
>> 
>> On the "motivation" side of things, this also learns me that I'll now
>> better were to contribute my free time nice ideas in the next weekend.
>> 
>> Conclusion: yes there is a balance. It might work for someone who is
>> working on Infinispan full time and able to follow all issues and is
>> familiar with all changes. It doesn't work at all for other people.
>> 
>> Lastly, my position is that tests are far more important than our
>> implementation code; implementation is rewritten all the time, but I
>> need the solid testsuite to tell if I'm doing right and make sure our
>> continual rewriting of internal components make *progress* rather than
>> pushing evolution back. As I said, we are having a team of impressive
>> people, and each is having great ideas rewriting things, but nobody
>> writes perfect code so if we stick with "rewriting" components and
>> internal services every time to address some performance/functional
>> problem, we also introduce new bugs. The value in which the project
>> makes real progress, is the improvement of the tests getting more
>> solid around all things we fix.
>> 
>> I had this plan to rewrite the FilesystemCacheStore, how would I do
>> it? it needs a full re-design, so reading the current code won't help
>> much, but I have some cool ideas I could try out in a weekend,
>> especially since I have performance tests ready.
>> Still to want to take advantage of the bugs resolved in the past, I'd
>> expect to inherit a solid testsuite around it which would enable me to
>> focus on performance. Still, I'm not going to start this task because
>> the testsuite is not instilling me enough confidence to be able to
>> fearlessly address my quest; result once more: you won't get my
>> patches, not this summer at least.
>> 
>> Just saying, this balance you mention is correct, but is very delicate
>> and feels very differently for part-time contributors, and I guess
>> much worse even for potential contributors.
>> 
>> Cheers,
>> Sanne
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> --
> Manik Surtani
> manik at jboss.org
> twitter.com/maniksurtani
> 
> Lead, Infinispan
> http://www.infinispan.org
> 
> 
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache