[infinispan-dev] infinispan test suite, reloaded

Mon Jul 14 09:43:07 EDT 2014

On Jul 10, 2014, at 16:12, Sanne Grinovero <sanne at infinispan.org> wrote:

> On 10 July 2014 15:13, Mircea Markus <mmarkus at redhat.com> wrote:
>> 
>> On Jul 10, 2014, at 15:03, Sanne Grinovero <sanne at infinispan.org> wrote:
>> 
>>> The important point for me is that patches don't get merged if they
>>> introduce any regression. I hope that rule stays?
>>> BTW this matches with the "classic" approach as far as I know it.
>> 
>> yes. A patch might pass the test when integrated and cause intermittent failures later on, so it's not straight forward to avoid patches introducing regressions, nor to identify which patch has caused it so that we can roll it back.
> 
> It's self-speaking that it's hard to immediately evaluate if a patch
> is going to cause intermittent / time-bound failures in the future: we
> have no crystal ball, so I'm not making absurd demands. There will
> always be cases in which developers will need to ask forgiveness.
> 
> But if anything in the test run fails during a review of a patch, and
> this patch still gets merged because "the cause is likely unrelated",
> or worse "we'll fix that problem later" that's unacceptable and needs
> to be investigated further before being merged.
> I always did that, and spending a lot of time on things which are not
> directly related on my goals, for sake of respect of other developers
> in the team and I expect no less from everyone else: when the
> testsuite doesn't pass I can't make progress on my own work and need
> to necessarily shift to firefighthing. Or go on holidays.
> That's what necessarily needs to happen when a bad patch slips in past
> our guard, even if doing so you need to reschedule other tasks:
> because otherwise it's other people, and more and more people needing
> to reschedule their own tasks. Worse yet, the other people who will
> need to look at it will not have any of the context to understand what
> might be going on. So I hope we're on the same page with this, because
> ultimately it's about respect for each person contributing or working
> on it.

I understand your frustration and you are absolutely right. 

> 
> I'm also puzzled on why exactly the current situation is not
> sustainable. Sure there is a lot of technical debt to pay, but you
> know debts come with interests and it's hard.I've also suggested many
> ways to improve our testsuite to make our life easier, like use
> Byteman, mock the timers via a TimeService, get rid of TestNG to move
> to something more reliable, remove unnecessary dependencies from each
> module, use more mocking in areas where we don't actually have an
> interest in testing JGroups..

IMO the problem is that ATM we lack the discipline, as a team, to maintain the suite green. In the last 7 years I've been working on JBossCache and then ISPN it has always been like this - intermittent failures - and always to big and too ugly of a task to fix it and maintain it green, especially with the schedule we had ahead of us. Now that the team is big, this compromise - intermittent failures - really impacts the productivity of a lot of people. The suite should stay green and we should treat any failure that keep us away from that as an blocker - a thing we didn't really do up to now.

> Nothing like that was done, so you can't say in all fairness that we
> tried to do better. My PR which is the first step to get rid of TestNG
> from the Query modules is open since 43 days.. so don't expect me to
> fix any problem quickly, it's not particularly motivating.
> 
> You can try playing with processes, and I don't disagree with idea,
> but when I'm able to find a regression is is that I will do: I'll
> revert all commits until the first stable point I find. I've
> suggested this approach to you as well, not sure why you don't apply
> it more regularly?

Because the last green commit will trigger intermittent failures, try it.

> There is no shame in reverting commits, especially
> if we do it regularly.
> It's stable today, so I'll make a note of this commit id, my crystal
> ball is telling that it will be useful soon ;-)
> 
> Sanne
> 
> 
>> 
>>> 
>>> On 10 July 2014 14:04, Mircea Markus <mmarkus at redhat.com> wrote:
>>>> I just had a chat with Dan and we don't think the current process for the test suite works. Not hard to see why, the suite is almost never green. So we will adopt a more classic and simple approach: if a test fails a blocker JIRA is created for it and assigned to a component lead then team member who'll start working on it *immediately*. Dan will be watch dog starting today so please expect blocker JIRAs coming your way and treat them accordingly.
>>>> 
>>>> Cheers,
>>>> --
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> Cheers,
>> --
>> Mircea Markus
>> Infinispan lead (www.infinispan.org)
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)