[infinispan-dev] infinispan test suite, reloaded

[Sanne Grinovero]

On 10 July 2014 15:13, Mircea Markus <mmarkus at redhat.com> wrote:
>
> On Jul 10, 2014, at 15:03, Sanne Grinovero <sanne at infinispan.org> wrote:
>
>> The important point for me is that patches don't get merged if they
>> introduce any regression. I hope that rule stays?
>> BTW this matches with the "classic" approach as far as I know it.
>
> yes. A patch might pass the test when integrated and cause intermittent failures later on, so it's not straight forward to avoid patches introducing regressions, nor to identify which patch has caused it so that we can roll it back.

It's self-speaking that it's hard to immediately evaluate if a patch
is going to cause intermittent / time-bound failures in the future: we
have no crystal ball, so I'm not making absurd demands. There will
always be cases in which developers will need to ask forgiveness.

But if anything in the test run fails during a review of a patch, and
this patch still gets merged because "the cause is likely unrelated",
or worse "we'll fix that problem later" that's unacceptable and needs
to be investigated further before being merged.
I always did that, and spending a lot of time on things which are not
directly related on my goals, for sake of respect of other developers
in the team and I expect no less from everyone else: when the
testsuite doesn't pass I can't make progress on my own work and need
to necessarily shift to firefighthing. Or go on holidays.
That's what necessarily needs to happen when a bad patch slips in past
our guard, even if doing so you need to reschedule other tasks:
because otherwise it's other people, and more and more people needing
to reschedule their own tasks. Worse yet, the other people who will
need to look at it will not have any of the context to understand what
might be going on. So I hope we're on the same page with this, because
ultimately it's about respect for each person contributing or working
on it.

I'm also puzzled on why exactly the current situation is not
sustainable. Sure there is a lot of technical debt to pay, but you
know debts come with interests and it's hard. I've also suggested many
ways to improve our testsuite to make our life easier, like use
Byteman, mock the timers via a TimeService, get rid of TestNG to move
to something more reliable, remove unnecessary dependencies from each
module, use more mocking in areas where we don't actually have an
interest in testing JGroups..
Nothing like that was done, so you can't say in all fairness that we
tried to do better. My PR which is the first step to get rid of TestNG
from the Query modules is open since 43 days.. so don't expect me to
fix any problem quickly, it's not particularly motivating.

You can try playing with processes, and I don't disagree with idea,
but when I'm able to find a regression is is that I will do: I'll
revert all commits until the first stable point I find. I've
suggested this approach to you as well, not sure why you don't apply
it more regularly? There is no shame in reverting commits, especially
if we do it regularly.
It's stable today, so I'll make a note of this commit id, my crystal
ball is telling that it will be useful soon ;-)

Sanne


>
>>
>> On 10 July 2014 14:04, Mircea Markus <mmarkus at redhat.com> wrote:
>>> I just had a chat with Dan and we don't think the current process for the test suite works. Not hard to see why, the suite is almost never green. So we will adopt a more classic and simple approach: if a test fails a blocker JIRA is created for it and assigned to a component lead then team member who'll start working on it *immediately*. Dan will be watch dog starting today so please expect blocker JIRAs coming your way and treat them accordingly.
>>>
>>> Cheers,
>>> --
>>> Mircea Markus
>>> Infinispan lead (www.infinispan.org)
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev