When I handle pull reqs, I sometimes see random testsuite failures
which often are not related to patch itself.
What I do then is send an email to an author, or expert in the test with TRACE log and
failure information, and ask to look into it.
They might not look into it immediately (for several), but eventually they should get
around to doing it, and it's a know issue then.
In this cases, we should maybe disable the tests and mark the person's name, but I
don't see an easy way of checking globally which tests are disabled.
I have a script for this.
Again, this tries to find a balance between improving our testsuite and moving forward
with the N other priorities we have.
Maybe each of these should be raised as JIRAs, with reasonably high priority, and assigned
to the test author?
On May 29, 2012, at 11:24 AM, Galder Zamarreño wrote:
> BuildHive might help somewhat (when it fully works...) but the real problem
testsuites will continue to be there:
>
> 1. Some tests fail randomly, due to concurrency issues or any kind.
> 2. Some tests fail due to differences in environments (CI vs your own machine)
>
> The problem with this tests will continue as long as people are individually tasked
to tackle these failures. I'm yet to see people proactively fixing this stuff, without
any external intervention. As long as this is like this, we'll always be reacting to
things.
>
> Either *we all* behave like Sanne suggests, or we carry on like we've done so
far.
>
> And then thing is that there's a balance to be found between chasing down this
annoying failures, and moving the project forward with new functionality, helping paying
customers, helping community, fixing bugs that affect functionality, evangelise….etc.
>
> On May 28, 2012, at 7:18 PM, Manik Surtani wrote:
>
>> We're actually in the process of setting up BuildHive. Galder's on it.
:)
>>
>> On 28 May 2012, at 17:20, Adrian Cole wrote:
>>
>>> FWIW, might be a good idea trying buildhive a bit, then deciding. It is
working pretty well for jenkins-ci projects, and so much easier than fetch, cherry-pick,
test push loop.
>>>
>>> In jclouds, we are setting this up as community members are starting to be
more brave (ex refactor things that other PRs can trip), and I've needed to put $1
into the jar a few times merging ;)
>>>
>>> Seems a pragmatic 'wait and see' to try BuildHive a while, but of
course, you know better than me about what's the right choice here.
>>>
>>> If you are having any struggle setting up that, let me or Andrew Phillips
know, as we had a change into BuilHive recently to deal with our massive build :)
>>>
>>> Have fun!
>>> -A
>>>
>>> On May 28, 2012 1:41 AM, "Manik Surtani" <manik(a)jboss.org>
wrote:
>>> I don't think everyone has to handle tens of PRs a day. It's more
like one per person per day, which IMO isn't unreasonable as long as everyone does
their fair share.
>>>
>>> On 27 May 2012, at 14:51, Bela Ban wrote:
>>>
>>>> +1000. I completely agree that if someone has to handle tens of pull
>>>> requests per day, he will *not* seriously look into the request, test it
>>>> etc. So IMO this is a farce, and we might as well go back to trusting
>>>> people, rather than wasting their time...
>>>>
>>>>
>>>>
>>>> On 5/25/12 1:47 PM, Sanne Grinovero wrote:
>>>>> guys, please don't take me as the one who is again complaining
about
>>>>> failing tests; I'm having doubts about the development process
and the
>>>>> amount of time this is wasting on all of us.
>>>>>
>>>>> We're all humans and do mistakes, still it happens so extremely
often
>>>>> that this is getting systemic, and discipline could definitely be
>>>>> improved: people regularly send pull requests with failing tests or
>>>>> broken code, and very regularly this is just merged in master.
>>>>>
>>>>> I did it myself a couple of days ago: didn't notice a failure,
all
>>>>> looked good, sent a pull, it was merged with no complaints. Three
days
>>>>> later, I resume my work and am appalled to see that it was broken.
Now
>>>>> fixing it, but I'll have to send another pull and wait for it -
which
>>>>> feels very pointless, as I'm pretty sure nobody is checking
anyway.
>>>>>
>>>>> It looks like as the pull request procedure is having this effect:
>>>>>
>>>>> # patch writer is not as carefull as he used to be: "someone
else
>>>>> will check if it's fine or not. I have no time to run the tests
>>>>> again..".
>>>>>
>>>>> # reviewer has as quick look. "Looks good - in fact I don't
care
>>>>> much, it's not my code and need to return to my own issues..
worst
>>>>> case someone else will fix it blaming the original author"
>>>>>
>>>>> And then again some incomplete test makes it to master, or a patch
>>>>> which doesn't even compile is integrated.
>>>>>
>>>>> This pull request process is being a big failure. Shall we stop
>>>>> wasting time on it and just push on master?
>>>>>
>>>>> Which doesn't mean I'm suggesting "let's make it
worse" | "unleash
>>>>> hell": we should all take responsibility on any change very
seriously.
>>>>>
>>>>> Again, I'm not enjoying the role of "whom who complains on
the
>>>>> testsuite again". Just stating a fact, and trying to propose
something
>>>>> to make it work better. We have great individuals on this team, but
we
>>>>> need to admit that team work isn't working and we should deal
with it
>>>>> at it's best; denying it won't help.
>>>>>
>>>>> Cheers,
>>>>> Sanne
>>>>
>>>>
>>>> --
>>>> Bela Ban, JGroups lead (
http://www.jgroups.org)
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev(a)lists.jboss.org
>>>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> --
>>> Manik Surtani
>>> manik(a)jboss.org
>>>
twitter.com/maniksurtani
>>>
>>> Lead, Infinispan
>>>
http://www.infinispan.org
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Manik Surtani
>> manik(a)jboss.org
>>
twitter.com/maniksurtani
>>
>> Lead, Infinispan
>>
http://www.infinispan.org
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev