[infinispan-dev] Test failures

Tue Jun 10 11:23:01 EDT 2014

Just recently I read this: http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

Food for thought, and maybe we should consider doing for next major. Maybe if we’re moving to JUnit we could look at this in detail too?

Cheers,

On 09 Jun 2014, at 23:06, Sanne Grinovero <sanne at infinispan.org> wrote:

> On 9 June 2014 14:42, Dan Berindei <dan.berindei at gmail.com> wrote:
>> 
>> 
>> On Mon, Jun 9, 2014 at 12:43 PM, Sanne Grinovero <sanne at infinispan.org>
>> wrote:
>>> 
>>> Hi Dan!
>>> 
>>> The reason is that I'm making substantial API changes in the Query
>>> module, and I've lost count on how many other modules and integration
>>> tests depend on it: I need to run all the testsuite to evaluate where
>>> I'm heading.. I don't need it just as last touches but continually, to
>>> be able to make good choices while work is in progress.
>> 
>> 
>> Wouldn't "mvn install -DskipTests" be enough for testing dependencies?
>> You could then use "mvn surefire:test -pl query,lucene-directory" to run
>> just the tests you're interested in.
>> I have a script that does just that - even for core, I don't want to think
>> about whether I changed something in commons or not, but I almost never want
>> to run the commons tests :)
> 
> I need to run all modules, and AFAIK there is no way to run the tests
> on all modules except one (unless you list all of them explicitly..
> good luck with that).
> 
> What I did to go ahead in this ball of mud is to patch the core
> pom.xml to add "skipTests" option to the surefire configuration, at
> least I could test my stuff.
> 
> 
>>> Not only I'm changing API but also substantial changes in the
>>> dependency tree.. without a working testsuite I can't make progress.
>>> I'm working around it by deleting core in a temporary commit... :-/
>>> (and even so the suite takes more than an hour ??!)
>>> 
>> 
>> 1 hour just for the core, or for everything? I used to get about 1h for
>> everything, if just the core takes that long to fail it's definitely a
>> problem. There was also a bit of a slowdown after the JGroups 3.5.0.Beta7
>> upgrade (ISPN-4355), but that should be fixed now.
> 
> It's 1h and 20 minutes now to test all modules. This took 12 minutes a
> couple of months ago. My opinion is that even 12 minutes is too slow,
> I would consider it acceptable if we where running some soak tests but
> as far as I know we're just load testing the testing framework.
> 
> Sanne
> 
>> 
>> I was also thinking of moving the failsafe plugin to the "extras" profile,
>> so that we can avoid running the server integration tests in dev builds. But
>> disabling the extras profile also disables the bundling for OSGi, so perhaps
>> a separate profile would be better.
>> 
>> Dan
>> 
>> 
>>> 
>>> I'll test your PRs ASAP, thanks a lot.
>>> 
>>> Cheers,
>>> Sanne
>>> 
>>> 
>>> On 9 June 2014 10:19, Dan Berindei <dan.berindei at gmail.com> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Fri, Jun 6, 2014 at 4:31 PM, Sanne Grinovero <sanne at infinispan.org>
>>>> wrote:
>>>>> 
>>>>> I'm having several failures, these are blocking our progress on Query.
>>>>> 
>>>>> Should I disable them all?
>>>> 
>>>> 
>>>> You could disable them, but I'm not quite sure how that would help the
>>>> query
>>>> module... surely you don't need to run the core tests every time you
>>>> modify
>>>> something in query?
>>>> 
>>>>> 
>>>>> 
>>>>> Sample output of three different runs:
>>>>> 
>>>>> Failed tests:
>>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>> 
>>>> 
>>>> I couldn't reproduce this failure on my machine, but I modified
>>>> ThreadLocalLeakTest to ignore that particular thread-local:
>>>> https://github.com/infinispan/infinispan/pull/2614
>>>> You're the only one that reported seeing it, so please test the PR on
>>>> your
>>>> machine and integrate it.
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>>>> 
>>>> 
>>>> I've created https://issues.jboss.org/browse/ISPN-4368 for Will to look
>>>> into, I'm not sure we need the "placeholder" key that is causing the
>>>> random
>>>> failures.
>>>> 
>>>>> 
>>>>> 
>>>>> Failed tests:
>>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>> 
>>>>> 
>>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>> 
>>>>> 
>>>>> L1StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPut:84->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>> 
>>>> 
>>>> The L1StateTransferOverwriteTest failure seems to have the same cause as
>>>> StateTransferOverwriteTest failure.
>>>> 
>>>>> 
>>>>> Tests run: 5430, Failures: 3, Errors: 0, Skipped: 0
>>>>> 
>>>>> Failed tests:
>>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>> 
>>>>> 
>>>>> CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered:611->testIterationBeganAndSegmentNotComplete:510
>>>>> expected [11] but found [6]
>>>> 
>>>> 
>>>> I've seen this a couple times on my machine as well, I've created
>>>> https://issues.jboss.org/browse/ISPN-4370
>>>> 
>>>>> 
>>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>>>> 
>>>> 
>>>> 
>>>> I'd let Will investigate ISPN-4368 and ISPN-4370 for a bit before
>>>> disabling
>>>> those tests - he may be able to issue a PR for them quite quickly.
>>>> 
>>>> Still, just because I don't like disabling tests it doesn't mean I don't
>>>> appreciate your stability reports - keep 'em coming!
>>>> 
>>>> 
>>>> Cheers
>>>> Dan
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
galder at redhat.com
twitter.com/galderz