Just recently I read this: http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf
Food for thought, and maybe we should consider doing for next major. Maybe if we’re moving to JUnit we could look at this in detail too?
Cheers,
--
On 09 Jun 2014, at 23:06, Sanne Grinovero <sanne@infinispan.org> wrote:
> On 9 June 2014 14:42, Dan Berindei <dan.berindei@gmail.com> wrote:
>>
>>
>> On Mon, Jun 9, 2014 at 12:43 PM, Sanne Grinovero <sanne@infinispan.org>
>> wrote:
>>>
>>> Hi Dan!
>>>
>>> The reason is that I'm making substantial API changes in the Query
>>> module, and I've lost count on how many other modules and integration
>>> tests depend on it: I need to run all the testsuite to evaluate where
>>> I'm heading.. I don't need it just as last touches but continually, to
>>> be able to make good choices while work is in progress.
>>
>>
>> Wouldn't "mvn install -DskipTests" be enough for testing dependencies?
>> You could then use "mvn surefire:test -pl query,lucene-directory" to run
>> just the tests you're interested in.
>> I have a script that does just that - even for core, I don't want to think
>> about whether I changed something in commons or not, but I almost never want
>> to run the commons tests :)
>
> I need to run all modules, and AFAIK there is no way to run the tests
> on all modules except one (unless you list all of them explicitly..
> good luck with that).
>
> What I did to go ahead in this ball of mud is to patch the core
> pom.xml to add "skipTests" option to the surefire configuration, at
> least I could test my stuff.
>
>
>>> Not only I'm changing API but also substantial changes in the
>>> dependency tree.. without a working testsuite I can't make progress.
>>> I'm working around it by deleting core in a temporary commit... :-/
>>> (and even so the suite takes more than an hour ??!)
>>>
>>
>> 1 hour just for the core, or for everything? I used to get about 1h for
>> everything, if just the core takes that long to fail it's definitely a
>> problem. There was also a bit of a slowdown after the JGroups 3.5.0.Beta7
>> upgrade (ISPN-4355), but that should be fixed now.
>
> It's 1h and 20 minutes now to test all modules. This took 12 minutes a
> couple of months ago. My opinion is that even 12 minutes is too slow,
> I would consider it acceptable if we where running some soak tests but
> as far as I know we're just load testing the testing framework.
>
> Sanne
>
>>
>> I was also thinking of moving the failsafe plugin to the "extras" profile,
>> so that we can avoid running the server integration tests in dev builds. But
>> disabling the extras profile also disables the bundling for OSGi, so perhaps
>> a separate profile would be better.
>>
>> Dan
>>
>>
>>>
>>> I'll test your PRs ASAP, thanks a lot.
>>>
>>> Cheers,
>>> Sanne
>>>
>>>
>>> On 9 June 2014 10:19, Dan Berindei <dan.berindei@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On Fri, Jun 6, 2014 at 4:31 PM, Sanne Grinovero <sanne@infinispan.org>
>>>> wrote:
>>>>>
>>>>> I'm having several failures, these are blocking our progress on Query.
>>>>>
>>>>> Should I disable them all?
>>>>
>>>>
>>>> You could disable them, but I'm not quite sure how that would help the
>>>> query
>>>> module... surely you don't need to run the core tests every time you
>>>> modify
>>>> something in query?
>>>>
>>>>>
>>>>>
>>>>> Sample output of three different runs:
>>>>>
>>>>> Failed tests:
>>>>> ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>
>>>>
>>>> I couldn't reproduce this failure on my machine, but I modified
>>>> ThreadLocalLeakTest to ignore that particular thread-local:
>>>> https://github.com/infinispan/infinispan/pull/2614
>>>> You're the only one that reported seeing it, so please test the PR on
>>>> your
>>>> machine and integrate it.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>>>>
>>>>
>>>> I've created https://issues.jboss.org/browse/ISPN-4368 for Will to look
>>>> into, I'm not sure we need the "placeholder" key that is causing the
>>>> random
>>>> failures.
>>>>
>>>>>
>>>>>
>>>>> Failed tests:
>>>>> ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>>
>>>>>
>>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>>
>>>>>
>>>>> L1StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPut:84->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>
>>>>
>>>> The L1StateTransferOverwriteTest failure seems to have the same cause as
>>>> StateTransferOverwriteTest failure.
>>>>
>>>>>
>>>>> Tests run: 5430, Failures: 3, Errors: 0, Skipped: 0
>>>>>
>>>>> Failed tests:
>>>>> ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>>
>>>>>
>>>>> CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered:611->testIterationBeganAndSegmentNotComplete:510
>>>>> expected [11] but found [6]
>>>>
>>>>
>>>> I've seen this a couple times on my machine as well, I've created
>>>> https://issues.jboss.org/browse/ISPN-4370
>>>>
>>>>>
>>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>>>>
>>>>
>>>>
>>>> I'd let Will investigate ISPN-4368 and ISPN-4370 for a bit before
>>>> disabling
>>>> those tests - he may be able to issue a PR for them quite quickly.
>>>>
>>>> Still, just because I don't like disabling tests it doesn't mean I don't
>>>> appreciate your stability reports - keep 'em coming!
>>>>
>>>>
>>>> Cheers
>>>> Dan
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev@lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Galder Zamarreño
galder@redhat.com
twitter.com/galderz
_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev