I'll skip straight to the conclusions in the paper:

> • Keep regression tests around for up to a year — but most of
those will be system-level tests rather than unit tests.

I would say most of our tests qualify as regression tests, and they are system-level (they start a full cache manager, with a JGroups channel and everything).
But how would we know that they haven't failed in a year? I mean we can find which tests never failed in CI, but we can't find which tests never failed on any dev's machine (thus helping that dev find a problem in his code).

> • Keep unit tests that test key algorithms for which there is a
broad, formal, independent oracle of correctness, and for
which there is ascribable business value.
> • Except for the preceding case, if X has business value and you
can text X with either a system test or a unit test, use a system
test — context is everything.

We have about 100 tests in the "unit" group, some of which should really be in the "functional" group (e.g. BaseStoreTest).
It shouldn't take long to walk over those and delete some that may be tested better by system tests, but it won't have a visible effect on the test suite run time either.

> • Design a test with more care than you design the code.

I admit that when reviewing PRs I pay less attention to the tests than to the production code. I will try to change that :)

> • Turn most unit tests into assertions.

I imagine some kinds of tests can indeed be replaced with assertions in production code, but again we don't have so many unit tests.

> • Throw away tests that haven’t failed in a year.
> • Testing can’t replace good development: a high test failure
rate suggests you should shorten development intervals,
perhaps radically, and make sure your architecture and design
regimens have teeth
> • If you find that individual functions being tested are trivial,
double-check the way you incentivize developers’
performance. Rewarding coverage or other meaningless
metrics can lead to rapid architecture decay.
> • Be humble about what tests can achieve. Tests don’t improve
quality: developers do.




On Tue, Jun 10, 2014 at 6:23 PM, Galder Zamarreño <galder@redhat.com> wrote:
Just recently I read this: http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

Food for thought, and maybe we should consider doing for next major. Maybe if we’re moving to JUnit we could look at this in detail too?

Cheers,

On 09 Jun 2014, at 23:06, Sanne Grinovero <sanne@infinispan.org> wrote:

> On 9 June 2014 14:42, Dan Berindei <dan.berindei@gmail.com> wrote:
>>
>>
>> On Mon, Jun 9, 2014 at 12:43 PM, Sanne Grinovero <sanne@infinispan.org>
>> wrote:
>>>
>>> Hi Dan!
>>>
>>> The reason is that I'm making substantial API changes in the Query
>>> module, and I've lost count on how many other modules and integration
>>> tests depend on it: I need to run all the testsuite to evaluate where
>>> I'm heading.. I don't need it just as last touches but continually, to
>>> be able to make good choices while work is in progress.
>>
>>
>> Wouldn't "mvn install -DskipTests" be enough for testing dependencies?
>> You could then use "mvn surefire:test -pl query,lucene-directory" to run
>> just the tests you're interested in.
>> I have a script that does just that - even for core, I don't want to think
>> about whether I changed something in commons or not, but I almost never want
>> to run the commons tests :)
>
> I need to run all modules, and AFAIK there is no way to run the tests
> on all modules except one (unless you list all of them explicitly..
> good luck with that).
>
> What I did to go ahead in this ball of mud is to patch the core
> pom.xml to add "skipTests" option to the surefire configuration, at
> least I could test my stuff.
>
>
>>> Not only I'm changing API but also substantial changes in the
>>> dependency tree.. without a working testsuite I can't make progress.
>>> I'm working around it by deleting core in a temporary commit... :-/
>>> (and even so the suite takes more than an hour ??!)
>>>
>>
>> 1 hour just for the core, or for everything? I used to get about 1h for
>> everything, if just the core takes that long to fail it's definitely a
>> problem. There was also a bit of a slowdown after the JGroups 3.5.0.Beta7
>> upgrade (ISPN-4355), but that should be fixed now.
>
> It's 1h and 20 minutes now to test all modules. This took 12 minutes a
> couple of months ago. My opinion is that even 12 minutes is too slow,
> I would consider it acceptable if we where running some soak tests but
> as far as I know we're just load testing the testing framework.
>
> Sanne
>
>>
>> I was also thinking of moving the failsafe plugin to the "extras" profile,
>> so that we can avoid running the server integration tests in dev builds. But
>> disabling the extras profile also disables the bundling for OSGi, so perhaps
>> a separate profile would be better.
>>
>> Dan
>>
>>
>>>
>>> I'll test your PRs ASAP, thanks a lot.
>>>
>>> Cheers,
>>> Sanne
>>>
>>>
>>> On 9 June 2014 10:19, Dan Berindei <dan.berindei@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On Fri, Jun 6, 2014 at 4:31 PM, Sanne Grinovero <sanne@infinispan.org>
>>>> wrote:
>>>>>
>>>>> I'm having several failures, these are blocking our progress on Query.
>>>>>
>>>>> Should I disable them all?
>>>>
>>>>
>>>> You could disable them, but I'm not quite sure how that would help the
>>>> query
>>>> module... surely you don't need to run the core tests every time you
>>>> modify
>>>> something in query?
>>>>
>>>>>
>>>>>
>>>>> Sample output of three different runs:
>>>>>
>>>>> Failed tests:
>>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>
>>>>
>>>> I couldn't reproduce this failure on my machine, but I modified
>>>> ThreadLocalLeakTest to ignore that particular thread-local:
>>>> https://github.com/infinispan/infinispan/pull/2614
>>>> You're the only one that reported seeing it, so please test the PR on
>>>> your
>>>> machine and integrate it.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>>>>
>>>>
>>>> I've created https://issues.jboss.org/browse/ISPN-4368 for Will to look
>>>> into, I'm not sure we need the "placeholder" key that is causing the
>>>> random
>>>> failures.
>>>>
>>>>>
>>>>>
>>>>> Failed tests:
>>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>>
>>>>>
>>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>>
>>>>>
>>>>> L1StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPut:84->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>>>>> » Runtime
>>>>
>>>>
>>>> The L1StateTransferOverwriteTest failure seems to have the same cause as
>>>> StateTransferOverwriteTest failure.
>>>>
>>>>>
>>>>> Tests run: 5430, Failures: 3, Errors: 0, Skipped: 0
>>>>>
>>>>> Failed tests:
>>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
>>>>> locals st...
>>>>>
>>>>>
>>>>> CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered:611->testIterationBeganAndSegmentNotComplete:510
>>>>> expected [11] but found [6]
>>>>
>>>>
>>>> I've seen this a couple times on my machine as well, I've created
>>>> https://issues.jboss.org/browse/ISPN-4370
>>>>
>>>>>
>>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>>>>
>>>>
>>>>
>>>> I'd let Will investigate ISPN-4368 and ISPN-4370 for a bit before
>>>> disabling
>>>> those tests - he may be able to issue a PR for them quite quickly.
>>>>
>>>> Still, just because I don't like disabling tests it doesn't mean I don't
>>>> appreciate your stability reports - keep 'em coming!
>>>>
>>>>
>>>> Cheers
>>>> Dan
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev@lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev@lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Galder Zamarreño
galder@redhat.com
twitter.com/galderz


_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev