[infinispan-dev] Test failures

Wed Jun 11 03:03:51 EDT 2014

I'll skip straight to the conclusions in the paper:

> • Keep regression tests around for up to a year — but most of
those will be system-level tests rather than unit tests.

I would say most of our tests qualify as regression tests, and they are
system-level (they start a full cache manager, with a JGroups channel and
everything).
But how would we know that they haven't failed in a year? I mean we can
find which tests never failed in CI, but we can't find which tests never
failed on any dev's machine (thus helping that dev find a problem in his
code).

> • Keep unit tests that test key algorithms for which there is a
broad, formal, independent oracle of correctness, and for
which there is ascribable business value.
> • Except for the preceding case, if X has business value and you
can text X with either a system test or a unit test, use a system
test — context is everything.

We have about 100 tests in the "unit" group, some of which should really be
in the "functional" group (e.g. BaseStoreTest).
It shouldn't take long to walk over those and delete some that may be
tested better by system tests, but it won't have a visible effect on the
test suite run time either.

> • Design a test with more care than you design the code.

I admit that when reviewing PRs I pay less attention to the tests than to
the production code. I will try to change that :)

> • Turn most unit tests into assertions.

I imagine some kinds of tests can indeed be replaced with assertions in
production code, but again we don't have so many unit tests.

> • Throw away tests that haven’t failed in a year.
> • Testing can’t replace good development: a high test failure
rate suggests you should shorten development intervals,
perhaps radically, and make sure your architecture and design
regimens have teeth
> • If you find that individual functions being tested are trivial,
double-check the way you incentivize developers’
performance. Rewarding coverage or other meaningless
metrics can lead to rapid architecture decay.
> • Be humble about what tests can achieve. Tests don’t improve
quality: developers do.

On Tue, Jun 10, 2014 at 6:23 PM, Galder Zamarreño <galder at redhat.com> wrote:

> Just recently I read this:
> http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf
>
> Food for thought, and maybe we should consider doing for next major. Maybe
> if we’re moving to JUnit we could look at this in detail too?
>
> Cheers,
>
> On 09 Jun 2014, at 23:06, Sanne Grinovero <sanne at infinispan.org> wrote:
>
> > On 9 June 2014 14:42, Dan Berindei <dan.berindei at gmail.com> wrote:
> >>
> >>
> >> On Mon, Jun 9, 2014 at 12:43 PM, Sanne Grinovero <sanne at infinispan.org>
> >> wrote:
> >>>
> >>> Hi Dan!
> >>>
> >>> The reason is that I'm making substantial API changes in the Query
> >>> module, and I've lost count on how many other modules and integration
> >>> tests depend on it: I need to run all the testsuite to evaluate where
> >>> I'm heading.. I don't need it just as last touches but continually, to
> >>> be able to make good choices while work is in progress.
> >>
> >>
> >> Wouldn't "mvn install -DskipTests" be enough for testing dependencies?
> >> You could then use "mvn surefire:test -pl query,lucene-directory" to run
> >> just the tests you're interested in.
> >> I have a script that does just that - even for core, I don't want to
> think
> >> about whether I changed something in commons or not, but I almost never
> want
> >> to run the commons tests :)
> >
> > I need to run all modules, and AFAIK there is no way to run the tests
> > on all modules except one (unless you list all of them explicitly..
> > good luck with that).
> >
> > What I did to go ahead in this ball of mud is to patch the core
> > pom.xml to add "skipTests" option to the surefire configuration, at
> > least I could test my stuff.
> >
> >
> >>> Not only I'm changing API but also substantial changes in the
> >>> dependency tree.. without a working testsuite I can't make progress.
> >>> I'm working around it by deleting core in a temporary commit... :-/
> >>> (and even so the suite takes more than an hour ??!)
> >>>
> >>
> >> 1 hour just for the core, or for everything? I used to get about 1h for
> >> everything, if just the core takes that long to fail it's definitely a
> >> problem. There was also a bit of a slowdown after the JGroups
> 3.5.0.Beta7
> >> upgrade (ISPN-4355), but that should be fixed now.
> >
> > It's 1h and 20 minutes now to test all modules. This took 12 minutes a
> > couple of months ago. My opinion is that even 12 minutes is too slow,
> > I would consider it acceptable if we where running some soak tests but
> > as far as I know we're just load testing the testing framework.
> >
> > Sanne
> >
> >>
> >> I was also thinking of moving the failsafe plugin to the "extras"
> profile,
> >> so that we can avoid running the server integration tests in dev
> builds. But
> >> disabling the extras profile also disables the bundling for OSGi, so
> perhaps
> >> a separate profile would be better.
> >>
> >> Dan
> >>
> >>
> >>>
> >>> I'll test your PRs ASAP, thanks a lot.
> >>>
> >>> Cheers,
> >>> Sanne
> >>>
> >>>
> >>> On 9 June 2014 10:19, Dan Berindei <dan.berindei at gmail.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Jun 6, 2014 at 4:31 PM, Sanne Grinovero <sanne at infinispan.org
> >
> >>>> wrote:
> >>>>>
> >>>>> I'm having several failures, these are blocking our progress on
> Query.
> >>>>>
> >>>>> Should I disable them all?
> >>>>
> >>>>
> >>>> You could disable them, but I'm not quite sure how that would help the
> >>>> query
> >>>> module... surely you don't need to run the core tests every time you
> >>>> modify
> >>>> something in query?
> >>>>
> >>>>>
> >>>>>
> >>>>> Sample output of three different runs:
> >>>>>
> >>>>> Failed tests:
> >>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
> >>>>> locals st...
> >>>>
> >>>>
> >>>> I couldn't reproduce this failure on my machine, but I modified
> >>>> ThreadLocalLeakTest to ignore that particular thread-local:
> >>>> https://github.com/infinispan/infinispan/pull/2614
> >>>> You're the only one that reported seeing it, so please test the PR on
> >>>> your
> >>>> machine and integrate it.
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
> >>>>> » Runtime
> >>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
> >>>>
> >>>>
> >>>> I've created https://issues.jboss.org/browse/ISPN-4368 for Will to
> look
> >>>> into, I'm not sure we need the "placeholder" key that is causing the
> >>>> random
> >>>> failures.
> >>>>
> >>>>>
> >>>>>
> >>>>> Failed tests:
> >>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
> >>>>> locals st...
> >>>>>
> >>>>>
> >>>>>
> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
> >>>>> » Runtime
> >>>>>
> >>>>>
> >>>>>
> L1StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPut:84->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
> >>>>> » Runtime
> >>>>
> >>>>
> >>>> The L1StateTransferOverwriteTest failure seems to have the same cause
> as
> >>>> StateTransferOverwriteTest failure.
> >>>>
> >>>>>
> >>>>> Tests run: 5430, Failures: 3, Errors: 0, Skipped: 0
> >>>>>
> >>>>> Failed tests:
> >>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState Thread
> >>>>> locals st...
> >>>>>
> >>>>>
> >>>>>
> CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered:611->testIterationBeganAndSegmentNotComplete:510
> >>>>> expected [11] but found [6]
> >>>>
> >>>>
> >>>> I've seen this a couple times on my machine as well, I've created
> >>>> https://issues.jboss.org/browse/ISPN-4370
> >>>>
> >>>>>
> >>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
> >>>>
> >>>>
> >>>>
> >>>> I'd let Will investigate ISPN-4368 and ISPN-4370 for a bit before
> >>>> disabling
> >>>> those tests - he may be able to issue a PR for them quite quickly.
> >>>>
> >>>> Still, just because I don't like disabling tests it doesn't mean I
> don't
> >>>> appreciate your stability reports - keep 'em coming!
> >>>>
> >>>>
> >>>> Cheers
> >>>> Dan
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> infinispan-dev mailing list
> >>>> infinispan-dev at lists.jboss.org
> >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Galder Zamarreño
> galder at redhat.com
> twitter.com/galderz
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20140611/7b72339e/attachment.html 