[infinispan-dev] Test failures

Fri Jun 20 08:34:14 EDT 2014

I've been trying again, current state:

Failed tests:
  ConditionalOperationsConcurrentWriteSkewTest.testPutIfAbsent:62->ConditionalOperationsConcurrentTest.testOnCaches:129
java.lang.RuntimeException: java.util.concurrent.TimeoutException

Tests run: 4436, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Infinispan BOM .................................... SUCCESS [  0.237 s]
[INFO] Infinispan Common Parent .......................... SUCCESS [  2.192 s]
[INFO] Infinispan Checkstyle Rules ....................... SUCCESS [  3.591 s]
[INFO] Infinispan Commons ................................ SUCCESS [  6.209 s]
[INFO] Infinispan Core ................................... FAILURE [11:10 min]

..

Sanne

On 11 June 2014 08:03, Dan Berindei <dan.berindei at gmail.com> wrote:
> I'll skip straight to the conclusions in the paper:
>
>> • Keep regression tests around for up to a year — but most of
> those will be system-level tests rather than unit tests.
>
> I would say most of our tests qualify as regression tests, and they are
> system-level (they start a full cache manager, with a JGroups channel and
> everything).
> But how would we know that they haven't failed in a year? I mean we can find
> which tests never failed in CI, but we can't find which tests never failed
> on any dev's machine (thus helping that dev find a problem in his code).
>
>> • Keep unit tests that test key algorithms for which there is a
> broad, formal, independent oracle of correctness, and for
> which there is ascribable business value.
>> • Except for the preceding case, if X has business value and you
> can text X with either a system test or a unit test, use a system
> test — context is everything.
>
> We have about 100 tests in the "unit" group, some of which should really be
> in the "functional" group (e.g. BaseStoreTest).
> It shouldn't take long to walk over those and delete some that may be tested
> better by system tests, but it won't have a visible effect on the test suite
> run time either.
>
>> • Design a test with more care than you design the code.
>
> I admit that when reviewing PRs I pay less attention to the tests than to
> the production code. I will try to change that :)
>
>> • Turn most unit tests into assertions.
>
> I imagine some kinds of tests can indeed be replaced with assertions in
> production code, but again we don't have so many unit tests.
>
>> • Throw away tests that haven’t failed in a year.
>> • Testing can’t replace good development: a high test failure
> rate suggests you should shorten development intervals,
> perhaps radically, and make sure your architecture and design
> regimens have teeth
>> • If you find that individual functions being tested are trivial,
> double-check the way you incentivize developers’
> performance. Rewarding coverage or other meaningless
> metrics can lead to rapid architecture decay.
>> • Be humble about what tests can achieve. Tests don’t improve
> quality: developers do.
>
>
>
>
> On Tue, Jun 10, 2014 at 6:23 PM, Galder Zamarreño <galder at redhat.com> wrote:
>>
>> Just recently I read this:
>> http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf
>>
>> Food for thought, and maybe we should consider doing for next major. Maybe
>> if we’re moving to JUnit we could look at this in detail too?
>>
>> Cheers,
>>
>> On 09 Jun 2014, at 23:06, Sanne Grinovero <sanne at infinispan.org> wrote:
>>
>> > On 9 June 2014 14:42, Dan Berindei <dan.berindei at gmail.com> wrote:
>> >>
>> >>
>> >> On Mon, Jun 9, 2014 at 12:43 PM, Sanne Grinovero <sanne at infinispan.org>
>> >> wrote:
>> >>>
>> >>> Hi Dan!
>> >>>
>> >>> The reason is that I'm making substantial API changes in the Query
>> >>> module, and I've lost count on how many other modules and integration
>> >>> tests depend on it: I need to run all the testsuite to evaluate where
>> >>> I'm heading.. I don't need it just as last touches but continually, to
>> >>> be able to make good choices while work is in progress.
>> >>
>> >>
>> >> Wouldn't "mvn install -DskipTests" be enough for testing dependencies?
>> >> You could then use "mvn surefire:test -pl query,lucene-directory" to
>> >> run
>> >> just the tests you're interested in.
>> >> I have a script that does just that - even for core, I don't want to
>> >> think
>> >> about whether I changed something in commons or not, but I almost never
>> >> want
>> >> to run the commons tests :)
>> >
>> > I need to run all modules, and AFAIK there is no way to run the tests
>> > on all modules except one (unless you list all of them explicitly..
>> > good luck with that).
>> >
>> > What I did to go ahead in this ball of mud is to patch the core
>> > pom.xml to add "skipTests" option to the surefire configuration, at
>> > least I could test my stuff.
>> >
>> >
>> >>> Not only I'm changing API but also substantial changes in the
>> >>> dependency tree.. without a working testsuite I can't make progress.
>> >>> I'm working around it by deleting core in a temporary commit... :-/
>> >>> (and even so the suite takes more than an hour ??!)
>> >>>
>> >>
>> >> 1 hour just for the core, or for everything? I used to get about 1h for
>> >> everything, if just the core takes that long to fail it's definitely a
>> >> problem. There was also a bit of a slowdown after the JGroups
>> >> 3.5.0.Beta7
>> >> upgrade (ISPN-4355), but that should be fixed now.
>> >
>> > It's 1h and 20 minutes now to test all modules. This took 12 minutes a
>> > couple of months ago. My opinion is that even 12 minutes is too slow,
>> > I would consider it acceptable if we where running some soak tests but
>> > as far as I know we're just load testing the testing framework.
>> >
>> > Sanne
>> >
>> >>
>> >> I was also thinking of moving the failsafe plugin to the "extras"
>> >> profile,
>> >> so that we can avoid running the server integration tests in dev
>> >> builds. But
>> >> disabling the extras profile also disables the bundling for OSGi, so
>> >> perhaps
>> >> a separate profile would be better.
>> >>
>> >> Dan
>> >>
>> >>
>> >>>
>> >>> I'll test your PRs ASAP, thanks a lot.
>> >>>
>> >>> Cheers,
>> >>> Sanne
>> >>>
>> >>>
>> >>> On 9 June 2014 10:19, Dan Berindei <dan.berindei at gmail.com> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Jun 6, 2014 at 4:31 PM, Sanne Grinovero
>> >>>> <sanne at infinispan.org>
>> >>>> wrote:
>> >>>>>
>> >>>>> I'm having several failures, these are blocking our progress on
>> >>>>> Query.
>> >>>>>
>> >>>>> Should I disable them all?
>> >>>>
>> >>>>
>> >>>> You could disable them, but I'm not quite sure how that would help
>> >>>> the
>> >>>> query
>> >>>> module... surely you don't need to run the core tests every time you
>> >>>> modify
>> >>>> something in query?
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> Sample output of three different runs:
>> >>>>>
>> >>>>> Failed tests:
>> >>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState
>> >>>>> Thread
>> >>>>> locals st...
>> >>>>
>> >>>>
>> >>>> I couldn't reproduce this failure on my machine, but I modified
>> >>>> ThreadLocalLeakTest to ignore that particular thread-local:
>> >>>> https://github.com/infinispan/infinispan/pull/2614
>> >>>> You're the only one that reported seeing it, so please test the PR on
>> >>>> your
>> >>>> machine and integrate it.
>> >>>>
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>> >>>>> » Runtime
>> >>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>> >>>>
>> >>>>
>> >>>> I've created https://issues.jboss.org/browse/ISPN-4368 for Will to
>> >>>> look
>> >>>> into, I'm not sure we need the "placeholder" key that is causing the
>> >>>> random
>> >>>> failures.
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> Failed tests:
>> >>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState
>> >>>>> Thread
>> >>>>> locals st...
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPutIfAbsent:104->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>> >>>>> » Runtime
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> L1StateTransferOverwriteTest>BaseTxStateTransferOverwriteTest.testStateTransferInBetweenPrepareCommitWithPut:84->BaseTxStateTransferOverwriteTest.doStateTransferInBetweenPrepareCommit:265
>> >>>>> » Runtime
>> >>>>
>> >>>>
>> >>>> The L1StateTransferOverwriteTest failure seems to have the same cause
>> >>>> as
>> >>>> StateTransferOverwriteTest failure.
>> >>>>
>> >>>>>
>> >>>>> Tests run: 5430, Failures: 3, Errors: 0, Skipped: 0
>> >>>>>
>> >>>>> Failed tests:
>> >>>>>  ThreadLocalLeakTest.testCheckThreadLocalLeaks:87 IllegalState
>> >>>>> Thread
>> >>>>> locals st...
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> CacheNotifierImplInitialTransferDistTest.testCreateAfterIterationBeganAndSegmentNotCompleteValueOwnerClustered:611->testIterationBeganAndSegmentNotComplete:510
>> >>>>> expected [11] but found [6]
>> >>>>
>> >>>>
>> >>>> I've seen this a couple times on my machine as well, I've created
>> >>>> https://issues.jboss.org/browse/ISPN-4370
>> >>>>
>> >>>>>
>> >>>>> Tests run: 5430, Failures: 2, Errors: 0, Skipped: 0
>> >>>>
>> >>>>
>> >>>>
>> >>>> I'd let Will investigate ISPN-4368 and ISPN-4370 for a bit before
>> >>>> disabling
>> >>>> those tests - he may be able to issue a PR for them quite quickly.
>> >>>>
>> >>>> Still, just because I don't like disabling tests it doesn't mean I
>> >>>> don't
>> >>>> appreciate your stability reports - keep 'em coming!
>> >>>>
>> >>>>
>> >>>> Cheers
>> >>>> Dan
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> infinispan-dev mailing list
>> >>>> infinispan-dev at lists.jboss.org
>> >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>>
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev at lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Galder Zamarreño
>> galder at redhat.com
>> twitter.com/galderz
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev