[infinispan-dev] Uber jars testing

Sanne Grinovero sanne at infinispan.org
Fri Sep 11 10:28:57 EDT 2015 

+1 Galder for your abstraction, we might even need a DSL.

An additional benefit would be that all "API functional tests" could
be tested also in conditions such as running in a in a race with
topology changes, instrumenting the timing of parallel code and
network operations.
As a DSL and using some Byteman we could automatically have it insert
network issues or timing issues at specific critical points; such
points follow a general pattern so this can be generalized w/o having
to code tests for specific critical paths in each functional test
independently.
A simple example would be to kill a node after a command was sent to
it, but before it replies: all the tests should be able to survive
that.



On 11 September 2015 at 15:19, Galder Zamarreno <galder at redhat.com> wrote:
> ----- Original Message -----
>> Any plans for tests that are just slightly different for different
>> configurations? With inheritance, it's simple - you just override the
>> method. If you just run that test on a huge matrix of configurations,
>> you end up with having a method with a very complicated switch for
>> certain configurations.
>
> ^ I see what you are getting at here. Normally such differences sometimes happen and can be divided into two: operations executed and assertions. Sometimes the operations executed are slightly different, and sometimes the operations are the same, but assertions slightly different.
>
> I don't have specific ideas about how to solve this but my gut feeling is something like this:
>
> If we can write tests as objects/types, where we define the operations and the assertions, then all the tests (testXXX methods) have to do is run this N objects against M configurations. With that in mind, running slightly different tests would be done extending or composing the test object/types, independent of the test classes themselves. To run these slight variations, we'd define a test class that runs the variations with M configurations.
>
> Note that I've not prototyped any of that and there are probably better ways to do this.
>
>>
>> I am not asking sarcastically, but I've run into similar issue when
>> implementing similar thing in 2LC testsuite.
>>
>> Radim
>>
>> On 09/09/2015 03:22 PM, Galder Zamarreno wrote:
>> > I agree pretty much with everything below:
>> >
>> > * We overuse test overriding to run the same test with different
>> > configuration. I did that same mistake with the functional map API stuff
>> > :(
>> >
>> > * I'm in favour of testsuite restructuring, but I think we really need to
>> > start from scratch in a separate testsuite maven project, since we can
>> > then add all functional test for all (not only core...etc, but also
>> > compatibility tests...etc), and leave its project to test implementation
>> > details? Adding this separation would open up the path to create a testkit
>> > (as I explained last year in Berlin)
>> >
>> > * I'm also in favour in defining the test once and running it with
>> > different configuration options automatically.
>> >
>> > * I'm in favour too of randomising (need to check that link) but also we
>> > need some quickcheck style tests [1], e.g. a test that verifies that
>> > put(K, V) works not matter the type of object passed in.
>> >
>> > Cheers,
>> >
>> > [1]
>> > https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing
>> > --
>> > Galder Zamarreño
>> > Infinispan, Red Hat
>> >
>> > ----- Original Message -----
>> >> Interesting subject. We also have many tests which (ab)use inheritance
>> >> to re-test the same API semantics in slightly different
>> >> configurations, like embedded/DIST and embedded/REPL, sometimes
>> >> becoming an @Override mess.
>> >> It would be far more useful to restructure the testsuite to have such
>> >> tests in a single class (no inheritance) and declare - maybe
>> >> annotations? - which permutations of configuration parameters should
>> >> be valid.
>> >>
>> >> Among those configuration permutations one would not have "just"
>> >> different replication models, but also things like
>> >>   - using the same API remotely (Hot Rod)
>> >>   - using the same feature but within a WildFly embedded module
>> >>   - using the uber jars vs small jars
>> >>   - uber jars & remote..
>> >>   - remote & embedded modules..
>> >>   - remote, uber jars, in OSGi..
>> >>
>> >> And finally combine with other options:
>> >>   - A Query test using: remote client, using uber jars, in OSGi, but
>> >> switching JTA implementation, using a new experimental JGroups stack!
>> >>
>> >> For example many Core API and Query tests are copy/pasted into other
>> >> modules as "integration tests", etc.. but we really should just run
>> >> the same one in a different environment.
>> >>
>> >> This would keep our code better maintainable, but also allow some neat
>> >> tricks like specify that some configurations should definitely be
>> >> tested in some test group (like Galder suggests, one could flag one of
>> >> these for "smoke tests", one for "nightly tests"), but you could also
>> >> want to flag some configuration settings as a "should work, low
>> >> priority for testing".
>> >> A smart testsuite could then use a randomizer to generate permutations
>> >> of configuration options for those low priority tests which are not
>> >> essential; there are great examples of such testsuites in the Haskell
>> >> world, and also Lucene and ElasticSearch do it.
>> >> A single random seed is used for the whole run, and it's printed
>> >> clearly at the start; a single seed will deterministically define all
>> >> parameters of the testsuite, so you can reproduce it all by setting a
>> >> specific seed when needing to debug a failure.
>> >>
>> >> http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html
>> >>
>> >> Thanks,
>> >> Sanne
>> >>
>> >> On 3 September 2015 at 11:34, Galder Zamarreno <galder at redhat.com> wrote:
>> >>> Another interesting improvement here would be if you could run all these
>> >>> smoke tests with an alternative implementation of AdvancedCache, e.g. one
>> >>> based with functional API.
>> >>>
>> >>> Cheers,
>> >>> --
>> >>> Galder Zamarreño
>> >>> Infinispan, Red Hat
>> >>>
>> >>> ----- Original Message -----
>> >>>> Good post Jiri, this got me thinking :)
>> >>>>
>> >>>> Running the entire testsuite again with uber jars would add a lot of
>> >>>> time
>> >>>> to
>> >>>> the build time.
>> >>>>
>> >>>> Maybe we should have a set of tests that must be executed for sure, e.g.
>> >>>> like
>> >>>> Wildfly's smoke tests [1]. We have "functional" group but right now it
>> >>>> covers pretty much all tests.
>> >>>>
>> >>>> Such tests should live in a separate testsuite, so that we could add the
>> >>>> essential tests for *all* components. In a way, we've already done some
>> >>>> of
>> >>>> this in integrationtests/ but it's not really well structured for this
>> >>>> aim.
>> >>>>
>> >>>> Also, if we would go down this path, something we should take advantage
>> >>>> of
>> >>>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests
>> >>>> in
>> >>>> [2], where he used suites to make it faster to run things, by starting a
>> >>>> cache manager for distributed caches, running all distributed
>> >>>> tests...etc.
>> >>>> In a way, I think we can already do this with Arquillian Infinispan
>> >>>> integration, so Arquillian would probably well suited for such smoke
>> >>>> testsuite.
>> >>>>
>> >>>> Thoughts?
>> >>>>
>> >>>> Cheers,
>> >>>>
>> >>>> [1] https://github.com/wildfly/wildfly#running-the-testsuite
>> >>>> [2]
>> >>>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala/org/infinispan/spark
>> >>>> --
>> >>>> Galder Zamarreño
>> >>>> Infinispan, Red Hat
>> >>>>
>> >>>> ----- Original Message -----
>> >>>>> Hi Jiri, comments inline.
>> >>>>>
>> >>>>> On 2.9.2015 10:40, Jiri Holusa wrote:
>> >>>>>> Hi all,
>> >>>>>>
>> >>>>>> we've been thinking for a while, how to test ISPN uber jars. The
>> >>>>>> current
>> >>>>>> status is that we actually don't have many tests in the testsuite,
>> >>>>>> there
>> >>>>>> are few tests in integrationtests/all-embedded-* modules that are
>> >>>>>> basically copies of the actual tests in corresponding modules. We
>> >>>>>> think
>> >>>>>> that this test coverage is not enough and more importantly, they are
>> >>>>>> duplicates.
>> >>>>>>
>> >>>>>> The questions are now following:
>> >>>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite?
>> >>>>>> Only
>> >>>>>> integrationtests module?
>> >>>>> The goal is to run the whole test suite because, as you said, we don't
>> >>>>> have enough tests in integrationtests/*   And we can't duplicate all
>> >>>>> test classes from individual modules here.
>> >>>>>
>> >>>>>> * how would it run? Create Maven different profiles for "classic" jars
>> >>>>>> and
>> >>>>>> uber jars? Or try to use some Maven exclusion magic if even possible?
>> >>>>>>
>> >>>>>> Some time ago, we had discussion about this with Sebastian, who
>> >>>>>> suggested
>> >>>>>> that running only integrationtests module would be sufficient, because
>> >>>>>> uber-jars are really about packaging, not the functionality itself.
>> >>>>>> But I
>> >>>>>> don't know if the tests coverage is sufficient in that level, I would
>> >>>>>> be
>> >>>>>> much more confident if we could run the whole ISPN testsuite against
>> >>>>>> uber-jars.
>> >>>>> Right. Uber-jars are about packaging but you don't know that the
>> >>>>> packiging is right until you try all the features and see that
>> >>>>> everything works. There might be some classes missing (just for some
>> >>>>> particular features), same classes in different packages, the
>> >>>>> Manifest.mf might be corrupted and then something won't work in OSGi.
>> >>>>>
>> >>>>>
>> >>>>> I'd prefer a separate Maven profile. IMO, exclusions are too
>> >>>>> error-prone.
>> >>>>>
>> >>>>>
>> >>>>> Martin
>> >>>>>> I'm opening this for wider discussion as we should agree on the way
>> >>>>>> how
>> >>>>>> to
>> >>>>>> do it, so we could do it right :)
>> >>>>>>
>> >>>>>> Cheers,
>> >>>>>> Jiri
>> >>>>>>
>> >>>>>>
>> >>>>> _______________________________________________
>> >>>>> infinispan-dev mailing list
>> >>>>> infinispan-dev at lists.jboss.org
>> >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>>>>
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev at lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> --
>> Radim Vansa <rvansa at redhat.com>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list