[infinispan-dev] Uber jars testing

Fri Sep 11 10:19:02 EDT 2015

----- Original Message -----
> Any plans for tests that are just slightly different for different
> configurations? With inheritance, it's simple - you just override the
> method. If you just run that test on a huge matrix of configurations,
> you end up with having a method with a very complicated switch for
> certain configurations.

^ I see what you are getting at here. Normally such differences sometimes happen and can be divided into two: operations executed and assertions. Sometimes the operations executed are slightly different, and sometimes the operations are the same, but assertions slightly different. 

I don't have specific ideas about how to solve this but my gut feeling is something like this:

If we can write tests as objects/types, where we define the operations and the assertions, then all the tests (testXXX methods) have to do is run this N objects against M configurations. With that in mind, running slightly different tests would be done extending or composing the test object/types, independent of the test classes themselves. To run these slight variations, we'd define a test class that runs the variations with M configurations.

Note that I've not prototyped any of that and there are probably better ways to do this.

> 
> I am not asking sarcastically, but I've run into similar issue when
> implementing similar thing in 2LC testsuite.
> 
> Radim
> 
> On 09/09/2015 03:22 PM, Galder Zamarreno wrote:
> > I agree pretty much with everything below:
> >
> > * We overuse test overriding to run the same test with different
> > configuration. I did that same mistake with the functional map API stuff
> > :(
> >
> > * I'm in favour of testsuite restructuring, but I think we really need to
> > start from scratch in a separate testsuite maven project, since we can
> > then add all functional test for all (not only core...etc, but also
> > compatibility tests...etc), and leave its project to test implementation
> > details? Adding this separation would open up the path to create a testkit
> > (as I explained last year in Berlin)
> >
> > * I'm also in favour in defining the test once and running it with
> > different configuration options automatically.
> >
> > * I'm in favour too of randomising (need to check that link) but also we
> > need some quickcheck style tests [1], e.g. a test that verifies that
> > put(K, V) works not matter the type of object passed in.
> >
> > Cheers,
> >
> > [1]
> > https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing
> > --
> > Galder Zamarreño
> > Infinispan, Red Hat
> >
> > ----- Original Message -----
> >> Interesting subject. We also have many tests which (ab)use inheritance
> >> to re-test the same API semantics in slightly different
> >> configurations, like embedded/DIST and embedded/REPL, sometimes
> >> becoming an @Override mess.
> >> It would be far more useful to restructure the testsuite to have such
> >> tests in a single class (no inheritance) and declare - maybe
> >> annotations? - which permutations of configuration parameters should
> >> be valid.
> >>
> >> Among those configuration permutations one would not have "just"
> >> different replication models, but also things like
> >>   - using the same API remotely (Hot Rod)
> >>   - using the same feature but within a WildFly embedded module
> >>   - using the uber jars vs small jars
> >>   - uber jars & remote..
> >>   - remote & embedded modules..
> >>   - remote, uber jars, in OSGi..
> >>
> >> And finally combine with other options:
> >>   - A Query test using: remote client, using uber jars, in OSGi, but
> >> switching JTA implementation, using a new experimental JGroups stack!
> >>
> >> For example many Core API and Query tests are copy/pasted into other
> >> modules as "integration tests", etc.. but we really should just run
> >> the same one in a different environment.
> >>
> >> This would keep our code better maintainable, but also allow some neat
> >> tricks like specify that some configurations should definitely be
> >> tested in some test group (like Galder suggests, one could flag one of
> >> these for "smoke tests", one for "nightly tests"), but you could also
> >> want to flag some configuration settings as a "should work, low
> >> priority for testing".
> >> A smart testsuite could then use a randomizer to generate permutations
> >> of configuration options for those low priority tests which are not
> >> essential; there are great examples of such testsuites in the Haskell
> >> world, and also Lucene and ElasticSearch do it.
> >> A single random seed is used for the whole run, and it's printed
> >> clearly at the start; a single seed will deterministically define all
> >> parameters of the testsuite, so you can reproduce it all by setting a
> >> specific seed when needing to debug a failure.
> >>
> >> http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html
> >>
> >> Thanks,
> >> Sanne
> >>
> >> On 3 September 2015 at 11:34, Galder Zamarreno <galder at redhat.com> wrote:
> >>> Another interesting improvement here would be if you could run all these
> >>> smoke tests with an alternative implementation of AdvancedCache, e.g. one
> >>> based with functional API.
> >>>
> >>> Cheers,
> >>> --
> >>> Galder Zamarreño
> >>> Infinispan, Red Hat
> >>>
> >>> ----- Original Message -----
> >>>> Good post Jiri, this got me thinking :)
> >>>>
> >>>> Running the entire testsuite again with uber jars would add a lot of
> >>>> time
> >>>> to
> >>>> the build time.
> >>>>
> >>>> Maybe we should have a set of tests that must be executed for sure, e.g.
> >>>> like
> >>>> Wildfly's smoke tests [1]. We have "functional" group but right now it
> >>>> covers pretty much all tests.
> >>>>
> >>>> Such tests should live in a separate testsuite, so that we could add the
> >>>> essential tests for *all* components. In a way, we've already done some
> >>>> of
> >>>> this in integrationtests/ but it's not really well structured for this
> >>>> aim.
> >>>>
> >>>> Also, if we would go down this path, something we should take advantage
> >>>> of
> >>>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests
> >>>> in
> >>>> [2], where he used suites to make it faster to run things, by starting a
> >>>> cache manager for distributed caches, running all distributed
> >>>> tests...etc.
> >>>> In a way, I think we can already do this with Arquillian Infinispan
> >>>> integration, so Arquillian would probably well suited for such smoke
> >>>> testsuite.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Cheers,
> >>>>
> >>>> [1] https://github.com/wildfly/wildfly#running-the-testsuite
> >>>> [2]
> >>>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala/org/infinispan/spark
> >>>> --
> >>>> Galder Zamarreño
> >>>> Infinispan, Red Hat
> >>>>
> >>>> ----- Original Message -----
> >>>>> Hi Jiri, comments inline.
> >>>>>
> >>>>> On 2.9.2015 10:40, Jiri Holusa wrote:
> >>>>>> Hi all,
> >>>>>>
> >>>>>> we've been thinking for a while, how to test ISPN uber jars. The
> >>>>>> current
> >>>>>> status is that we actually don't have many tests in the testsuite,
> >>>>>> there
> >>>>>> are few tests in integrationtests/all-embedded-* modules that are
> >>>>>> basically copies of the actual tests in corresponding modules. We
> >>>>>> think
> >>>>>> that this test coverage is not enough and more importantly, they are
> >>>>>> duplicates.
> >>>>>>
> >>>>>> The questions are now following:
> >>>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite?
> >>>>>> Only
> >>>>>> integrationtests module?
> >>>>> The goal is to run the whole test suite because, as you said, we don't
> >>>>> have enough tests in integrationtests/*   And we can't duplicate all
> >>>>> test classes from individual modules here.
> >>>>>
> >>>>>> * how would it run? Create Maven different profiles for "classic" jars
> >>>>>> and
> >>>>>> uber jars? Or try to use some Maven exclusion magic if even possible?
> >>>>>>
> >>>>>> Some time ago, we had discussion about this with Sebastian, who
> >>>>>> suggested
> >>>>>> that running only integrationtests module would be sufficient, because
> >>>>>> uber-jars are really about packaging, not the functionality itself.
> >>>>>> But I
> >>>>>> don't know if the tests coverage is sufficient in that level, I would
> >>>>>> be
> >>>>>> much more confident if we could run the whole ISPN testsuite against
> >>>>>> uber-jars.
> >>>>> Right. Uber-jars are about packaging but you don't know that the
> >>>>> packiging is right until you try all the features and see that
> >>>>> everything works. There might be some classes missing (just for some
> >>>>> particular features), same classes in different packages, the
> >>>>> Manifest.mf might be corrupted and then something won't work in OSGi.
> >>>>>
> >>>>>
> >>>>> I'd prefer a separate Maven profile. IMO, exclusions are too
> >>>>> error-prone.
> >>>>>
> >>>>>
> >>>>> Martin
> >>>>>> I'm opening this for wider discussion as we should agree on the way
> >>>>>> how
> >>>>>> to
> >>>>>> do it, so we could do it right :)
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Jiri
> >>>>>>
> >>>>>>
> >>>>> _______________________________________________
> >>>>> infinispan-dev mailing list
> >>>>> infinispan-dev at lists.jboss.org
> >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> --
> Radim Vansa <rvansa at redhat.com>
> JBoss Performance Team
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev