[infinispan-dev] Uber jars testing

[Sanne Grinovero]

Interesting subject. We also have many tests which (ab)use inheritance
to re-test the same API semantics in slightly different
configurations, like embedded/DIST and embedded/REPL, sometimes
becoming an @Override mess.
It would be far more useful to restructure the testsuite to have such
tests in a single class (no inheritance) and declare - maybe
annotations? - which permutations of configuration parameters should
be valid.

Among those configuration permutations one would not have "just"
different replication models, but also things like
 - using the same API remotely (Hot Rod)
 - using the same feature but within a WildFly embedded module
 - using the uber jars vs small jars
 - uber jars & remote..
 - remote & embedded modules..
 - remote, uber jars, in OSGi..

And finally combine with other options:
 - A Query test using: remote client, using uber jars, in OSGi, but
switching JTA implementation, using a new experimental JGroups stack!

For example many Core API and Query tests are copy/pasted into other
modules as "integration tests", etc.. but we really should just run
the same one in a different environment.

This would keep our code better maintainable, but also allow some neat
tricks like specify that some configurations should definitely be
tested in some test group (like Galder suggests, one could flag one of
these for "smoke tests", one for "nightly tests"), but you could also
want to flag some configuration settings as a "should work, low
priority for testing".
A smart testsuite could then use a randomizer to generate permutations
of configuration options for those low priority tests which are not
essential; there are great examples of such testsuites in the Haskell
world, and also Lucene and ElasticSearch do it.
A single random seed is used for the whole run, and it's printed
clearly at the start; a single seed will deterministically define all
parameters of the testsuite, so you can reproduce it all by setting a
specific seed when needing to debug a failure.

http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-fail.html

Thanks,
Sanne

On 3 September 2015 at 11:34, Galder Zamarreno <galder at redhat.com> wrote:
> Another interesting improvement here would be if you could run all these smoke tests with an alternative implementation of AdvancedCache, e.g. one based with functional API.
>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
> ----- Original Message -----
>> Good post Jiri, this got me thinking :)
>>
>> Running the entire testsuite again with uber jars would add a lot of time to
>> the build time.
>>
>> Maybe we should have a set of tests that must be executed for sure, e.g. like
>> Wildfly's smoke tests [1]. We have "functional" group but right now it
>> covers pretty much all tests.
>>
>> Such tests should live in a separate testsuite, so that we could add the
>> essential tests for *all* components. In a way, we've already done some of
>> this in integrationtests/ but it's not really well structured for this aim.
>>
>> Also, if we would go down this path, something we should take advantage of
>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests in
>> [2], where he used suites to make it faster to run things, by starting a
>> cache manager for distributed caches, running all distributed tests...etc.
>> In a way, I think we can already do this with Arquillian Infinispan
>> integration, so Arquillian would probably well suited for such smoke
>> testsuite.
>>
>> Thoughts?
>>
>> Cheers,
>>
>> [1] https://github.com/wildfly/wildfly#running-the-testsuite
>> [2]
>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala/org/infinispan/spark
>> --
>> Galder Zamarreño
>> Infinispan, Red Hat
>>
>> ----- Original Message -----
>> > Hi Jiri, comments inline.
>> >
>> > On 2.9.2015 10:40, Jiri Holusa wrote:
>> > > Hi all,
>> > >
>> > > we've been thinking for a while, how to test ISPN uber jars. The current
>> > > status is that we actually don't have many tests in the testsuite, there
>> > > are few tests in integrationtests/all-embedded-* modules that are
>> > > basically copies of the actual tests in corresponding modules. We think
>> > > that this test coverage is not enough and more importantly, they are
>> > > duplicates.
>> > >
>> > > The questions are now following:
>> > > * which tests should be invoked with uber-jars? Whole ISPN testsuite?
>> > > Only
>> > > integrationtests module?
>> >
>> > The goal is to run the whole test suite because, as you said, we don't
>> > have enough tests in integrationtests/*   And we can't duplicate all
>> > test classes from individual modules here.
>> >
>> > > * how would it run? Create Maven different profiles for "classic" jars
>> > > and
>> > > uber jars? Or try to use some Maven exclusion magic if even possible?
>> > >
>> > > Some time ago, we had discussion about this with Sebastian, who suggested
>> > > that running only integrationtests module would be sufficient, because
>> > > uber-jars are really about packaging, not the functionality itself. But I
>> > > don't know if the tests coverage is sufficient in that level, I would be
>> > > much more confident if we could run the whole ISPN testsuite against
>> > > uber-jars.
>> > Right. Uber-jars are about packaging but you don't know that the
>> > packiging is right until you try all the features and see that
>> > everything works. There might be some classes missing (just for some
>> > particular features), same classes in different packages, the
>> > Manifest.mf might be corrupted and then something won't work in OSGi.
>> >
>> >
>> > I'd prefer a separate Maven profile. IMO, exclusions are too error-prone.
>> >
>> >
>> > Martin
>> > >
>> > > I'm opening this for wider discussion as we should agree on the way how
>> > > to
>> > > do it, so we could do it right :)
>> > >
>> > > Cheers,
>> > > Jiri
>> > >
>> > >
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev