Re: [infinispan-dev] Uber jars testing

I agree pretty much with everything below: * We overuse test overriding to run the same test with different configuration. I did that same mistake with the functional map API stuff :( * I'm in favour of testsuite restructuring, but I think we really need to start from scratch in a separate testsuite maven project, since we can then add all functional test for all (not only core...etc, but also compatibility tests...etc), and leave its project to test implementation details? Adding this separation would open up the path to create a testkit (as I explained last year in Berlin) * I'm also in favour in defining the test once and running it with different configuration options automatically. * I'm in favour too of randomising (need to check that link) but also we need some quickcheck style tests [1], e.g. a test that verifies that put(K, V) works not matter the type of object passed in. Cheers, [1] https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing -- Galder Zamarreño Infinispan, Red Hat ----- Original Message ----- > Interesting subject. We also have many tests which (ab)use inheritance > to re-test the same API semantics in slightly different > configurations, like embedded/DIST and embedded/REPL, sometimes > becoming an @Override mess. > It would be far more useful to restructure the testsuite to have such > tests in a single class (no inheritance) and declare - maybe > annotations? - which permutations of configuration parameters should > be valid. > > Among those configuration permutations one would not have "just" > different replication models, but also things like > - using the same API remotely (Hot Rod) > - using the same feature but within a WildFly embedded module > - using the uber jars vs small jars > - uber jars & remote.. > - remote & embedded modules.. > - remote, uber jars, in OSGi.. > > And finally combine with other options: > - A Query test using: remote client, using uber jars, in OSGi, but > switching JTA implementation, using a new experimental JGroups stack! > > For example many Core API and Query tests are copy/pasted into other > modules as "integration tests", etc.. but we really should just run > the same one in a different environment. > > This would keep our code better maintainable, but also allow some neat > tricks like specify that some configurations should definitely be > tested in some test group (like Galder suggests, one could flag one of > these for "smoke tests", one for "nightly tests"), but you could also > want to flag some configuration settings as a "should work, low > priority for testing". > A smart testsuite could then use a randomizer to generate permutations > of configuration options for those low priority tests which are not > essential; there are great examples of such testsuites in the Haskell > world, and also Lucene and ElasticSearch do it. > A single random seed is used for the whole run, and it's printed > clearly at the start; a single seed will deterministically define all > parameters of the testsuite, so you can reproduce it all by setting a > specific seed when needing to debug a failure. > > http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-f... > > Thanks, > Sanne > > On 3 September 2015 at 11:34, Galder Zamarreno <galder(a)redhat.com&gt; wrote: >> Another interesting improvement here would be if you could run all these >> smoke tests with an alternative implementation of AdvancedCache, e.g. one >> based with functional API. >> >> Cheers, >> -- >> Galder Zamarreño >> Infinispan, Red Hat >> >> ----- Original Message ----- >>> Good post Jiri, this got me thinking :) >>> >>> Running the entire testsuite again with uber jars would add a lot of time >>> to >>> the build time. >>> >>> Maybe we should have a set of tests that must be executed for sure, e.g. >>> like >>> Wildfly's smoke tests [1]. We have "functional" group but right now it >>> covers pretty much all tests. >>> >>> Such tests should live in a separate testsuite, so that we could add the >>> essential tests for *all* components. In a way, we've already done some of >>> this in integrationtests/ but it's not really well structured for this >>> aim. >>> >>> Also, if we would go down this path, something we should take advantage of >>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests >>> in >>> [2], where he used suites to make it faster to run things, by starting a >>> cache manager for distributed caches, running all distributed tests...etc. >>> In a way, I think we can already do this with Arquillian Infinispan >>> integration, so Arquillian would probably well suited for such smoke >>> testsuite. >>> >>> Thoughts? >>> >>> Cheers, >>> >>> [1] https://github.com/wildfly/wildfly#running-the-testsuite >>> [2] >>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala... >>> -- >>> Galder Zamarreño >>> Infinispan, Red Hat >>> >>> ----- Original Message ----- >>>> Hi Jiri, comments inline. >>>> >>>> On 2.9.2015 10:40, Jiri Holusa wrote: >>>>> Hi all, >>>>> >>>>> we've been thinking for a while, how to test ISPN uber jars. The >>>>> current >>>>> status is that we actually don't have many tests in the testsuite, >>>>> there >>>>> are few tests in integrationtests/all-embedded-* modules that are >>>>> basically copies of the actual tests in corresponding modules. We >>>>> think >>>>> that this test coverage is not enough and more importantly, they are >>>>> duplicates. >>>>> >>>>> The questions are now following: >>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite? >>>>> Only >>>>> integrationtests module? >>>> The goal is to run the whole test suite because, as you said, we don't >>>> have enough tests in integrationtests/* And we can't duplicate all >>>> test classes from individual modules here. >>>> >>>>> * how would it run? Create Maven different profiles for "classic" jars >>>>> and >>>>> uber jars? Or try to use some Maven exclusion magic if even possible? >>>>> >>>>> Some time ago, we had discussion about this with Sebastian, who >>>>> suggested >>>>> that running only integrationtests module would be sufficient, because >>>>> uber-jars are really about packaging, not the functionality itself. >>>>> But I >>>>> don't know if the tests coverage is sufficient in that level, I would >>>>> be >>>>> much more confident if we could run the whole ISPN testsuite against >>>>> uber-jars. >>>> Right. Uber-jars are about packaging but you don't know that the >>>> packiging is right until you try all the features and see that >>>> everything works. There might be some classes missing (just for some >>>> particular features), same classes in different packages, the >>>> Manifest.mf might be corrupted and then something won't work in OSGi. >>>> >>>> >>>> I'd prefer a separate Maven profile. IMO, exclusions are too >>>> error-prone. >>>> >>>> >>>> Martin >>>>> I'm opening this for wider discussion as we should agree on the way >>>>> how >>>>> to >>>>> do it, so we could do it right :) >>>>> >>>>> Cheers, >>>>> Jiri >>>>> >>>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev