Re: [infinispan-dev] Uber jars testing

Any plans for tests that are just slightly different for different configurations? With inheritance, it's simple - you just override the method. If you just run that test on a huge matrix of configurations, you end up with having a method with a very complicated switch for certain configurations.

I am not asking sarcastically, but I've run into similar issue when implementing similar thing in 2LC testsuite. Radim On 09/09/2015 03:22 PM, Galder Zamarreno wrote: > I agree pretty much with everything below: > > * We overuse test overriding to run the same test with different > configuration. I did that same mistake with the functional map API stuff > :( > > * I'm in favour of testsuite restructuring, but I think we really need to > start from scratch in a separate testsuite maven project, since we can > then add all functional test for all (not only core...etc, but also > compatibility tests...etc), and leave its project to test implementation > details? Adding this separation would open up the path to create a testkit > (as I explained last year in Berlin) > > * I'm also in favour in defining the test once and running it with > different configuration options automatically. > > * I'm in favour too of randomising (need to check that link) but also we > need some quickcheck style tests [1], e.g. a test that verifies that > put(K, V) works not matter the type of object passed in. > > Cheers, > > [1] > https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing > -- > Galder Zamarreño > Infinispan, Red Hat > > ----- Original Message ----- >> Interesting subject. We also have many tests which (ab)use inheritance >> to re-test the same API semantics in slightly different >> configurations, like embedded/DIST and embedded/REPL, sometimes >> becoming an @Override mess. >> It would be far more useful to restructure the testsuite to have such >> tests in a single class (no inheritance) and declare - maybe >> annotations? - which permutations of configuration parameters should >> be valid. >> >> Among those configuration permutations one would not have "just" >> different replication models, but also things like >> - using the same API remotely (Hot Rod) >> - using the same feature but within a WildFly embedded module >> - using the uber jars vs small jars >> - uber jars & remote.. >> - remote & embedded modules.. >> - remote, uber jars, in OSGi.. >> >> And finally combine with other options: >> - A Query test using: remote client, using uber jars, in OSGi, but >> switching JTA implementation, using a new experimental JGroups stack! >> >> For example many Core API and Query tests are copy/pasted into other >> modules as "integration tests", etc.. but we really should just run >> the same one in a different environment. >> >> This would keep our code better maintainable, but also allow some neat >> tricks like specify that some configurations should definitely be >> tested in some test group (like Galder suggests, one could flag one of >> these for "smoke tests", one for "nightly tests"), but you could also >> want to flag some configuration settings as a "should work, low >> priority for testing". >> A smart testsuite could then use a randomizer to generate permutations >> of configuration options for those low priority tests which are not >> essential; there are great examples of such testsuites in the Haskell >> world, and also Lucene and ElasticSearch do it. >> A single random seed is used for the whole run, and it's printed >> clearly at the start; a single seed will deterministically define all >> parameters of the testsuite, so you can reproduce it all by setting a >> specific seed when needing to debug a failure. >> >> http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-f... >> >> Thanks, >> Sanne >> >> On 3 September 2015 at 11:34, Galder Zamarreno <galder(a)redhat.com&gt; wrote: >>> Another interesting improvement here would be if you could run all these >>> smoke tests with an alternative implementation of AdvancedCache, e.g. one >>> based with functional API. >>> >>> Cheers, >>> -- >>> Galder Zamarreño >>> Infinispan, Red Hat >>> >>> ----- Original Message ----- >>>> Good post Jiri, this got me thinking :) >>>> >>>> Running the entire testsuite again with uber jars would add a lot of >>>> time >>>> to >>>> the build time. >>>> >>>> Maybe we should have a set of tests that must be executed for sure, e.g. >>>> like >>>> Wildfly's smoke tests [1]. We have "functional" group but right now it >>>> covers pretty much all tests. >>>> >>>> Such tests should live in a separate testsuite, so that we could add the >>>> essential tests for *all* components. In a way, we've already done some >>>> of >>>> this in integrationtests/ but it's not really well structured for this >>>> aim. >>>> >>>> Also, if we would go down this path, something we should take advantage >>>> of >>>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests >>>> in >>>> [2], where he used suites to make it faster to run things, by starting a >>>> cache manager for distributed caches, running all distributed >>>> tests...etc. >>>> In a way, I think we can already do this with Arquillian Infinispan >>>> integration, so Arquillian would probably well suited for such smoke >>>> testsuite. >>>> >>>> Thoughts? >>>> >>>> Cheers, >>>> >>>> [1] https://github.com/wildfly/wildfly#running-the-testsuite >>>> [2] >>>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala... >>>> -- >>>> Galder Zamarreño >>>> Infinispan, Red Hat >>>> >>>> ----- Original Message ----- >>>>> Hi Jiri, comments inline. >>>>> >>>>> On 2.9.2015 10:40, Jiri Holusa wrote: >>>>>> Hi all, >>>>>> >>>>>> we've been thinking for a while, how to test ISPN uber jars. The >>>>>> current >>>>>> status is that we actually don't have many tests in the testsuite, >>>>>> there >>>>>> are few tests in integrationtests/all-embedded-* modules that are >>>>>> basically copies of the actual tests in corresponding modules. We >>>>>> think >>>>>> that this test coverage is not enough and more importantly, they are >>>>>> duplicates. >>>>>> >>>>>> The questions are now following: >>>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite? >>>>>> Only >>>>>> integrationtests module? >>>>> The goal is to run the whole test suite because, as you said, we don't >>>>> have enough tests in integrationtests/* And we can't duplicate all >>>>> test classes from individual modules here. >>>>> >>>>>> * how would it run? Create Maven different profiles for "classic" jars >>>>>> and >>>>>> uber jars? Or try to use some Maven exclusion magic if even possible? >>>>>> >>>>>> Some time ago, we had discussion about this with Sebastian, who >>>>>> suggested >>>>>> that running only integrationtests module would be sufficient, because >>>>>> uber-jars are really about packaging, not the functionality itself. >>>>>> But I >>>>>> don't know if the tests coverage is sufficient in that level, I would >>>>>> be >>>>>> much more confident if we could run the whole ISPN testsuite against >>>>>> uber-jars. >>>>> Right. Uber-jars are about packaging but you don't know that the >>>>> packiging is right until you try all the features and see that >>>>> everything works. There might be some classes missing (just for some >>>>> particular features), same classes in different packages, the >>>>> Manifest.mf might be corrupted and then something won't work in OSGi. >>>>> >>>>> >>>>> I'd prefer a separate Maven profile. IMO, exclusions are too >>>>> error-prone. >>>>> >>>>> >>>>> Martin >>>>>> I'm opening this for wider discussion as we should agree on the way >>>>>> how >>>>>> to >>>>>> do it, so we could do it right :) >>>>>> >>>>>> Cheers, >>>>>> Jiri >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Radim Vansa <rvansa(a)redhat.com&gt; JBoss Performance Team _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev