Re: [infinispan-dev] Uber jars testing

----- Original Message ----- > Any plans for tests that are just slightly different for different > configurations? With inheritance, it's simple - you just override the > method. If you just run that test on a huge matrix of configurations, > you end up with having a method with a very complicated switch for > certain configurations. ^ I see what you are getting at here. Normally such differences sometimes happen and can be divided into two: operations executed and assertions. Sometimes the operations executed are slightly different, and sometimes the operations are the same, but assertions slightly different. I don't have specific ideas about how to solve this but my gut feeling is something like this: If we can write tests as objects/types, where we define the operations and the assertions, then all the tests (testXXX methods) have to do is run this N objects against M configurations. With that in mind, running slightly different tests would be done extending or composing the test object/types, independent of the test classes themselves. To run these slight variations, we'd define a test class that runs the variations with M configurations. Note that I've not prototyped any of that and there are probably better ways to do this. > > I am not asking sarcastically, but I've run into similar issue when > implementing similar thing in 2LC testsuite. > > Radim > > On 09/09/2015 03:22 PM, Galder Zamarreno wrote: > > I agree pretty much with everything below: > > > > * We overuse test overriding to run the same test with different > > configuration. I did that same mistake with the functional map API stuff > > :( > > > > * I'm in favour of testsuite restructuring, but I think we really need to > > start from scratch in a separate testsuite maven project, since we can > > then add all functional test for all (not only core...etc, but also > > compatibility tests...etc), and leave its project to test implementation > > details? Adding this separation would open up the path to create a testkit > > (as I explained last year in Berlin) > > > > * I'm also in favour in defining the test once and running it with > > different configuration options automatically. > > > > * I'm in favour too of randomising (need to check that link) but also we > > need some quickcheck style tests [1], e.g. a test that verifies that > > put(K, V) works not matter the type of object passed in. > > > > Cheers, > > > > [1] > > https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing > > -- > > Galder Zamarreño > > Infinispan, Red Hat > > > > ----- Original Message ----- > >> Interesting subject. We also have many tests which (ab)use inheritance > >> to re-test the same API semantics in slightly different > >> configurations, like embedded/DIST and embedded/REPL, sometimes > >> becoming an @Override mess. > >> It would be far more useful to restructure the testsuite to have such > >> tests in a single class (no inheritance) and declare - maybe > >> annotations? - which permutations of configuration parameters should > >> be valid. > >> > >> Among those configuration permutations one would not have "just" > >> different replication models, but also things like > >> - using the same API remotely (Hot Rod) > >> - using the same feature but within a WildFly embedded module > >> - using the uber jars vs small jars > >> - uber jars & remote.. > >> - remote & embedded modules.. > >> - remote, uber jars, in OSGi.. > >> > >> And finally combine with other options: > >> - A Query test using: remote client, using uber jars, in OSGi, but > >> switching JTA implementation, using a new experimental JGroups stack! > >> > >> For example many Core API and Query tests are copy/pasted into other > >> modules as "integration tests", etc.. but we really should just run > >> the same one in a different environment. > >> > >> This would keep our code better maintainable, but also allow some neat > >> tricks like specify that some configurations should definitely be > >> tested in some test group (like Galder suggests, one could flag one of > >> these for "smoke tests", one for "nightly tests"), but you could also > >> want to flag some configuration settings as a "should work, low > >> priority for testing". > >> A smart testsuite could then use a randomizer to generate permutations > >> of configuration options for those low priority tests which are not > >> essential; there are great examples of such testsuites in the Haskell > >> world, and also Lucene and ElasticSearch do it. > >> A single random seed is used for the whole run, and it's printed > >> clearly at the start; a single seed will deterministically define all > >> parameters of the testsuite, so you can reproduce it all by setting a > >> specific seed when needing to debug a failure. > >> > >> http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-f... > >> > >> Thanks, > >> Sanne > >> > >> On 3 September 2015 at 11:34, Galder Zamarreno <galder(a)redhat.com&gt; wrote: > >>> Another interesting improvement here would be if you could run all these > >>> smoke tests with an alternative implementation of AdvancedCache, e.g. one > >>> based with functional API. > >>> > >>> Cheers, > >>> -- > >>> Galder Zamarreño > >>> Infinispan, Red Hat > >>> > >>> ----- Original Message ----- > >>>> Good post Jiri, this got me thinking :) > >>>> > >>>> Running the entire testsuite again with uber jars would add a lot of > >>>> time > >>>> to > >>>> the build time. > >>>> > >>>> Maybe we should have a set of tests that must be executed for sure, e.g. > >>>> like > >>>> Wildfly's smoke tests [1]. We have "functional" group but right now it > >>>> covers pretty much all tests. > >>>> > >>>> Such tests should live in a separate testsuite, so that we could add the > >>>> essential tests for *all* components. In a way, we've already done some > >>>> of > >>>> this in integrationtests/ but it's not really well structured for this > >>>> aim. > >>>> > >>>> Also, if we would go down this path, something we should take advantage > >>>> of > >>>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests > >>>> in > >>>> [2], where he used suites to make it faster to run things, by starting a > >>>> cache manager for distributed caches, running all distributed > >>>> tests...etc. > >>>> In a way, I think we can already do this with Arquillian Infinispan > >>>> integration, so Arquillian would probably well suited for such smoke > >>>> testsuite. > >>>> > >>>> Thoughts? > >>>> > >>>> Cheers, > >>>> > >>>> [1] https://github.com/wildfly/wildfly#running-the-testsuite > >>>> [2] > >>>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala... > >>>> -- > >>>> Galder Zamarreño > >>>> Infinispan, Red Hat > >>>> > >>>> ----- Original Message ----- > >>>>> Hi Jiri, comments inline. > >>>>> > >>>>> On 2.9.2015 10:40, Jiri Holusa wrote: > >>>>>> Hi all, > >>>>>> > >>>>>> we've been thinking for a while, how to test ISPN uber jars. The > >>>>>> current > >>>>>> status is that we actually don't have many tests in the testsuite, > >>>>>> there > >>>>>> are few tests in integrationtests/all-embedded-* modules that are > >>>>>> basically copies of the actual tests in corresponding modules. We > >>>>>> think > >>>>>> that this test coverage is not enough and more importantly, they are > >>>>>> duplicates. > >>>>>> > >>>>>> The questions are now following: > >>>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite? > >>>>>> Only > >>>>>> integrationtests module? > >>>>> The goal is to run the whole test suite because, as you said, we don't > >>>>> have enough tests in integrationtests/* And we can't duplicate all > >>>>> test classes from individual modules here. > >>>>> > >>>>>> * how would it run? Create Maven different profiles for "classic" jars > >>>>>> and > >>>>>> uber jars? Or try to use some Maven exclusion magic if even possible? > >>>>>> > >>>>>> Some time ago, we had discussion about this with Sebastian, who > >>>>>> suggested > >>>>>> that running only integrationtests module would be sufficient, because > >>>>>> uber-jars are really about packaging, not the functionality itself. > >>>>>> But I > >>>>>> don't know if the tests coverage is sufficient in that level, I would > >>>>>> be > >>>>>> much more confident if we could run the whole ISPN testsuite against > >>>>>> uber-jars. > >>>>> Right. Uber-jars are about packaging but you don't know that the > >>>>> packiging is right until you try all the features and see that > >>>>> everything works. There might be some classes missing (just for some > >>>>> particular features), same classes in different packages, the > >>>>> Manifest.mf might be corrupted and then something won't work in OSGi. > >>>>> > >>>>> > >>>>> I'd prefer a separate Maven profile. IMO, exclusions are too > >>>>> error-prone. > >>>>> > >>>>> > >>>>> Martin > >>>>>> I'm opening this for wider discussion as we should agree on the way > >>>>>> how > >>>>>> to > >>>>>> do it, so we could do it right :) > >>>>>> > >>>>>> Cheers, > >>>>>> Jiri > >>>>>> > >>>>>> > >>>>> _______________________________________________ > >>>>> infinispan-dev mailing list > >>>>> infinispan-dev(a)lists.jboss.org > >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > >>>>> > >>> _______________________________________________ > >>> infinispan-dev mailing list > >>> infinispan-dev(a)lists.jboss.org > >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev > >> _______________________________________________ > >> infinispan-dev mailing list > >> infinispan-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > > infinispan-dev mailing list > > infinispan-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa <rvansa(a)redhat.com&gt; > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev