Re: [infinispan-dev] Uber jars testing

+1 Galder for your abstraction, we might even need a DSL. An additional benefit would be that all "API functional tests" could be tested also in conditions such as running in a in a race with topology changes, instrumenting the timing of parallel code and network operations. As a DSL and using some Byteman we could automatically have it insert network issues or timing issues at specific critical points; such points follow a general pattern so this can be generalized w/o having to code tests for specific critical paths in each functional test independently. A simple example would be to kill a node after a command was sent to it, but before it replies: all the tests should be able to survive that. On 11 September 2015 at 15:19, Galder Zamarreno <galder(a)redhat.com&gt; wrote: > ----- Original Message ----- >> Any plans for tests that are just slightly different for different >> configurations? With inheritance, it's simple - you just override the >> method. If you just run that test on a huge matrix of configurations, >> you end up with having a method with a very complicated switch for >> certain configurations. > ^ I see what you are getting at here. Normally such differences sometimes happen and can be divided into two: operations executed and assertions. Sometimes the operations executed are slightly different, and sometimes the operations are the same, but assertions slightly different. > > I don't have specific ideas about how to solve this but my gut feeling is something like this: > > If we can write tests as objects/types, where we define the operations and the assertions, then all the tests (testXXX methods) have to do is run this N objects against M configurations. With that in mind, running slightly different tests would be done extending or composing the test object/types, independent of the test classes themselves. To run these slight variations, we'd define a test class that runs the variations with M configurations. > > Note that I've not prototyped any of that and there are probably better ways to do this. > >> I am not asking sarcastically, but I've run into similar issue when >> implementing similar thing in 2LC testsuite. >> >> Radim >> >> On 09/09/2015 03:22 PM, Galder Zamarreno wrote: >>> I agree pretty much with everything below: >>> >>> * We overuse test overriding to run the same test with different >>> configuration. I did that same mistake with the functional map API stuff >>> :( >>> >>> * I'm in favour of testsuite restructuring, but I think we really need to >>> start from scratch in a separate testsuite maven project, since we can >>> then add all functional test for all (not only core...etc, but also >>> compatibility tests...etc), and leave its project to test implementation >>> details? Adding this separation would open up the path to create a testkit >>> (as I explained last year in Berlin) >>> >>> * I'm also in favour in defining the test once and running it with >>> different configuration options automatically. >>> >>> * I'm in favour too of randomising (need to check that link) but also we >>> need some quickcheck style tests [1], e.g. a test that verifies that >>> put(K, V) works not matter the type of object passed in. >>> >>> Cheers, >>> >>> [1] >>> https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing >>> -- >>> Galder Zamarreño >>> Infinispan, Red Hat >>> >>> ----- Original Message ----- >>>> Interesting subject. We also have many tests which (ab)use inheritance >>>> to re-test the same API semantics in slightly different >>>> configurations, like embedded/DIST and embedded/REPL, sometimes >>>> becoming an @Override mess. >>>> It would be far more useful to restructure the testsuite to have such >>>> tests in a single class (no inheritance) and declare - maybe >>>> annotations? - which permutations of configuration parameters should >>>> be valid. >>>> >>>> Among those configuration permutations one would not have "just" >>>> different replication models, but also things like >>>> - using the same API remotely (Hot Rod) >>>> - using the same feature but within a WildFly embedded module >>>> - using the uber jars vs small jars >>>> - uber jars & remote.. >>>> - remote & embedded modules.. >>>> - remote, uber jars, in OSGi.. >>>> >>>> And finally combine with other options: >>>> - A Query test using: remote client, using uber jars, in OSGi, but >>>> switching JTA implementation, using a new experimental JGroups stack! >>>> >>>> For example many Core API and Query tests are copy/pasted into other >>>> modules as "integration tests", etc.. but we really should just run >>>> the same one in a different environment. >>>> >>>> This would keep our code better maintainable, but also allow some neat >>>> tricks like specify that some configurations should definitely be >>>> tested in some test group (like Galder suggests, one could flag one of >>>> these for "smoke tests", one for "nightly tests"), but you could also >>>> want to flag some configuration settings as a "should work, low >>>> priority for testing". >>>> A smart testsuite could then use a randomizer to generate permutations >>>> of configuration options for those low priority tests which are not >>>> essential; there are great examples of such testsuites in the Haskell >>>> world, and also Lucene and ElasticSearch do it. >>>> A single random seed is used for the whole run, and it's printed >>>> clearly at the start; a single seed will deterministically define all >>>> parameters of the testsuite, so you can reproduce it all by setting a >>>> specific seed when needing to debug a failure. >>>> >>>> http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-f... >>>> >>>> Thanks, >>>> Sanne >>>> >>>> On 3 September 2015 at 11:34, Galder Zamarreno <galder(a)redhat.com&gt; wrote: >>>>> Another interesting improvement here would be if you could run all these >>>>> smoke tests with an alternative implementation of AdvancedCache, e.g. one >>>>> based with functional API. >>>>> >>>>> Cheers, >>>>> -- >>>>> Galder Zamarreño >>>>> Infinispan, Red Hat >>>>> >>>>> ----- Original Message ----- >>>>>> Good post Jiri, this got me thinking :) >>>>>> >>>>>> Running the entire testsuite again with uber jars would add a lot of >>>>>> time >>>>>> to >>>>>> the build time. >>>>>> >>>>>> Maybe we should have a set of tests that must be executed for sure, e.g. >>>>>> like >>>>>> Wildfly's smoke tests [1]. We have "functional" group but right now it >>>>>> covers pretty much all tests. >>>>>> >>>>>> Such tests should live in a separate testsuite, so that we could add the >>>>>> essential tests for *all* components. In a way, we've already done some >>>>>> of >>>>>> this in integrationtests/ but it's not really well structured for this >>>>>> aim. >>>>>> >>>>>> Also, if we would go down this path, something we should take advantage >>>>>> of >>>>>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests >>>>>> in >>>>>> [2], where he used suites to make it faster to run things, by starting a >>>>>> cache manager for distributed caches, running all distributed >>>>>> tests...etc. >>>>>> In a way, I think we can already do this with Arquillian Infinispan >>>>>> integration, so Arquillian would probably well suited for such smoke >>>>>> testsuite. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> Cheers, >>>>>> >>>>>> [1] https://github.com/wildfly/wildfly#running-the-testsuite >>>>>> [2] >>>>>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala... >>>>>> -- >>>>>> Galder Zamarreño >>>>>> Infinispan, Red Hat >>>>>> >>>>>> ----- Original Message ----- >>>>>>> Hi Jiri, comments inline. >>>>>>> >>>>>>> On 2.9.2015 10:40, Jiri Holusa wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> we've been thinking for a while, how to test ISPN uber jars. The >>>>>>>> current >>>>>>>> status is that we actually don't have many tests in the testsuite, >>>>>>>> there >>>>>>>> are few tests in integrationtests/all-embedded-* modules that are >>>>>>>> basically copies of the actual tests in corresponding modules. We >>>>>>>> think >>>>>>>> that this test coverage is not enough and more importantly, they are >>>>>>>> duplicates. >>>>>>>> >>>>>>>> The questions are now following: >>>>>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite? >>>>>>>> Only >>>>>>>> integrationtests module? >>>>>>> The goal is to run the whole test suite because, as you said, we don't >>>>>>> have enough tests in integrationtests/* And we can't duplicate all >>>>>>> test classes from individual modules here. >>>>>>> >>>>>>>> * how would it run? Create Maven different profiles for "classic" jars >>>>>>>> and >>>>>>>> uber jars? Or try to use some Maven exclusion magic if even possible? >>>>>>>> >>>>>>>> Some time ago, we had discussion about this with Sebastian, who >>>>>>>> suggested >>>>>>>> that running only integrationtests module would be sufficient, because >>>>>>>> uber-jars are really about packaging, not the functionality itself. >>>>>>>> But I >>>>>>>> don't know if the tests coverage is sufficient in that level, I would >>>>>>>> be >>>>>>>> much more confident if we could run the whole ISPN testsuite against >>>>>>>> uber-jars. >>>>>>> Right. Uber-jars are about packaging but you don't know that the >>>>>>> packiging is right until you try all the features and see that >>>>>>> everything works. There might be some classes missing (just for some >>>>>>> particular features), same classes in different packages, the >>>>>>> Manifest.mf might be corrupted and then something won't work in OSGi. >>>>>>> >>>>>>> >>>>>>> I'd prefer a separate Maven profile. IMO, exclusions are too >>>>>>> error-prone. >>>>>>> >>>>>>> >>>>>>> Martin >>>>>>>> I'm opening this for wider discussion as we should agree on the way >>>>>>>> how >>>>>>>> to >>>>>>>> do it, so we could do it right :) >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Jiri >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Radim Vansa <rvansa(a)redhat.com&gt; >> JBoss Performance Team >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev