Re: [infinispan-dev] Uber jars testing

Yes I understand, and I also use Byteman only as last resort (sometimes it is). But while it might not be practical to maintain hundreds of tests using it and relying on specific internals - and in different ways - if you have a single general "test runner" which makes use of it you can limit the maintenance a lot, and it becomes feasible to test the tester to make sure your rules are still valid. And like you suggest, you'd only inject code in specific points, maybe marked with annotations. I didn't mean to suggest a specific way for doing it, I guess most of what we need can be done with a custom interceptor or JGroups protocols. The point is to inject failures at crucial points: nothing new, we do it all the time in many tests, but we don't reuse such patterns for other tests. TBH if I look into some cluster "resilience" tests I wrote myself I'm ashamed from reading the style and the assumptions I'd make some years ago, but there are many such code points scattered around so I'd rather centralize the logic and reuse one properly maintained runner. On 11 September 2015 at 15:54, Radim Vansa <rvansa(a)redhat.com&gt; wrote: > -0.1 for Byteman - although generally I am a fan of Byteman, it seems to > me that the rules are too fragile, since IDE does not give any hints > like "hey, there's a test that inserts some logic to the place you're > modifying". IMO, using Byteman regularly will end up with many tests > silently passing, since the timing is broken after few changes in the > source. > > If we're going to use any instrumentation for testing, I'd consider > putting annotation to the spot I want to hook. I know that this mixes > the test code and actual processing, but makes any dirty tricks more > obvious - and you can consider such annotation an abstract comment > what's happening. > > Haven't prototyped that either :) > > Anyway, I'm eager to see how will the approach described by Galder work, > can't imagine that fully atm. > > My $0.02 > > Radim > > On 09/11/2015 04:28 PM, Sanne Grinovero wrote: >> +1 Galder for your abstraction, we might even need a DSL. >> >> An additional benefit would be that all "API functional tests" could >> be tested also in conditions such as running in a in a race with >> topology changes, instrumenting the timing of parallel code and >> network operations. >> As a DSL and using some Byteman we could automatically have it insert >> network issues or timing issues at specific critical points; such >> points follow a general pattern so this can be generalized w/o having >> to code tests for specific critical paths in each functional test >> independently. >> A simple example would be to kill a node after a command was sent to >> it, but before it replies: all the tests should be able to survive >> that. >> >> >> >> On 11 September 2015 at 15:19, Galder Zamarreno <galder(a)redhat.com&gt; wrote: >>> ----- Original Message ----- >>>> Any plans for tests that are just slightly different for different >>>> configurations? With inheritance, it's simple - you just override the >>>> method. If you just run that test on a huge matrix of configurations, >>>> you end up with having a method with a very complicated switch for >>>> certain configurations. >>> ^ I see what you are getting at here. Normally such differences sometimes happen and can be divided into two: operations executed and assertions. Sometimes the operations executed are slightly different, and sometimes the operations are the same, but assertions slightly different. >>> >>> I don't have specific ideas about how to solve this but my gut feeling is something like this: >>> >>> If we can write tests as objects/types, where we define the operations and the assertions, then all the tests (testXXX methods) have to do is run this N objects against M configurations. With that in mind, running slightly different tests would be done extending or composing the test object/types, independent of the test classes themselves. To run these slight variations, we'd define a test class that runs the variations with M configurations. >>> >>> Note that I've not prototyped any of that and there are probably better ways to do this. >>> >>>> I am not asking sarcastically, but I've run into similar issue when >>>> implementing similar thing in 2LC testsuite. >>>> >>>> Radim >>>> >>>> On 09/09/2015 03:22 PM, Galder Zamarreno wrote: >>>>> I agree pretty much with everything below: >>>>> >>>>> * We overuse test overriding to run the same test with different >>>>> configuration. I did that same mistake with the functional map API stuff >>>>> :( >>>>> >>>>> * I'm in favour of testsuite restructuring, but I think we really need to >>>>> start from scratch in a separate testsuite maven project, since we can >>>>> then add all functional test for all (not only core...etc, but also >>>>> compatibility tests...etc), and leave its project to test implementation >>>>> details? Adding this separation would open up the path to create a testkit >>>>> (as I explained last year in Berlin) >>>>> >>>>> * I'm also in favour in defining the test once and running it with >>>>> different configuration options automatically. >>>>> >>>>> * I'm in favour too of randomising (need to check that link) but also we >>>>> need some quickcheck style tests [1], e.g. a test that verifies that >>>>> put(K, V) works not matter the type of object passed in. >>>>> >>>>> Cheers, >>>>> >>>>> [1] >>>>> https://www.fpcomplete.com/user/pbv/an-introduction-to-quickcheck-testing >>>>> -- >>>>> Galder Zamarreño >>>>> Infinispan, Red Hat >>>>> >>>>> ----- Original Message ----- >>>>>> Interesting subject. We also have many tests which (ab)use inheritance >>>>>> to re-test the same API semantics in slightly different >>>>>> configurations, like embedded/DIST and embedded/REPL, sometimes >>>>>> becoming an @Override mess. >>>>>> It would be far more useful to restructure the testsuite to have such >>>>>> tests in a single class (no inheritance) and declare - maybe >>>>>> annotations? - which permutations of configuration parameters should >>>>>> be valid. >>>>>> >>>>>> Among those configuration permutations one would not have "just" >>>>>> different replication models, but also things like >>>>>> - using the same API remotely (Hot Rod) >>>>>> - using the same feature but within a WildFly embedded module >>>>>> - using the uber jars vs small jars >>>>>> - uber jars & remote.. >>>>>> - remote & embedded modules.. >>>>>> - remote, uber jars, in OSGi.. >>>>>> >>>>>> And finally combine with other options: >>>>>> - A Query test using: remote client, using uber jars, in OSGi, but >>>>>> switching JTA implementation, using a new experimental JGroups stack! >>>>>> >>>>>> For example many Core API and Query tests are copy/pasted into other >>>>>> modules as "integration tests", etc.. but we really should just run >>>>>> the same one in a different environment. >>>>>> >>>>>> This would keep our code better maintainable, but also allow some neat >>>>>> tricks like specify that some configurations should definitely be >>>>>> tested in some test group (like Galder suggests, one could flag one of >>>>>> these for "smoke tests", one for "nightly tests"), but you could also >>>>>> want to flag some configuration settings as a "should work, low >>>>>> priority for testing". >>>>>> A smart testsuite could then use a randomizer to generate permutations >>>>>> of configuration options for those low priority tests which are not >>>>>> essential; there are great examples of such testsuites in the Haskell >>>>>> world, and also Lucene and ElasticSearch do it. >>>>>> A single random seed is used for the whole run, and it's printed >>>>>> clearly at the start; a single seed will deterministically define all >>>>>> parameters of the testsuite, so you can reproduce it all by setting a >>>>>> specific seed when needing to debug a failure. >>>>>> >>>>>> http://blog.mikemccandless.com/2011/03/your-test-cases-should-sometimes-f... >>>>>> >>>>>> Thanks, >>>>>> Sanne >>>>>> >>>>>> On 3 September 2015 at 11:34, Galder Zamarreno <galder(a)redhat.com&gt; wrote: >>>>>>> Another interesting improvement here would be if you could run all these >>>>>>> smoke tests with an alternative implementation of AdvancedCache, e.g. one >>>>>>> based with functional API. >>>>>>> >>>>>>> Cheers, >>>>>>> -- >>>>>>> Galder Zamarreño >>>>>>> Infinispan, Red Hat >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> Good post Jiri, this got me thinking :) >>>>>>>> >>>>>>>> Running the entire testsuite again with uber jars would add a lot of >>>>>>>> time >>>>>>>> to >>>>>>>> the build time. >>>>>>>> >>>>>>>> Maybe we should have a set of tests that must be executed for sure, e.g. >>>>>>>> like >>>>>>>> Wildfly's smoke tests [1]. We have "functional" group but right now it >>>>>>>> covers pretty much all tests. >>>>>>>> >>>>>>>> Such tests should live in a separate testsuite, so that we could add the >>>>>>>> essential tests for *all* components. In a way, we've already done some >>>>>>>> of >>>>>>>> this in integrationtests/ but it's not really well structured for this >>>>>>>> aim. >>>>>>>> >>>>>>>> Also, if we would go down this path, something we should take advantage >>>>>>>> of >>>>>>>> (if possible with JUnit/TestNG) is what Gustavo did with the Spark tests >>>>>>>> in >>>>>>>> [2], where he used suites to make it faster to run things, by starting a >>>>>>>> cache manager for distributed caches, running all distributed >>>>>>>> tests...etc. >>>>>>>> In a way, I think we can already do this with Arquillian Infinispan >>>>>>>> integration, so Arquillian would probably well suited for such smoke >>>>>>>> testsuite. >>>>>>>> >>>>>>>> Thoughts? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> [1] https://github.com/wildfly/wildfly#running-the-testsuite >>>>>>>> [2] >>>>>>>> https://github.com/infinispan/infinispan-spark/tree/master/src/test/scala... >>>>>>>> -- >>>>>>>> Galder Zamarreño >>>>>>>> Infinispan, Red Hat >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> Hi Jiri, comments inline. >>>>>>>>> >>>>>>>>> On 2.9.2015 10:40, Jiri Holusa wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> we've been thinking for a while, how to test ISPN uber jars. The >>>>>>>>>> current >>>>>>>>>> status is that we actually don't have many tests in the testsuite, >>>>>>>>>> there >>>>>>>>>> are few tests in integrationtests/all-embedded-* modules that are >>>>>>>>>> basically copies of the actual tests in corresponding modules. We >>>>>>>>>> think >>>>>>>>>> that this test coverage is not enough and more importantly, they are >>>>>>>>>> duplicates. >>>>>>>>>> >>>>>>>>>> The questions are now following: >>>>>>>>>> * which tests should be invoked with uber-jars? Whole ISPN testsuite? >>>>>>>>>> Only >>>>>>>>>> integrationtests module? >>>>>>>>> The goal is to run the whole test suite because, as you said, we don't >>>>>>>>> have enough tests in integrationtests/* And we can't duplicate all >>>>>>>>> test classes from individual modules here. >>>>>>>>> >>>>>>>>>> * how would it run? Create Maven different profiles for "classic" jars >>>>>>>>>> and >>>>>>>>>> uber jars? Or try to use some Maven exclusion magic if even possible? >>>>>>>>>> >>>>>>>>>> Some time ago, we had discussion about this with Sebastian, who >>>>>>>>>> suggested >>>>>>>>>> that running only integrationtests module would be sufficient, because >>>>>>>>>> uber-jars are really about packaging, not the functionality itself. >>>>>>>>>> But I >>>>>>>>>> don't know if the tests coverage is sufficient in that level, I would >>>>>>>>>> be >>>>>>>>>> much more confident if we could run the whole ISPN testsuite against >>>>>>>>>> uber-jars. >>>>>>>>> Right. Uber-jars are about packaging but you don't know that the >>>>>>>>> packiging is right until you try all the features and see that >>>>>>>>> everything works. There might be some classes missing (just for some >>>>>>>>> particular features), same classes in different packages, the >>>>>>>>> Manifest.mf might be corrupted and then something won't work in OSGi. >>>>>>>>> >>>>>>>>> >>>>>>>>> I'd prefer a separate Maven profile. IMO, exclusions are too >>>>>>>>> error-prone. >>>>>>>>> >>>>>>>>> >>>>>>>>> Martin >>>>>>>>>> I'm opening this for wider discussion as we should agree on the way >>>>>>>>>> how >>>>>>>>>> to >>>>>>>>>> do it, so we could do it right :) >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Jiri >>>>>>>>>> >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> infinispan-dev mailing list >>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> infinispan-dev mailing list >>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Radim Vansa <rvansa(a)redhat.com&gt; >>>> JBoss Performance Team >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Radim Vansa <rvansa(a)redhat.com&gt; > JBoss Performance Team > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev