Re: [jbosscache-dev] AS Integration tests in JBC test suites

Mircea Markus wrote: > Brian Stansberry wrote: >> Manik Surtani wrote: >>> +1. >>> >>> How do you foresee this, as a separate suite of tests? E.g., src/ >>> integration-test/java? And how about integration with Hudson? >>> Run on every build? Every release? >>> >> >> Hadn't really thought that far. :-) >> >> I *was* thinking in terms of just a separate package (with >> subpackages) under src/test/java. Say org.jboss.cache.integration >> or something. But, yeah, there's no need to run these every build, >> just before release, so lets organize in whatever way makes it >> easy to control when they get run. >> > Do we expect these tests to perform slow? Maybe a bit. The general AS use cases involve things like failover after async repl, so you need to pause a bit to be sure the repl had time to happen. Also passivation, where there is some pausing to allow passivation timeouts to happen. Stuff like that. > If not, I guess running them together with the common test suite > would only bring an additional safety net... I also think that it > worths having them in an different package as they will mainly be > written by the teams that needs integration (Brian I think :) ) For sure we want a distinct package. :) Gotta ensure we cover all the use cases; if tests are spread in random places it's too hard to track whether we did that or not. >> Re: hudson, since we don't want them run on every build, IMO it >> would be good to have a separate, unscheduled, hudson job to run >> w/ them included. They need to run before release, but we should >> also make it easy for a dev to kick off a run that includes them >> so problems can be caught early. >> >>> On 18 Mar 2008, at 17:26, Brian Stansberry wrote: >>>> I'd like to propose adding a package structure in the core cache >>>> and pojo cache testsuites for adding tests of compliance with >>>> the behavior the AS and Hibernate expect from JBC. Basically, in >>>> these packages the AS developers would write tests that simulate >>>> the AS and Hibernate usage of JBC. Simulations would *not* add >>>> any new dependencies. Goal over time is to 99.5% eliminate the >>>> chances of finding JBC-caused regressions in the AS/Hibernate >>>> testsuites by ensuring they are caught in JBC itself. >>>> >>>> Right now a lot of issues slip through because the JBC test >>>> suites have a hard time covering all of its permutations of >>>> features. Something gets changed and a test gets added, but it >>>> doesn't cover the configuration set AS/Hibernate uses, so we get >>>> a regression at the AS level. >>>> >>>> Downside to this is theoretically everything we add in these new >>>> packages should be redundant. But in practice that will be hard >>>> to achieve. >>>> >>> >>> When you say redundant, do you mean duplication between the AS/ >>> Hibernate test suite and JBC's test suite (+1 in this case), or >>> duplication between JBC's unit test suite and integration test >>> suite (-1 in this case)? >>> >> >> Meant both, although I was really talking about the latter. The >> idea here is these integration tests set up caches configured that >> match what AS/Hib uses and then exercise the call patterns that AS/ >> Hib makes. In a perfect world, those call patterns are exercised >> with those configuration options somewhere else in the JBC >> testsuite. But the world isn't perfect; there are too many >> permutations. >> >> Part of the tradeoff here is Paul and I will be working to expand >> the JBC suite and covering code paths not otherwise covered. But >> when we make this effort we just think "here's the config, here's >> the call pattern." We don't spend time digging through all the >> hundreds of tests in the main testsuite looking for redundancy. >> >> Of course, if one of these tests surfaces a bug, a test in the >> regular testsuite showing the bug should be added. >> -- Brian Stansberry Lead, AS Clustering JBoss, a division of Red Hat brian.stansberry(a)redhat.com