[jbosscache-dev] AS Integration tests in JBC test suites

[Brian Stansberry]

Manik Surtani wrote:
> +1.
> 
> How do you foresee this, as a separate suite of tests?  E.g., 
> src/integration-test/java?  And how about integration with Hudson?  Run 
> on every build?  Every release?
> 

Hadn't really thought that far. :-)

I *was* thinking in terms of just a separate package (with subpackages) 
under src/test/java. Say org.jboss.cache.integration or something. But, 
yeah, there's no need to run these every build, just before release, so 
lets organize in whatever way makes it easy to control when they get run.

Re: hudson, since we don't want them run on every build,  IMO it would 
be good to have a separate, unscheduled, hudson job to run w/ them 
included.  They need to run before release, but we should also make it 
easy for a dev to kick off a run that includes them so problems can be 
caught early.

> On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
>> I'd like to propose adding a package structure in the core cache and 
>> pojo cache testsuites for adding tests of compliance with the behavior 
>> the AS and Hibernate expect from JBC. Basically, in these packages the 
>> AS developers would write tests that simulate the AS and Hibernate 
>> usage of JBC. Simulations would *not* add any new dependencies.  Goal 
>> over time is to 99.5% eliminate the chances of finding JBC-caused 
>> regressions in the AS/Hibernate testsuites by ensuring they are caught 
>> in JBC itself.
>>
>> Right now a lot of issues slip through because the JBC test suites 
>> have a hard time covering all of its permutations of features.  
>> Something gets changed and a test gets added, but it doesn't cover the 
>> configuration set AS/Hibernate uses, so we get a regression at the AS 
>> level.
>>
>> Downside to this is theoretically everything we add in these new 
>> packages should be redundant. But in practice that will be hard to 
>> achieve.
>>
> 
> When you say redundant, do you mean duplication between the AS/Hibernate 
> test suite and JBC's test suite (+1 in this case), or duplication 
> between JBC's unit test suite and integration test suite (-1 in this case)?
> 

Meant both, although I was really talking about the latter. The idea 
here is these integration tests set up caches configured that match what 
AS/Hib uses and then exercise the call patterns that AS/Hib makes.  In a 
perfect world, those call patterns are exercised with those 
configuration options somewhere else in the JBC testsuite.  But the 
world isn't perfect; there are too many permutations.

Part of the tradeoff here is Paul and I will be working to expand the 
JBC suite and covering code paths not otherwise covered. But when we 
make this effort we just think "here's the config, here's the call 
pattern."  We don't spend time digging through all the hundreds of tests 
in the main testsuite looking for redundancy.

Of course, if one of these tests surfaces a bug, a test in the regular 
testsuite showing the bug should be added.

-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry at redhat.com