[jbosscache-dev] AS Integration tests in JBC test suites

[Brian Stansberry]

Mircea Markus wrote:
> 
> 
> Brian Stansberry wrote:
>> Manik Surtani wrote:
>>> +1.
>>>
>>> How do you foresee this, as a separate suite of tests?  E.g., 
>>> src/integration-test/java?  And how about integration with Hudson?  
>>> Run on every build?  Every release?
>>>
>>
>> Hadn't really thought that far. :-)
>>
>> I *was* thinking in terms of just a separate package (with 
>> subpackages) under src/test/java. Say org.jboss.cache.integration or 
>> something. But, yeah, there's no need to run these every build, just 
>> before release, so lets organize in whatever way makes it easy to 
>> control when they get run.
>>
> Do we expect these tests to perform slow? 

Maybe a bit. The general AS use cases involve things like failover after 
  async repl, so you need to pause a bit to be sure the repl had time to 
happen. Also passivation, where there is some pausing to allow 
passivation timeouts to happen. Stuff like that.

> If not, I guess running them 
> together with the common test suite would only bring an additional 
> safety net... I also think that it worths having them in an different 
> package as they will mainly be written by the teams that needs 
> integration (Brian I think :) )

For sure we want a distinct package. :)  Gotta ensure we cover all the 
use cases; if tests are spread in random places it's too hard to track 
whether we did that or not.

>> Re: hudson, since we don't want them run on every build,  IMO it would 
>> be good to have a separate, unscheduled, hudson job to run w/ them 
>> included.  They need to run before release, but we should also make it 
>> easy for a dev to kick off a run that includes them so problems can be 
>> caught early.
>>
>>> On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
>>>> I'd like to propose adding a package structure in the core cache and 
>>>> pojo cache testsuites for adding tests of compliance with the 
>>>> behavior the AS and Hibernate expect from JBC. Basically, in these 
>>>> packages the AS developers would write tests that simulate the AS 
>>>> and Hibernate usage of JBC. Simulations would *not* add any new 
>>>> dependencies.  Goal over time is to 99.5% eliminate the chances of 
>>>> finding JBC-caused regressions in the AS/Hibernate testsuites by 
>>>> ensuring they are caught in JBC itself.
>>>>
>>>> Right now a lot of issues slip through because the JBC test suites 
>>>> have a hard time covering all of its permutations of features.  
>>>> Something gets changed and a test gets added, but it doesn't cover 
>>>> the configuration set AS/Hibernate uses, so we get a regression at 
>>>> the AS level.
>>>>
>>>> Downside to this is theoretically everything we add in these new 
>>>> packages should be redundant. But in practice that will be hard to 
>>>> achieve.
>>>>
>>>
>>> When you say redundant, do you mean duplication between the 
>>> AS/Hibernate test suite and JBC's test suite (+1 in this case), or 
>>> duplication between JBC's unit test suite and integration test suite 
>>> (-1 in this case)?
>>>
>>
>> Meant both, although I was really talking about the latter. The idea 
>> here is these integration tests set up caches configured that match 
>> what AS/Hib uses and then exercise the call patterns that AS/Hib 
>> makes.  In a perfect world, those call patterns are exercised with 
>> those configuration options somewhere else in the JBC testsuite.  But 
>> the world isn't perfect; there are too many permutations.
>>
>> Part of the tradeoff here is Paul and I will be working to expand the 
>> JBC suite and covering code paths not otherwise covered. But when we 
>> make this effort we just think "here's the config, here's the call 
>> pattern."  We don't spend time digging through all the hundreds of 
>> tests in the main testsuite looking for redundancy.
>>
>> Of course, if one of these tests surfaces a bug, a test in the regular 
>> testsuite showing the bug should be added.
>>

-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry at redhat.com