[jbosscache-dev] AS Integration tests in JBC test suites

[Manik Surtani]

Ok, let's do this in the main test set for now, and use TestNG's  
grouping to separate.

E.g., put the integration tests in src/test/java, in a new package  
(org.jboss.cache.integration ?) and annotate tests as being in the  
"integration" group.  They can be run by passing in the necessary  
parameters to mvn according to README-Maven.txt [1] (see section 2.3)  
and we can set up a separate Hudson run to run these integration tests.

- Manik

[1] http://anonsvn.jboss.org/repos/jbosscache/core/tags/2.1.0.GA/README-Maven.txt

On 24 Mar 2008, at 18:51, Brian Stansberry wrote:
> Mircea Markus wrote:
>> Brian Stansberry wrote:
>>> Manik Surtani wrote:
>>>> +1.
>>>>
>>>> How do you foresee this, as a separate suite of tests?  E.g., src/ 
>>>> integration-test/java?  And how about integration with Hudson?   
>>>> Run on every build?  Every release?
>>>>
>>>
>>> Hadn't really thought that far. :-)
>>>
>>> I *was* thinking in terms of just a separate package (with  
>>> subpackages) under src/test/java. Say org.jboss.cache.integration  
>>> or something. But, yeah, there's no need to run these every build,  
>>> just before release, so lets organize in whatever way makes it  
>>> easy to control when they get run.
>>>
>> Do we expect these tests to perform slow?
>
> Maybe a bit. The general AS use cases involve things like failover  
> after  async repl, so you need to pause a bit to be sure the repl  
> had time to happen. Also passivation, where there is some pausing to  
> allow passivation timeouts to happen. Stuff like that.
>
>> If not, I guess running them together with the common test suite  
>> would only bring an additional safety net... I also think that it  
>> worths having them in an different package as they will mainly be  
>> written by the teams that needs integration (Brian I think :) )
>
> For sure we want a distinct package. :)  Gotta ensure we cover all  
> the use cases; if tests are spread in random places it's too hard to  
> track whether we did that or not.
>
>>> Re: hudson, since we don't want them run on every build,  IMO it  
>>> would be good to have a separate, unscheduled, hudson job to run  
>>> w/ them included.  They need to run before release, but we should  
>>> also make it easy for a dev to kick off a run that includes them  
>>> so problems can be caught early.
>>>
>>>> On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
>>>>> I'd like to propose adding a package structure in the core cache  
>>>>> and pojo cache testsuites for adding tests of compliance with  
>>>>> the behavior the AS and Hibernate expect from JBC. Basically, in  
>>>>> these packages the AS developers would write tests that simulate  
>>>>> the AS and Hibernate usage of JBC. Simulations would *not* add  
>>>>> any new dependencies.  Goal over time is to 99.5% eliminate the  
>>>>> chances of finding JBC-caused regressions in the AS/Hibernate  
>>>>> testsuites by ensuring they are caught in JBC itself.
>>>>>
>>>>> Right now a lot of issues slip through because the JBC test  
>>>>> suites have a hard time covering all of its permutations of  
>>>>> features.  Something gets changed and a test gets added, but it  
>>>>> doesn't cover the configuration set AS/Hibernate uses, so we get  
>>>>> a regression at the AS level.
>>>>>
>>>>> Downside to this is theoretically everything we add in these new  
>>>>> packages should be redundant. But in practice that will be hard  
>>>>> to achieve.
>>>>>
>>>>
>>>> When you say redundant, do you mean duplication between the AS/ 
>>>> Hibernate test suite and JBC's test suite (+1 in this case), or  
>>>> duplication between JBC's unit test suite and integration test  
>>>> suite (-1 in this case)?
>>>>
>>>
>>> Meant both, although I was really talking about the latter. The  
>>> idea here is these integration tests set up caches configured that  
>>> match what AS/Hib uses and then exercise the call patterns that AS/ 
>>> Hib makes.  In a perfect world, those call patterns are exercised  
>>> with those configuration options somewhere else in the JBC  
>>> testsuite.  But the world isn't perfect; there are too many  
>>> permutations.
>>>
>>> Part of the tradeoff here is Paul and I will be working to expand  
>>> the JBC suite and covering code paths not otherwise covered. But  
>>> when we make this effort we just think "here's the config, here's  
>>> the call pattern."  We don't spend time digging through all the  
>>> hundreds of tests in the main testsuite looking for redundancy.
>>>
>>> Of course, if one of these tests surfaces a bug, a test in the  
>>> regular testsuite showing the bug should be added.
>>>
>
> -- 
> Brian Stansberry
> Lead, AS Clustering
> JBoss, a division of Red Hat
> brian.stansberry at redhat.com

--
Manik Surtani
Lead, JBoss Cache
manik at jboss.org