Re: [jbosscache-dev] AS Integration tests in JBC test suites
Mircea Markus wrote:
Brian Stansberry wrote:
> Manik Surtani wrote:
>> +1.
>>
>> How do you foresee this, as a separate suite of tests? E.g.,
>> src/integration-test/java? And how about integration with Hudson?
>> Run on every build? Every release?
>>
>
> Hadn't really thought that far. :-)
>
> I *was* thinking in terms of just a separate package (with
> subpackages) under src/test/java. Say org.jboss.cache.integration or
> something. But, yeah, there's no need to run these every build, just
> before release, so lets organize in whatever way makes it easy to
> control when they get run.
>
Do we expect these tests to perform slow?
Maybe a bit. The general AS use cases involve things like failover after
async repl, so you need to pause a bit to be sure the repl had time to
happen. Also passivation, where there is some pausing to allow
passivation timeouts to happen. Stuff like that.
If not, I guess running them
together with the common test suite would only bring an additional
safety net... I also think that it worths having them in an different
package as they will mainly be written by the teams that needs
integration (Brian I think :) )
For sure we want a distinct package. :) Gotta ensure we cover all the
use cases; if tests are spread in random places it's too hard to track
whether we did that or not.
> Re: hudson, since we don't want them run on every build, IMO
it would
> be good to have a separate, unscheduled, hudson job to run w/ them
> included. They need to run before release, but we should also make it
> easy for a dev to kick off a run that includes them so problems can be
> caught early.
>
>> On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
>>> I'd like to propose adding a package structure in the core cache and
>>> pojo cache testsuites for adding tests of compliance with the
>>> behavior the AS and Hibernate expect from JBC. Basically, in these
>>> packages the AS developers would write tests that simulate the AS
>>> and Hibernate usage of JBC. Simulations would *not* add any new
>>> dependencies. Goal over time is to 99.5% eliminate the chances of
>>> finding JBC-caused regressions in the AS/Hibernate testsuites by
>>> ensuring they are caught in JBC itself.
>>>
>>> Right now a lot of issues slip through because the JBC test suites
>>> have a hard time covering all of its permutations of features.
>>> Something gets changed and a test gets added, but it doesn't cover
>>> the configuration set AS/Hibernate uses, so we get a regression at
>>> the AS level.
>>>
>>> Downside to this is theoretically everything we add in these new
>>> packages should be redundant. But in practice that will be hard to
>>> achieve.
>>>
>>
>> When you say redundant, do you mean duplication between the
>> AS/Hibernate test suite and JBC's test suite (+1 in this case), or
>> duplication between JBC's unit test suite and integration test suite
>> (-1 in this case)?
>>
>
> Meant both, although I was really talking about the latter. The idea
> here is these integration tests set up caches configured that match
> what AS/Hib uses and then exercise the call patterns that AS/Hib
> makes. In a perfect world, those call patterns are exercised with
> those configuration options somewhere else in the JBC testsuite. But
> the world isn't perfect; there are too many permutations.
>
> Part of the tradeoff here is Paul and I will be working to expand the
> JBC suite and covering code paths not otherwise covered. But when we
> make this effort we just think "here's the config, here's the call
> pattern." We don't spend time digging through all the hundreds of
> tests in the main testsuite looking for redundancy.
>
> Of course, if one of these tests surfaces a bug, a test in the regular
> testsuite showing the bug should be added.
>
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com