Re: [jbosscache-dev] AS Integration tests in JBC test suites
Manik Surtani wrote:
+1.
How do you foresee this, as a separate suite of tests? E.g.,
src/integration-test/java? And how about integration with Hudson? Run
on every build? Every release?
Hadn't really thought that far. :-)
I *was* thinking in terms of just a separate package (with subpackages)
under src/test/java. Say org.jboss.cache.integration or something. But,
yeah, there's no need to run these every build, just before release, so
lets organize in whatever way makes it easy to control when they get run.
Re: hudson, since we don't want them run on every build, IMO it would
be good to have a separate, unscheduled, hudson job to run w/ them
included. They need to run before release, but we should also make it
easy for a dev to kick off a run that includes them so problems can be
caught early.
On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
> I'd like to propose adding a package structure in the core cache and
> pojo cache testsuites for adding tests of compliance with the behavior
> the AS and Hibernate expect from JBC. Basically, in these packages the
> AS developers would write tests that simulate the AS and Hibernate
> usage of JBC. Simulations would *not* add any new dependencies. Goal
> over time is to 99.5% eliminate the chances of finding JBC-caused
> regressions in the AS/Hibernate testsuites by ensuring they are caught
> in JBC itself.
>
> Right now a lot of issues slip through because the JBC test suites
> have a hard time covering all of its permutations of features.
> Something gets changed and a test gets added, but it doesn't cover the
> configuration set AS/Hibernate uses, so we get a regression at the AS
> level.
>
> Downside to this is theoretically everything we add in these new
> packages should be redundant. But in practice that will be hard to
> achieve.
>
When you say redundant, do you mean duplication between the AS/Hibernate
test suite and JBC's test suite (+1 in this case), or duplication
between JBC's unit test suite and integration test suite (-1 in this case)?
Meant both, although I was really talking about the latter. The idea
here is these integration tests set up caches configured that match what
AS/Hib uses and then exercise the call patterns that AS/Hib makes. In a
perfect world, those call patterns are exercised with those
configuration options somewhere else in the JBC testsuite. But the
world isn't perfect; there are too many permutations.
Part of the tradeoff here is Paul and I will be working to expand the
JBC suite and covering code paths not otherwise covered. But when we
make this effort we just think "here's the config, here's the call
pattern." We don't spend time digging through all the hundreds of tests
in the main testsuite looking for redundancy.
Of course, if one of these tests surfaces a bug, a test in the regular
testsuite showing the bug should be added.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com