12:26 p.m.
I'd like to propose adding a package structure in the core cache and
pojo cache testsuites for adding tests of compliance with the behavior
the AS and Hibernate expect from JBC. Basically, in these packages the
AS developers would write tests that simulate the AS and Hibernate usage
of JBC. Simulations would *not* add any new dependencies. Goal over
time is to 99.5% eliminate the chances of finding JBC-caused regressions
in the AS/Hibernate testsuites by ensuring they are caught in JBC itself.
Right now a lot of issues slip through because the JBC test suites have
a hard time covering all of its permutations of features. Something
gets changed and a test gets added, but it doesn't cover the
configuration set AS/Hibernate uses, so we get a regression at the AS level.
Downside to this is theoretically everything we add in these new
packages should be redundant. But in practice that will be hard to achieve.
Comments?
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
8:34 a.m.
+1.
How do you foresee this, as a separate suite of tests? E.g., src/
integration-test/java? And how about integration with Hudson? Run on
every build? Every release?
On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
I'd like to propose adding a package structure in the core cache
and
pojo cache testsuites for adding tests of compliance with the
behavior the AS and Hibernate expect from JBC. Basically, in these
packages the AS developers would write tests that simulate the AS
and Hibernate usage of JBC. Simulations would *not* add any new
dependencies. Goal over time is to 99.5% eliminate the chances of
finding JBC-caused regressions in the AS/Hibernate testsuites by
ensuring they are caught in JBC itself.
Right now a lot of issues slip through because the JBC test suites
have a hard time covering all of its permutations of features.
Something gets changed and a test gets added, but it doesn't cover
the configuration set AS/Hibernate uses, so we get a regression at
the AS level.
Downside to this is theoretically everything we add in these new
packages should be redundant. But in practice that will be hard to
achieve.
When you say redundant, do you mean duplication between the AS/
Hibernate test suite and JBC's test suite (+1 in this case), or
duplication between JBC's unit test suite and integration test suite
(-1 in this case)?
Cheers
Manik
--
Manik Surtani
Lead, JBoss Cache
manik(a)jboss.org
9:13 a.m.
Manik Surtani wrote:
+1.
How do you foresee this, as a separate suite of tests? E.g.,
src/integration-test/java? And how about integration with Hudson? Run
on every build? Every release?
Hadn't really thought that far. :-)
I *was* thinking in terms of just a separate package (with subpackages)
under src/test/java. Say org.jboss.cache.integration or something. But,
yeah, there's no need to run these every build, just before release, so
lets organize in whatever way makes it easy to control when they get run.
Re: hudson, since we don't want them run on every build, IMO it would
be good to have a separate, unscheduled, hudson job to run w/ them
included. They need to run before release, but we should also make it
easy for a dev to kick off a run that includes them so problems can be
caught early.
On 18 Mar 2008, at 17:26, Brian Stansberry wrote:
> I'd like to propose adding a package structure in the core cache and
> pojo cache testsuites for adding tests of compliance with the behavior
> the AS and Hibernate expect from JBC. Basically, in these packages the
> AS developers would write tests that simulate the AS and Hibernate
> usage of JBC. Simulations would *not* add any new dependencies. Goal
> over time is to 99.5% eliminate the chances of finding JBC-caused
> regressions in the AS/Hibernate testsuites by ensuring they are caught
> in JBC itself.
>
> Right now a lot of issues slip through because the JBC test suites
> have a hard time covering all of its permutations of features.
> Something gets changed and a test gets added, but it doesn't cover the
> configuration set AS/Hibernate uses, so we get a regression at the AS
> level.
>
> Downside to this is theoretically everything we add in these new
> packages should be redundant. But in practice that will be hard to
> achieve.
>
When you say redundant, do you mean duplication between the AS/Hibernate
test suite and JBC's test suite (+1 in this case), or duplication
between JBC's unit test suite and integration test suite (-1 in this case)?
Meant both, although I was really talking about the latter. The idea
here is these integration tests set up caches configured that match what
AS/Hib uses and then exercise the call patterns that AS/Hib makes. In a
perfect world, those call patterns are exercised with those
configuration options somewhere else in the JBC testsuite. But the
world isn't perfect; there are too many permutations.
Part of the tradeoff here is Paul and I will be working to expand the
JBC suite and covering code paths not otherwise covered. But when we
make this effort we just think "here's the config, here's the call
pattern." We don't spend time digging through all the hundreds of tests
in the main testsuite looking for redundancy.
Of course, if one of these tests surfaces a bug, a test in the regular
testsuite showing the bug should be added.
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
6092
days inactive
6093
days old
jbosscache-dev@lists.jboss.org
2 comments
2 participants
participants (2)
-
Brian Stansberry -
Manik Surtani