[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Scott Marlow]

If this reorganization doesn't slow us down in the creation of tests, 
then I don't mind the extra constraints.

On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>

What happens to the testsuite/compat stuff that I added for testing 
older versions of Hibernate?

Currently, on the problems list with "compat" is lack of support for 
testing custom AS7 modules being added to the AS7/modules folder by unit 
tests.  There should be a way to test custom modules with a unit test 
somehow.  It would be nice if this reorganization, added support for 
that.  ;)


> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build

I would like to see the ability to run with different persistence 
providers as well.  So that I can run the current JPA tests against 
Hibernate 3 or OGM or something else.

>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.

Currently, we have a few (five?) AS7 committers, they review everything. 
  Are you proposing separation of the AS7 committer task from the 
reviewing of changes?  Or adding more specialized AS7 committers that 
only handle certain source modules in AS7?

>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev