[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Andrew Lee Rubinger]

A summary of where I believe we stand on the 3 points raised here:

1) TestSuite Organization

Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.

While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks.  By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete.  In short, paying attention to user dependencies to validate against them is very important.

2) Run Modes, Test Subsets

I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout.  For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.

3) An authoritative maintainer

No one has commented on this.

By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed.  Failing that, we need to agree on an alternate path forward.  I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined.

S,
ALR

----- Original Message -----
From: "Andrew Lee Rubinger" <andrew.rubinger at redhat.com>
To: "JBoss AS7 Development" <jboss-as7-dev at lists.jboss.org>
Sent: Thursday, August 11, 2011 7:38:51 AM
Subject: [jboss-as7-dev] Revisited: Integration TestSuite Organization and	Maintenance

Hi guys:

I'd like to reopen the discussion regarding the testsuite organization 
and its ongoing maintenance.  This issue dates back a few months with 
some debates and differing opinions, so I'll do my best to outline the 
guiding principles I'd like to see put in place concisely.

To start off, I've a Proof-of-Concept for many of the following points 
now located:

https://github.com/ALRubinger/jboss-as/tree/AS7-999

The relevant JIRA I've been using to track things:

https://issues.jboss.org/browse/AS7-999

So:

1) TestSuite Organization

I believe we need a single top-level categorization by which we may 
organize integration tests which are deployment-based and run within the 
context of the server.  Because we use Maven modules (which are bound to 
a dependency structure), it makes sense to file these modules by the 
compile-time dependencies they require.  So in place I've put:

testsuite/spec - Java SE and Java EE APIs only
testsuite/api - AS7 APIs + Spec
testsuite/internals - Use anything in the AS7 runtime in your 
deployments; not guaranteed to be back-compat across releases

The primary motivation here is to ensure that the dependencies we export 
(ie. "spec-api", and "api" modules) are complete enough for users to 
create their own deployments.  In this setup, we act as *users* of our 
own APIs, and everything in src/main is limited to the relevant 
dependencies.

I know the source of some disagreements earlier centered around placing 
the tests right next to the deployments, and some folks consider the 
deployments as part of the test itself.  That's not a bad argument at 
all, but again consider that we then lose the ability to validate our 
tests in the context of our exported APIs.

2) Run Modes, Test Subsets

Because the primary organizational criteria proposed in 1) is by 
dependency, these modules will grow large over time.  The AS build over 
time will take longer and longer to run.  Additionally, there are 
runtime options to consider when starting tests.  So consider the 
following requirements:

   * Running the testsuite in IPv6
   * Running only a subset of tests as part of the main build

These lend themselves well to using build profiles.  By default, I think 
the "smoke tests" should simply be a set of tests we deem important or 
indicative of the general health of AS7 with respect to each subsystem. 
  As it stands now, "smoke" is its own module with a bunch of 
Embedded-based tests, and I think these should move to the 
organizational structure in 1) and instead we can apply some filtering 
to make the "smoke" some default set of includes.

3) An authoritative maintainer

I'd like to treat the Arquillian and TestSuite modules as true 
subsystems of the Application Server, and as such we'll need someone to 
assume the responsibility to review incoming commits/pull requests and 
ensure they fit the criteria for acceptance.  Simple things like 
consistent package names, using ARQ correctly, and not leaking 
dependencies are very important.

So assuming we come to agreement on these points, I'd like to request 
push access to the AS7 repo to field testsuite and ARQ-related pull 
requests.

...there's much more to discuss (I've more issues to raise alongside the 
upcoming EAP requirements), but let's start with those first 3 major 
points and my POC, and run from there.

S,
ALR
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev