[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Andrew Lee Rubinger]

I'll add this link, too:

http://jboss.hudson.alrubinger.com/job/AS7-999/1/

Just to show that my Big Ol Patch Which Moves Everything Around is 
working. :)

S,
ALR

On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>
> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build
>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.
>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev