[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Andrew Lee Rubinger]

On 08/11/2011 09:50 AM, Scott Marlow wrote:
> If this reorganization doesn't slow us down in the creation of tests,
> then I don't mind the extra constraints.

Can't see why it would, really.  The biggest difference would simply be 
in separating out deployable components (src/main) from the test itself 
(src/test).

S,
ALR

>
> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>> Hi guys:
>>
>> I'd like to reopen the discussion regarding the testsuite organization
>> and its ongoing maintenance.  This issue dates back a few months with
>> some debates and differing opinions, so I'll do my best to outline the
>> guiding principles I'd like to see put in place concisely.
>>
>> To start off, I've a Proof-of-Concept for many of the following points
>> now located:
>>
>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>
>> The relevant JIRA I've been using to track things:
>>
>> https://issues.jboss.org/browse/AS7-999
>>
>> So:
>>
>> 1) TestSuite Organization
>>
>> I believe we need a single top-level categorization by which we may
>> organize integration tests which are deployment-based and run within the
>> context of the server.  Because we use Maven modules (which are bound to
>> a dependency structure), it makes sense to file these modules by the
>> compile-time dependencies they require.  So in place I've put:
>>
>> testsuite/spec - Java SE and Java EE APIs only
>> testsuite/api - AS7 APIs + Spec
>> testsuite/internals - Use anything in the AS7 runtime in your
>> deployments; not guaranteed to be back-compat across releases
>>
>
> What happens to the testsuite/compat stuff that I added for testing
> older versions of Hibernate?
>
> Currently, on the problems list with "compat" is lack of support for
> testing custom AS7 modules being added to the AS7/modules folder by unit
> tests.  There should be a way to test custom modules with a unit test
> somehow.  It would be nice if this reorganization, added support for
> that.  ;)
>
>
>> The primary motivation here is to ensure that the dependencies we export
>> (ie. "spec-api", and "api" modules) are complete enough for users to
>> create their own deployments.  In this setup, we act as *users* of our
>> own APIs, and everything in src/main is limited to the relevant
>> dependencies.
>>
>> I know the source of some disagreements earlier centered around placing
>> the tests right next to the deployments, and some folks consider the
>> deployments as part of the test itself.  That's not a bad argument at
>> all, but again consider that we then lose the ability to validate our
>> tests in the context of our exported APIs.
>>
>> 2) Run Modes, Test Subsets
>>
>> Because the primary organizational criteria proposed in 1) is by
>> dependency, these modules will grow large over time.  The AS build over
>> time will take longer and longer to run.  Additionally, there are
>> runtime options to consider when starting tests.  So consider the
>> following requirements:
>>
>>      * Running the testsuite in IPv6
>>      * Running only a subset of tests as part of the main build
>
> I would like to see the ability to run with different persistence
> providers as well.  So that I can run the current JPA tests against
> Hibernate 3 or OGM or something else.
>
>>
>> These lend themselves well to using build profiles.  By default, I think
>> the "smoke tests" should simply be a set of tests we deem important or
>> indicative of the general health of AS7 with respect to each subsystem.
>>     As it stands now, "smoke" is its own module with a bunch of
>> Embedded-based tests, and I think these should move to the
>> organizational structure in 1) and instead we can apply some filtering
>> to make the "smoke" some default set of includes.
>>
>> 3) An authoritative maintainer
>>
>> I'd like to treat the Arquillian and TestSuite modules as true
>> subsystems of the Application Server, and as such we'll need someone to
>> assume the responsibility to review incoming commits/pull requests and
>> ensure they fit the criteria for acceptance.  Simple things like
>> consistent package names, using ARQ correctly, and not leaking
>> dependencies are very important.
>
> Currently, we have a few (five?) AS7 committers, they review everything.
>    Are you proposing separation of the AS7 committer task from the
> reviewing of changes?  Or adding more specialized AS7 committers that
> only handle certain source modules in AS7?
>
>>
>> So assuming we come to agreement on these points, I'd like to request
>> push access to the AS7 repo to field testsuite and ARQ-related pull
>> requests.
>>
>> ...there's much more to discuss (I've more issues to raise alongside the
>> upcoming EAP requirements), but let's start with those first 3 major
>> points and my POC, and run from there.
>>
>> S,
>> ALR
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev