[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

[Andrew Lee Rubinger]

Inline.

On 08/11/2011 07:06 PM, Andrew Lee Rubinger wrote:
> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
>> Andrew
>>
>> Can you give a simple, concrete example of a spec test case (citing part
>> of the J2EE spec, say), which illustrates:
>
> Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly
> necessarily citing the EE specs, because we're *not* building a TCK in AS7.

My mistake, I'd meant to throw a link in here as Good Example:

src/main: 
https://github.com/ALRubinger/jboss-as/tree/AS7-999/testsuite/spec/src/main/java/org/jboss/as/test/spec/web/ejb
src/test: 
https://github.com/ALRubinger/jboss-as/blob/AS7-999/testsuite/spec/src/test/java/org/jboss/as/test/spec/web/ejb/ServletInjectionTestCase.java

S,
ALR


>
>> - what dependencies you want to be on the classpath, as well as those
>> you don't want to be only the classpath
>
> For this discussion, I'm primarily concerned with the *compilation*
> ClassPath, as this is the view users see when building their
> applications.  It's about explicitly defining what the AS7 API is.  So
> the deps for the compilation ClassPath should be as they are in by
> AS7-999 branch and as described in the last email:
>
> * spec - Java SE and Java EE
> * api - JBoss-specific extensions to "spec" which comprise our AS7 API
> * internals - Anything in the AS7 runtime
>
>> - what sample artifacts you would put in src/main and what you would put
>> in src/test
>
> * src/main - Stuff that is part of the deployment.  Anything that you'd
> be including in the @Deployment archive defined by an Arquillian-based
> test case.
> * src/test - The test infrastructure itself, like the JUnit test and
> other util/helpers
>
>> - what compile and test execution time constraints you would want to enforce
>> This would help me to see the motivation for the refactoring and your
>> planned organization of artifacts in src/main and src/test, which
>> differ from the standard organization of putting all test related
>> artifacts in src/test.
>
> Test execution time is an orthogonal concern, and relates instead back
> to using "smoke" tests to run by default instead of the entire AS7
> integration test suite (which won't scale to run on every build over
> time).  IMO it's already taking too long for standard builds.
>
> Again, the motivation for moving/organizing in this fashion is to honor
> dependencies and assert that the APIs we export ("api" and "spec-api"
> modules) are complete.  For instance, while moving tests around I
> discovered that our POMs were incompete:
>
> https://issues.jboss.org/browse/AS7-1489
> https://issues.jboss.org/browse/AS7-1493
> https://issues.jboss.org/browse/AS7-1479
> https://issues.jboss.org/browse/AS7-1478
>
>> Also, is it not possible to control what is on the classpath by maven
>> elements like<classpathDependencyExcludes/>   and the like?
>
> That's for test runtime.  Because Arquillian is starting (or connecting
> to) the container in a remote process, we're less concerned with the
> client runtime ClassPath, so long as it's enough for Arquillian to do
> its thing.
>
> S,
> ALR
>
>>
>> Richard
>>
>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>>> Hi guys:
>>>
>>> I'd like to reopen the discussion regarding the testsuite organization
>>> and its ongoing maintenance.  This issue dates back a few months with
>>> some debates and differing opinions, so I'll do my best to outline the
>>> guiding principles I'd like to see put in place concisely.
>>>
>>> To start off, I've a Proof-of-Concept for many of the following points
>>> now located:
>>>
>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>>
>>> The relevant JIRA I've been using to track things:
>>>
>>> https://issues.jboss.org/browse/AS7-999
>>>
>>> So:
>>>
>>> 1) TestSuite Organization
>>>
>>> I believe we need a single top-level categorization by which we may
>>> organize integration tests which are deployment-based and run within the
>>> context of the server.  Because we use Maven modules (which are bound to
>>> a dependency structure), it makes sense to file these modules by the
>>> compile-time dependencies they require.  So in place I've put:
>>>
>>> testsuite/spec - Java SE and Java EE APIs only
>>> testsuite/api - AS7 APIs + Spec
>>> testsuite/internals - Use anything in the AS7 runtime in your
>>> deployments; not guaranteed to be back-compat across releases
>>>
>>> The primary motivation here is to ensure that the dependencies we export
>>> (ie. "spec-api", and "api" modules) are complete enough for users to
>>> create their own deployments.  In this setup, we act as *users* of our
>>> own APIs, and everything in src/main is limited to the relevant
>>> dependencies.
>>>
>>> I know the source of some disagreements earlier centered around placing
>>> the tests right next to the deployments, and some folks consider the
>>> deployments as part of the test itself.  That's not a bad argument at
>>> all, but again consider that we then lose the ability to validate our
>>> tests in the context of our exported APIs.
>>>
>>> 2) Run Modes, Test Subsets
>>>
>>> Because the primary organizational criteria proposed in 1) is by
>>> dependency, these modules will grow large over time.  The AS build over
>>> time will take longer and longer to run.  Additionally, there are
>>> runtime options to consider when starting tests.  So consider the
>>> following requirements:
>>>
>>>       * Running the testsuite in IPv6
>>>       * Running only a subset of tests as part of the main build
>>>
>>> These lend themselves well to using build profiles.  By default, I think
>>> the "smoke tests" should simply be a set of tests we deem important or
>>> indicative of the general health of AS7 with respect to each subsystem.
>>>      As it stands now, "smoke" is its own module with a bunch of
>>> Embedded-based tests, and I think these should move to the
>>> organizational structure in 1) and instead we can apply some filtering
>>> to make the "smoke" some default set of includes.
>>>
>>> 3) An authoritative maintainer
>>>
>>> I'd like to treat the Arquillian and TestSuite modules as true
>>> subsystems of the Application Server, and as such we'll need someone to
>>> assume the responsibility to review incoming commits/pull requests and
>>> ensure they fit the criteria for acceptance.  Simple things like
>>> consistent package names, using ARQ correctly, and not leaking
>>> dependencies are very important.
>>>
>>> So assuming we come to agreement on these points, I'd like to request
>>> push access to the AS7 repo to field testsuite and ARQ-related pull
>>> requests.
>>>
>>> ...there's much more to discuss (I've more issues to raise alongside the
>>> upcoming EAP requirements), but let's start with those first 3 major
>>> points and my POC, and run from there.
>>>
>>> S,
>>> ALR
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev