[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

Andrew Lee Rubinger arubinge at redhat.com
Fri Aug 12 06:59:14 EDT 2011 

Inline.

----- Original Message -----
From: Kabir Khan <kabir.khan at jboss.com>
To: Stuart Douglas <stuart.w.douglas at gmail.com>
Cc: Andrew Lee Rubinger <andrew.rubinger at redhat.com>, jboss-as7-dev at lists.jboss.org
Sent: Fri, 12 Aug 2011 05:14:25 -0400 (EDT)
Subject: Re: [jboss-as7-dev] Revisited: Integration TestSuite Organization	and Maintenance


On 12 Aug 2011, at 04:28, Stuart Douglas wrote:

> 
> On 12/08/2011, at 9:06 AM, Andrew Lee Rubinger wrote:
> 
>> Inline.
>> 
>> On 08/11/2011 05:12 PM, Richard Achmatowicz wrote:
>>> Andrew
>>> 
>>> Can you give a simple, concrete example of a spec test case (citing part
>>> of the J2EE spec, say), which illustrates:
>> 
>> Hehe, my PoC is filled with 'em. :)  But keep in mind we're not directly 
>> necessarily citing the EE specs, because we're *not* building a TCK in AS7.
>> 
>>> - what dependencies you want to be on the classpath, as well as those
>>> you don't want to be only the classpath
>> 
>> For this discussion, I'm primarily concerned with the *compilation* 
>> ClassPath, as this is the view users see when building their 
>> applications.  It's about explicitly defining what the AS7 API is.  So 
>> the deps for the compilation ClassPath should be as they are in by 
>> AS7-999 branch and as described in the last email:
>> 
>> * spec - Java SE and Java EE
>> * api - JBoss-specific extensions to "spec" which comprise our AS7 API
>> * internals - Anything in the AS7 runtime
> 
> I am ok with splitting the tests up in this manner, but I think we will need additional test suite modules for purely technical reasons. 
> 
> For instance, at the moment we also have compat for hibernate 3 testing, and timer service, to test the ejb timer service.
> 
> Compat is needed because maven cannot handle having multiple hibernate versions in the same module, and timer service 
> is separate because it needs the preview configuration, and need to run multiple times in order to test restoring persistent timers.
> 
>> 
>>> - what sample artifacts you would put in src/main and what you would put
>>> in src/test
>> 
>> * src/main - Stuff that is part of the deployment.  Anything that you'd 
>> be including in the @Deployment archive defined by an Arquillian-based 
>> test case.
>> * src/test - The test infrastructure itself, like the JUnit test and 
>> other util/helpers
> 
> I am very much opposed to this split. IMHO splitting the test classes into two different places makes it much harder to read and write tests. 
> 
> I also don't see how having additional test scoped dependencies on the class path is a problem, as in practice this works out to being Arquillian and JUnit. 

> I agree here, I don't like having tests split up over two source folders

Sure, it'd be nice to have everything right next to each other; I'll give you that.  But I don't see how we can do that without opening ourselves up to miss errors in dependency management.

Having additional dependencies on the compilation classpath is a very big problem, Stuart.  Take for instance Arquillian in src/test.  You need not only the Arquillian API, but also all the transitive deps needed to fire up the ARQ runtime.  To get a sense of what would leak in, just crack open the "arquillian/container-managed/pom.xml" in Eclipse and switch over to the "Dependency Hierarchy" tab.  By unifying src/main and src/test, everything listed there would leak onto the compilation classpath, killing our ability to validate the "spec-api" and "api" POMs are doing their job.  We'd be checking against a polluted ClassPath.

So yeah, putting things in 2 places is a bit less convenient, but let's be honest - it's not that bad.  Same package names in different source folders makes it really easy to see what's going on.  And taking a more strict approach lets us ensure things are working *correctly*, which is the most important thing.

S,
ALR

> 
> Stuart
> 
>> 
>>> - what compile and test execution time constraints you would want to enforce
>>> This would help me to see the motivation for the refactoring and your
>>> planned organization of artifacts in src/main and src/test, which
>>> differ from the standard organization of putting all test related
>>> artifacts in src/test.
>> 
>> Test execution time is an orthogonal concern, and relates instead back 
>> to using "smoke" tests to run by default instead of the entire AS7 
>> integration test suite (which won't scale to run on every build over 
>> time).  IMO it's already taking too long for standard builds.
>> 
>> Again, the motivation for moving/organizing in this fashion is to honor 
>> dependencies and assert that the APIs we export ("api" and "spec-api" 
>> modules) are complete.  For instance, while moving tests around I 
>> discovered that our POMs were incompete:
>> 
>> https://issues.jboss.org/browse/AS7-1489
>> https://issues.jboss.org/browse/AS7-1493
>> https://issues.jboss.org/browse/AS7-1479
>> https://issues.jboss.org/browse/AS7-1478
>> 
>>> Also, is it not possible to control what is on the classpath by maven
>>> elements like<classpathDependencyExcludes/>  and the like?
>> 
>> That's for test runtime.  Because Arquillian is starting (or connecting 
>> to) the container in a remote process, we're less concerned with the 
>> client runtime ClassPath, so long as it's enough for Arquillian to do 
>> its thing.
>> 
>> S,
>> ALR
>> 
>>> 
>>> Richard
>>> 
>>> On 08/11/2011 07:38 AM, Andrew Lee Rubinger wrote:
>>>> Hi guys:
>>>> 
>>>> I'd like to reopen the discussion regarding the testsuite organization
>>>> and its ongoing maintenance.  This issue dates back a few months with
>>>> some debates and differing opinions, so I'll do my best to outline the
>>>> guiding principles I'd like to see put in place concisely.
>>>> 
>>>> To start off, I've a Proof-of-Concept for many of the following points
>>>> now located:
>>>> 
>>>> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>>>> 
>>>> The relevant JIRA I've been using to track things:
>>>> 
>>>> https://issues.jboss.org/browse/AS7-999
>>>> 
>>>> So:
>>>> 
>>>> 1) TestSuite Organization
>>>> 
>>>> I believe we need a single top-level categorization by which we may
>>>> organize integration tests which are deployment-based and run within the
>>>> context of the server.  Because we use Maven modules (which are bound to
>>>> a dependency structure), it makes sense to file these modules by the
>>>> compile-time dependencies they require.  So in place I've put:
>>>> 
>>>> testsuite/spec - Java SE and Java EE APIs only
>>>> testsuite/api - AS7 APIs + Spec
>>>> testsuite/internals - Use anything in the AS7 runtime in your
>>>> deployments; not guaranteed to be back-compat across releases
>>>> 
>>>> The primary motivation here is to ensure that the dependencies we export
>>>> (ie. "spec-api", and "api" modules) are complete enough for users to
>>>> create their own deployments.  In this setup, we act as *users* of our
>>>> own APIs, and everything in src/main is limited to the relevant
>>>> dependencies.
>>>> 
>>>> I know the source of some disagreements earlier centered around placing
>>>> the tests right next to the deployments, and some folks consider the
>>>> deployments as part of the test itself.  That's not a bad argument at
>>>> all, but again consider that we then lose the ability to validate our
>>>> tests in the context of our exported APIs.
>>>> 
>>>> 2) Run Modes, Test Subsets
>>>> 
>>>> Because the primary organizational criteria proposed in 1) is by
>>>> dependency, these modules will grow large over time.  The AS build over
>>>> time will take longer and longer to run.  Additionally, there are
>>>> runtime options to consider when starting tests.  So consider the
>>>> following requirements:
>>>> 
>>>>     * Running the testsuite in IPv6
>>>>     * Running only a subset of tests as part of the main build
>>>> 
>>>> These lend themselves well to using build profiles.  By default, I think
>>>> the "smoke tests" should simply be a set of tests we deem important or
>>>> indicative of the general health of AS7 with respect to each subsystem.
>>>>    As it stands now, "smoke" is its own module with a bunch of
>>>> Embedded-based tests, and I think these should move to the
>>>> organizational structure in 1) and instead we can apply some filtering
>>>> to make the "smoke" some default set of includes.
>>>> 
>>>> 3) An authoritative maintainer
>>>> 
>>>> I'd like to treat the Arquillian and TestSuite modules as true
>>>> subsystems of the Application Server, and as such we'll need someone to
>>>> assume the responsibility to review incoming commits/pull requests and
>>>> ensure they fit the criteria for acceptance.  Simple things like
>>>> consistent package names, using ARQ correctly, and not leaking
>>>> dependencies are very important.
>>>> 
>>>> So assuming we come to agreement on these points, I'd like to request
>>>> push access to the AS7 repo to field testsuite and ARQ-related pull
>>>> requests.
>>>> 
>>>> ...there's much more to discuss (I've more issues to raise alongside the
>>>> upcoming EAP requirements), but let's start with those first 3 major
>>>> points and my POC, and run from there.
>>>> 
>>>> S,
>>>> ALR
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>> 
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> 
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

More information about the jboss-as7-dev mailing list