[jboss-as7-dev] Revisited: Integration TestSuite Organization and Maintenance

Mon Aug 15 17:37:26 EDT 2011

On 08/15/2011 03:27 AM, Andrew Lee Rubinger wrote:
> A summary of where I believe we stand on the 3 points raised here:
>
> 1) TestSuite Organization
>
> Stuart and Kabir have voiced concerns that separating src/main and src/test make authoring more complex.
>
> While I'll concede that may be true to a small extent, I believe the benefits of separation far outweigh the drawbacks.  By separating these out I uncovered 4 JIRAs which proved that our "api" and "spec-api" modules were incomplete.  In short, paying attention to user dependencies to validate against them is very important.
I've been trying to understand in more detail the issues you have raised 
concerning creating separate modules for api testing and another for 
general integration testing.  I think what is missing here (I may be 
missing the full point) is a description of what you are specifically 
trying to achieve and we are getting caught up instead in implementation 
details. To make things clear, i'm going to refer to this effort as 
'spec testing'. Some related questions:

1. I looked at the ServletInjectionTestCase example you gave, and it 
seems that your motivation for putting artifacts which may be deployed 
during the test case in src/main is simply to be able to:
* compile them against a minimal set of dependencies which are only JDK 
jars and J2EE jars (in the case of the spec suite)
* compile them against a minimal set of dependencies which are only JDK 
jars, J2EE jars and the AS API jars (in the case of the AS-specific API 
suite suite)
Is that all you want to do for your spec testing? Simply compile a set 
of java classes representing deployable artifacts against minimal set of 
dependencies and check that the compilation succeeds? I suspect that you 
have a wider purpose, but, if that is the case, there could be other 
ways to achieve the same end.

I also see the word 'complete' being used, which leads me to believe 
that you don't just want to perform this process for one specific 
deployable artifact, but rather you have some important 'covering' set 
of deployable artifacts in mind and maybe test cases as well? What does 
the word 'complete' mean with regard to this 'spec testing'? And how 
will you know when you have achieved full coverage, whether it is in a 
separate module or not?

2. I can see how the JBoss specific 'spec testing' would be include 
things not found in the TCK, but how is your spec module different from 
what the TCK would do? Does it cover cases which are not covered by the 
TCK?

3. Is your ultimate goal to:
(i) test to some level of detail every user-facing API (and every method 
within that API), whether that be accessed from deployable components 
(like EJBs and Servlets) or components which interact with the AS 
remotely (like the CLI or the Domain management interface) and
(ii) for these components, make sure they compile against the minimal 
set of dependecies

Or are there other criteria for performing this spec-style testing?
> 2) Run Modes, Test Subsets
>
> I don't think there's been much discussion here, with the exception of the QE team who have provided some use cases that may not be possible given a standard layout.  For instance the CLI and clustering tests are more than the standard "deploy something and make assertions" format we cover in point 1), so these will likely get their own modules as is appropriate.
On the matter of how to concretely organize the testsuite (what we need 
to be able to do when executing a test case, where to put sources and 
resources for test cases, where to put server configs, where to put 
descriptions of sets of test cases, etc), there has been some discussion.

On that front, there are two proposals which have been documented:
* one from QE, found here: 
https://docspace.corp.redhat.com/docs/DOC-69049, based on using:
-- submodules to organize test case sources
-- submodules to create server configurations
-- surefire plugin executions to execute sets of test cases
* one from me, found here (parts still under construction) : 
https://docspace.corp.redhat.com/docs/DOC-74146, based on using:
-- a single module and package structure to organize test case sources
-- ant snippets to create server configurations (a simple mojo could be 
created as well and tied to a lifecycle phase)
-- plugin executions and profile aggregation to structure test cases and 
execute sets of test cases
Both approaches will probably work. Essentially, one is based on 
modules, the other on flat files. What needs to be decided is which is 
simplest to use and maintain. Or maybe there is a third option.

Another issue is the JUnit/testNG split: are we going to support two 
test execution frameworks for the next 5 years or simplify down to one. 
At the moment, JUnit has been used by default (largely through 
intertia), whereas both JGroups and Infinispan converted to testNG years 
ago. testNG has many features for grouping test cases which JUnit does 
not, for example. In my view, we should scrap JUnit.

> 3) An authoritative maintainer
>
> No one has commented on this.
>
> By Tuesday evening I need to report back on at least the 3 points above, and also have approval for getting my patch committed.  Failing that, we need to agree on an alternate path forward.  I'm happy with any solution which addresses the respect for dependencies in the testsuite as I've outlined.
>
> S,
> ALR
>
> ----- Original Message -----
> From: "Andrew Lee Rubinger"<andrew.rubinger at redhat.com>
> To: "JBoss AS7 Development"<jboss-as7-dev at lists.jboss.org>
> Sent: Thursday, August 11, 2011 7:38:51 AM
> Subject: [jboss-as7-dev] Revisited: Integration TestSuite Organization and	Maintenance
>
> Hi guys:
>
> I'd like to reopen the discussion regarding the testsuite organization
> and its ongoing maintenance.  This issue dates back a few months with
> some debates and differing opinions, so I'll do my best to outline the
> guiding principles I'd like to see put in place concisely.
>
> To start off, I've a Proof-of-Concept for many of the following points
> now located:
>
> https://github.com/ALRubinger/jboss-as/tree/AS7-999
>
> The relevant JIRA I've been using to track things:
>
> https://issues.jboss.org/browse/AS7-999
>
> So:
>
> 1) TestSuite Organization
>
> I believe we need a single top-level categorization by which we may
> organize integration tests which are deployment-based and run within the
> context of the server.  Because we use Maven modules (which are bound to
> a dependency structure), it makes sense to file these modules by the
> compile-time dependencies they require.  So in place I've put:
>
> testsuite/spec - Java SE and Java EE APIs only
> testsuite/api - AS7 APIs + Spec
> testsuite/internals - Use anything in the AS7 runtime in your
> deployments; not guaranteed to be back-compat across releases
>
> The primary motivation here is to ensure that the dependencies we export
> (ie. "spec-api", and "api" modules) are complete enough for users to
> create their own deployments.  In this setup, we act as *users* of our
> own APIs, and everything in src/main is limited to the relevant
> dependencies.
>
> I know the source of some disagreements earlier centered around placing
> the tests right next to the deployments, and some folks consider the
> deployments as part of the test itself.  That's not a bad argument at
> all, but again consider that we then lose the ability to validate our
> tests in the context of our exported APIs.
>
> 2) Run Modes, Test Subsets
>
> Because the primary organizational criteria proposed in 1) is by
> dependency, these modules will grow large over time.  The AS build over
> time will take longer and longer to run.  Additionally, there are
> runtime options to consider when starting tests.  So consider the
> following requirements:
>
>     * Running the testsuite in IPv6
>     * Running only a subset of tests as part of the main build
>
> These lend themselves well to using build profiles.  By default, I think
> the "smoke tests" should simply be a set of tests we deem important or
> indicative of the general health of AS7 with respect to each subsystem.
>    As it stands now, "smoke" is its own module with a bunch of
> Embedded-based tests, and I think these should move to the
> organizational structure in 1) and instead we can apply some filtering
> to make the "smoke" some default set of includes.
>
> 3) An authoritative maintainer
>
> I'd like to treat the Arquillian and TestSuite modules as true
> subsystems of the Application Server, and as such we'll need someone to
> assume the responsibility to review incoming commits/pull requests and
> ensure they fit the criteria for acceptance.  Simple things like
> consistent package names, using ARQ correctly, and not leaking
> dependencies are very important.
>
> So assuming we come to agreement on these points, I'd like to request
> push access to the AS7 repo to field testsuite and ARQ-related pull
> requests.
>
> ...there's much more to discuss (I've more issues to raise alongside the
> upcoming EAP requirements), but let's start with those first 3 major
> points and my POC, and run from there.
>
> S,
> ALR
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev