[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

[Andrew Lee Rubinger]

To follow up on this with some JIRAs:

Nothing proposed here is too complex, and reading through the lengthy 
responses to this topic I think the only real point of disagreement is 
the "JIRA per test" thing.  As there's no policy in AS right now to 
reference a JIRA for each commit, we'll just skip this part.

To that end, I've some tracking JIRAs:

1) (JBAS-9285) Create AS Specification TestSuite

...this to cover most user cases.  No reliance on AS APIs or internals.

2) (JBAS-9284) Create AS API TestSuite

...this to cover other user cases depending upon the AS7 APIs.  Which 
depends upon our providing a unified view of the APIs:

(JBAS-9283) Create POM and Assembly for AS7 Public APIs

Still underway from a couple angles is the Arquillian connectors for AS7 
and the upgrade to ARQ 1.0.0.Alpha5: JBAS-8946.  The current state of 
that is in my fork, noted in the comments of the issue.

S,
ALR

On 03/18/2011 03:21 AM, Andrew Lee Rubinger wrote:
> Looks like a lot of us have different ideas for what the AS7 Integration
> TestSuite should consist of, so I'll kickoff with what I believe is the
> first design proposal towards getting coverage focused on the end-user
> (not certifying our own internals).
>
> I suspect this breaks down into two categories, which may be modelled as
> separate modules under the existing "testsuite" aggregator parent:
>
> * Specification
> * AS-specific APIs
>
> This isn't difficult work, though I do think it's important we consider
> some hard rules.  IMO we should be developing these suites as if we were
> application developers, not wearing our server dev hats.
>
> ----------------
>
> [End Goal]
>
> 1) No compile-time dependencies in the module except for what's
> absolutely necessary.
>
> For the spec suite, this means: JDK and EE Spec APIs only in the
> compilation classpath.  Testable asset sources and resources (ie. EJBs,
> Servlets, etc) would live under src/main/* to enforce that.  Only the
> tests themselves would be located under "src/test/*".
>
> The AS-specific API suite may also add in our own APIs to the
> compilation classpath, but the line should end there.  In "test" scope
> we can place all runtime dependencies.
>
> For the specification suite, AS-specific grammars like our own
> deployment descriptors are fine; these are in many ways equivalent to
> the TCK porting layer.  We're not building a TCK; we're showing that our
> implementation supports the features advertised.
>
> 2) Every single new test created is to have an associated JIRA.
>
> We all remember the nightmare it was when the old AS4-6 suite would fall
> down.  We'd comb through each test, at times trying to determine its
> purpose.  By linking to JIRA we get history of intent, which acts as a
> nice record even in the case that the test isn't so well-documented.
> I'd argue that tests are a bigger asset than our code, and we should be
> thinking about these in terms of long-term maintenance to outlive any
> specific impl.
>
> 3) Documentation
>
> Alongside the JIRA reference, a quick note about we're looking to
> accomplish is something I find very helpful.  I don't personally buy the
> argument that code is self-documenting if written well.  It gets
> refactored and stale over time.
>
> 4) Run-mode profiles
>
> Arquillian provides a wonderful abstraction such that we can get
> coverage for AS in both remote managed *and* embedded modes without
> changing the test itself.  To certify that everything is working as
> advertised no matter the runtime, we should be able to run the same
> suite in standalone, domain, and embedded modes (generally speaking).
>
> 5) Porting of AS6 Tests
>
> There's no discounting the value this coverage has given us, though I
> question the purpose of a lot of these tests.  I think a great majority
> of these need to come into the new codebase, refactored to align if needed.
>
> ----------------
>
> [Current State]
>
> Here[1] is an example of what I believe to be a simple, well-written
> test, with the exception that the tested Servlet and EJB are in the same
> test source folder.
>
> The current "testsuite" aggregator contains modules which mix our
> end-user certification stuff alongside internals, so I think these
> should be separated out.
>
> A lot of this is set up in some fashion already, but I would like to see us:
>
> 1) Agree upon a strict scope for each type of testsuite along the lines
> of my points above, once we reach agreement
> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
> 1.0.0-alpha-12), just released tonight.  Currently AS is on a forked
> release of ARQ for OSGi purposes, and these changes, if necessary, need
> to get upstream so we can do upgrades.
>
> It's clear that AS7 has made full-steam-ahead progress since last
> summer, and with a little organization our testsuite can give us a great
> view of where we stand, from an end-user's perspective, with minimal
> investment.
>
> S,
> ALR
>
>
> [1]
> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev