[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

[Carlo de Wolf]

An additional note on the 'integration' tests. While [1] is a nice 
example for the API test-suite in the integration test-suite I want a 
no-bar, so I can do stuff like [2].

[1] also needs a spot of comment like [3]. The moment we go "what or 
why" on a test, it's a goner.

Carlo

[1] 
https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
[2] 
https://github.com/wolfc/jboss-as/blob/weld/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/weldejb/SimpleServlet.java
[3] 
https://github.com/wolfc/jboss-as/blob/weld/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/weldejb/SessionObjectReferenceTestCase.java#L45

On 03/18/2011 08:21 AM, Andrew Lee Rubinger wrote:
> Looks like a lot of us have different ideas for what the AS7 Integration
> TestSuite should consist of, so I'll kickoff with what I believe is the
> first design proposal towards getting coverage focused on the end-user
> (not certifying our own internals).
>
> I suspect this breaks down into two categories, which may be modelled as
> separate modules under the existing "testsuite" aggregator parent:
>
> * Specification
> * AS-specific APIs
>
> This isn't difficult work, though I do think it's important we consider
> some hard rules.  IMO we should be developing these suites as if we were
> application developers, not wearing our server dev hats.
>
> ----------------
>
> [End Goal]
>
> 1) No compile-time dependencies in the module except for what's
> absolutely necessary.
>
> For the spec suite, this means: JDK and EE Spec APIs only in the
> compilation classpath.  Testable asset sources and resources (ie. EJBs,
> Servlets, etc) would live under src/main/* to enforce that.  Only the
> tests themselves would be located under "src/test/*".
>
> The AS-specific API suite may also add in our own APIs to the
> compilation classpath, but the line should end there.  In "test" scope
> we can place all runtime dependencies.
>
> For the specification suite, AS-specific grammars like our own
> deployment descriptors are fine; these are in many ways equivalent to
> the TCK porting layer.  We're not building a TCK; we're showing that our
> implementation supports the features advertised.
>
> 2) Every single new test created is to have an associated JIRA.
>
> We all remember the nightmare it was when the old AS4-6 suite would fall
> down.  We'd comb through each test, at times trying to determine its
> purpose.  By linking to JIRA we get history of intent, which acts as a
> nice record even in the case that the test isn't so well-documented.
> I'd argue that tests are a bigger asset than our code, and we should be
> thinking about these in terms of long-term maintenance to outlive any
> specific impl.
>
> 3) Documentation
>
> Alongside the JIRA reference, a quick note about we're looking to
> accomplish is something I find very helpful.  I don't personally buy the
> argument that code is self-documenting if written well.  It gets
> refactored and stale over time.
>
> 4) Run-mode profiles
>
> Arquillian provides a wonderful abstraction such that we can get
> coverage for AS in both remote managed *and* embedded modes without
> changing the test itself.  To certify that everything is working as
> advertised no matter the runtime, we should be able to run the same
> suite in standalone, domain, and embedded modes (generally speaking).
>
> 5) Porting of AS6 Tests
>
> There's no discounting the value this coverage has given us, though I
> question the purpose of a lot of these tests.  I think a great majority
> of these need to come into the new codebase, refactored to align if needed.
>
> ----------------
>
> [Current State]
>
> Here[1] is an example of what I believe to be a simple, well-written
> test, with the exception that the tested Servlet and EJB are in the same
> test source folder.
>
> The current "testsuite" aggregator contains modules which mix our
> end-user certification stuff alongside internals, so I think these
> should be separated out.
>
> A lot of this is set up in some fashion already, but I would like to see us:
>
> 1) Agree upon a strict scope for each type of testsuite along the lines
> of my points above, once we reach agreement
> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
> 1.0.0-alpha-12), just released tonight.  Currently AS is on a forked
> release of ARQ for OSGi purposes, and these changes, if necessary, need
> to get upstream so we can do upgrades.
>
> It's clear that AS7 has made full-steam-ahead progress since last
> summer, and with a little organization our testsuite can give us a great
> view of where we stand, from an end-user's perspective, with minimal
> investment.
>
> S,
> ALR
>
>
> [1]
> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev