[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

[Andrew Lee Rubinger]

The only objection I see to [2] is the use of:

org.jboss.stdio.WriterOutputStream

...unless that class is a part of the published AS API, in which case 
this Servlet can go in the User API TestSuite.

Otherwise, if you wanna be hacking into internals, we gotta look 
elsewhere for a place to put this.

S,
ALR

On 03/22/2011 08:18 AM, Carlo de Wolf wrote:
> An additional note on the 'integration' tests. While [1] is a nice
> example for the API test-suite in the integration test-suite I want a
> no-bar, so I can do stuff like [2].
>
> [1] also needs a spot of comment like [3]. The moment we go "what or
> why" on a test, it's a goner.
>
> Carlo
>
> [1]
> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
>
> [2]
> https://github.com/wolfc/jboss-as/blob/weld/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/weldejb/SimpleServlet.java
>
> [3]
> https://github.com/wolfc/jboss-as/blob/weld/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/weldejb/SessionObjectReferenceTestCase.java#L45
>
>
> On 03/18/2011 08:21 AM, Andrew Lee Rubinger wrote:
>> Looks like a lot of us have different ideas for what the AS7 Integration
>> TestSuite should consist of, so I'll kickoff with what I believe is the
>> first design proposal towards getting coverage focused on the end-user
>> (not certifying our own internals).
>>
>> I suspect this breaks down into two categories, which may be modelled as
>> separate modules under the existing "testsuite" aggregator parent:
>>
>> * Specification
>> * AS-specific APIs
>>
>> This isn't difficult work, though I do think it's important we consider
>> some hard rules. IMO we should be developing these suites as if we were
>> application developers, not wearing our server dev hats.
>>
>> ----------------
>>
>> [End Goal]
>>
>> 1) No compile-time dependencies in the module except for what's
>> absolutely necessary.
>>
>> For the spec suite, this means: JDK and EE Spec APIs only in the
>> compilation classpath. Testable asset sources and resources (ie. EJBs,
>> Servlets, etc) would live under src/main/* to enforce that. Only the
>> tests themselves would be located under "src/test/*".
>>
>> The AS-specific API suite may also add in our own APIs to the
>> compilation classpath, but the line should end there. In "test" scope
>> we can place all runtime dependencies.
>>
>> For the specification suite, AS-specific grammars like our own
>> deployment descriptors are fine; these are in many ways equivalent to
>> the TCK porting layer. We're not building a TCK; we're showing that our
>> implementation supports the features advertised.
>>
>> 2) Every single new test created is to have an associated JIRA.
>>
>> We all remember the nightmare it was when the old AS4-6 suite would fall
>> down. We'd comb through each test, at times trying to determine its
>> purpose. By linking to JIRA we get history of intent, which acts as a
>> nice record even in the case that the test isn't so well-documented.
>> I'd argue that tests are a bigger asset than our code, and we should be
>> thinking about these in terms of long-term maintenance to outlive any
>> specific impl.
>>
>> 3) Documentation
>>
>> Alongside the JIRA reference, a quick note about we're looking to
>> accomplish is something I find very helpful. I don't personally buy the
>> argument that code is self-documenting if written well. It gets
>> refactored and stale over time.
>>
>> 4) Run-mode profiles
>>
>> Arquillian provides a wonderful abstraction such that we can get
>> coverage for AS in both remote managed *and* embedded modes without
>> changing the test itself. To certify that everything is working as
>> advertised no matter the runtime, we should be able to run the same
>> suite in standalone, domain, and embedded modes (generally speaking).
>>
>> 5) Porting of AS6 Tests
>>
>> There's no discounting the value this coverage has given us, though I
>> question the purpose of a lot of these tests. I think a great majority
>> of these need to come into the new codebase, refactored to align if
>> needed.
>>
>> ----------------
>>
>> [Current State]
>>
>> Here[1] is an example of what I believe to be a simple, well-written
>> test, with the exception that the tested Servlet and EJB are in the same
>> test source folder.
>>
>> The current "testsuite" aggregator contains modules which mix our
>> end-user certification stuff alongside internals, so I think these
>> should be separated out.
>>
>> A lot of this is set up in some fashion already, but I would like to
>> see us:
>>
>> 1) Agree upon a strict scope for each type of testsuite along the lines
>> of my points above, once we reach agreement
>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap
>> 1.0.0-alpha-12), just released tonight. Currently AS is on a forked
>> release of ARQ for OSGi purposes, and these changes, if necessary, need
>> to get upstream so we can do upgrades.
>>
>> It's clear that AS7 has made full-steam-ahead progress since last
>> summer, and with a little organization our testsuite can give us a great
>> view of where we stand, from an end-user's perspective, with minimal
>> investment.
>>
>> S,
>> ALR
>>
>>
>> [1]
>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src/test/java/org/jboss/as/testsuite/integration/webejb/ServletInjectionTestCase.java
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>