[jboss-as7-dev] Requirements and Design Proposal: AS7 TestSuite

[Jaikiran Pai]

On Friday 18 March 2011 12:51 PM, Andrew Lee Rubinger wrote:
>
> I suspect this breaks down into two categories, which may be modelled as
> separate modules under the existing "testsuite" aggregator parent:
>
> * Specification
> * AS-specific APIs
On a slightly different categorization, we also have to come up with 
tests for stress testing. They can either be for "Specification" related 
tests or even for "AS-specific APIs". Perhaps test "profiles" (need 
_not_ be Maven profile) within those 2 modules would be better instead 
of a separate module?

>
> [End Goal]
>
> 1) No compile-time dependencies in the module except for what's
> absolutely necessary.
>
> For the spec suite, this means: JDK and EE Spec APIs only in the
> compilation classpath.
We might also have to include JBoss specific APIs which are exposed to 
the users (for ex: org.jboss.ejb3.annotation which currently isn't there 
but might be introduced soon in AS7).
> Testable asset sources and resources (ie. EJBs,
> Servlets, etc) would live under src/main/* to enforce that.  Only the
> tests themselves would be located under "src/test/*".
Sounds fine. Although, do we want to end up with all tech specific tests 
(like for EJB3, web, etc...) all in one source path? i.e. 
spec-tests/src/main/org/jboss/spec/<ejb3, web, jsf>....? Or should we 
plan to have different maven modules under the "Spec" tests for each 
individual tech?

>
> 2) Every single new test created is to have an associated JIRA.
I'm not quite sure that this will work out. I'm not against it, but it 
might not be practical.
>
> By linking to JIRA we get history of intent, which acts as a
> nice record even in the case that the test isn't so well-documented.
> I'd argue that tests are a bigger asset than our code, and we should be
> thinking about these in terms of long-term maintenance to outlive any
> specific impl.
I agree about the well-documented tests, but JIRA linking shouldn't 
always be an absolute necessity.
>
> 3) Documentation
>
> Alongside the JIRA reference, a quick note about we're looking to
> accomplish is something I find very helpful.  I don't personally buy the
> argument that code is self-documenting if written well.  It gets
> refactored and stale over time.
+1000. A simple brief explanation on what the test is supposed to test 
is really useful.
>
> 4) Run-mode profiles
>
> Arquillian provides a wonderful abstraction such that we can get
> coverage for AS in both remote managed *and* embedded modes without
> changing the test itself.  To certify that everything is working as
> advertised no matter the runtime, we should be able to run the same
> suite in standalone, domain, and embedded modes (generally speaking).
Are these container specific modes that we are talking about? Or are 
these more of test profiles (a.k.a groups) wherein we could perhaps add 
a test under a "stress-test" profile?

-Jaikiran