[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Ondřej Žižka]

Hi everyone interested,

there are these motivations for grouping the tests:

1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
3) Running tests with different configs.
4) Running various groups of tests.

4) can be done using e.g.
-Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
works fine), so I'll drop it from further considerations).

1-3)  is currently done by a combination of modules and Surefire
executions.
The problem with the later is that a failure in one execution prevents
the successive from running.

Here are options how that can be solved:

A) Naive solution: Keep status quo, keep tests in the first execution in
a good condition.
  Pros:   * We already have it.
  Cons:  * It will fail now and then, and people would need an extra run
for the other group, using various params.

B) Megalomanic solution: Make every combination a maven module.
Pros:
  * Avaliable right away.
Cons:
  * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
basic-full, ...
  * => much harder to maintain, esp. for the product features (need to
distribute stuff to too many pom.xml's).
  * Slower testsuite and AS7 build.
  * Would not scale for the cases when each test needs a different
config.

C) Optimistic engineer's solution:  Use JUnit's new @Category  and
Arquillians ability to restart a server with other config, based on
@ContainerConfig("path-to-config.xml").
Pros:
  * Simple, clean approach; no pom.xml hell.
  * Easy to maintain.
  * Easy to reorganize later.
  * Would get 3) and 4) out of maven
Cons:
  * Arq's impl of @Category is not tested
  * We would need to put @Category to all tests
  * @ContainerConfig("path-to-config.xml") is not implemented. We could
(mis)use @TargetsContainer and @OperatesOnDeployment.

Are there others?
Did I miss some pros or cons?
Which one do you like?

Ondra

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-as7-dev/attachments/20111202/6d034244/attachment.html