[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Aslak Knutsen]

I think you missed the most obvious one.. 

E) Fix Maven Execution / Surefire

-aslak-


----- Original Message -----
> Hi everyone interested,
> 
> there are these motivations for grouping the tests:
> 
> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> 3) Running tests with different configs.
> 4) Running various groups of tests.
> 
> 4) can be done using e.g.
> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
> -Dtest works fine), so I'll drop it from further considerations).
> 
> 1-3) is currently done by a combination of modules and Surefire
> executions.
> The problem with the later is that a failure in one execution
> prevents the successive from running.
> 
> Here are options how that can be solved:
> 
> A) Naive solution: Keep status quo, keep tests in the first execution
> in a good condition.
> Pros: * We already have it.
> Cons: * It will fail now and then, and people would need an extra run
> for the other group, using various params.
> 
> B) Megalomanic solution: Make every combination a maven module.
> Pros:
> * Avaliable right away.
> Cons:
> * Will result in many modules - e.g. smoke-web, smoke-full,
> basic-web, basic-full, ...
> * => much harder to maintain, esp. for the product features (need to
> distribute stuff to too many pom.xml's).
> * Slower testsuite and AS7 build.
> * Would not scale for the cases when each test needs a different
> config.
> 
> C) Optimistic engineer's solution: Use JUnit's new @Category and
> Arquillians ability to restart a server with other config, based on
> @ContainerConfig("path-to-config.xml").
> Pros:
> * Simple, clean approach; no pom.xml hell.
> * Easy to maintain.
> * Easy to reorganize later.
> * Would get 3) and 4) out of maven
> Cons:
> * Arq's impl of @Category is not tested
> * We would need to put @Category to all tests
> * @ContainerConfig("path-to-config.xml") is not implemented. We could
> (mis)use @TargetsContainer and @OperatesOnDeployment.
> 
> Are there others?
> Did I miss some pros or cons?
> Which one do you like?
> 
> Ondra
> 
>