[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Brian Stansberry]

Until 7.1 is out, I want A).

I am tired of testsuite refactors; they have cost me far more 
productivity than dealing with failures has.

IMHO we don't have time on the CR1 schedule to be significantly 
refactoring the testsuite.

On 12/1/11 11:47 PM, Ondřej Žižka wrote:
> Hi everyone interested,
>
> there are these motivations for grouping the tests:
>
> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> 3) Running tests with different configs.
> 4) Running various groups of tests.
>
> 4) can be done using e.g.
> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
> works fine), so I'll drop it from further considerations).
>
> 1-3) is currently done by a combination of modules and Surefire executions.
> The problem with the later is that a failure in one execution prevents
> the successive from running.
>
> Here are options how that can be solved:
>
> *A) Naive solution: Keep status quo, keep tests in the first execution
> in a good condition.*
> Pros: * We already have it.
> Cons: * It will fail now and then, and people would need an extra run
> for the other group, using various params.
>
> *B) Megalomanic solution: Make every combination a maven module.*
> Pros:
> * Avaliable right away.
> Cons:
> * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
> basic-full, ...
> * => much harder to maintain, esp. for the product features (need to
> distribute stuff to too many pom.xml's).
> * Slower testsuite and AS7 build.
> * Would not scale for the cases when each test needs a different config.
>
> *C) Optimistic engineer's solution: Use JUnit's new @Category and
> Arquillians ability to restart a server with other config, based on
> @ContainerConfig("path-to-config.xml").*
> Pros:
> * Simple, clean approach; no pom.xml hell.
> * Easy to maintain.
> * Easy to reorganize later.
> * Would get 3) and 4) out of maven
> Cons:
> * Arq's impl of @Category is not tested
> * We would need to put @Category to all tests
> * @ContainerConfig("path-to-config.xml") is not implemented. We could
> (mis)use @TargetsContainer and @OperatesOnDeployment.
>
> *Are there others?*
> *Did I miss some pros or cons?*
> *Which one do you like?*
>
> Ondra
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev


-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat