[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Jason T. Greene]

On 12/1/11 11:47 PM, Ondřej Žižka wrote:
> Hi everyone interested,
>
> there are these motivations for grouping the tests:
>
> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> 3) Running tests with different configs.
> 4) Running various groups of tests.
>
> 4) can be done using e.g.
> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
> works fine), so I'll drop it from further considerations).
>
> 1-3) is currently done by a combination of modules and Surefire executions.
> The problem with the later is that a failure in one execution prevents
> the successive from running.
>
> Here are options how that can be solved:
>
> *A) Naive solution: Keep status quo, keep tests in the first execution
> in a good condition.*
> Pros: * We already have it.
> Cons: * It will fail now and then, and people would need an extra run
> for the other group, using various params.
>
> *B) Megalomanic solution: Make every combination a maven module.*
> Pros:
> * Avaliable right away.
> Cons:
> * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
> basic-full, ...

I don't think we need MxN modules that accounts for [2/3]. That can just 
be an optional property that allows the entire testsuite to use a 
different configuration. Note that the default only runs "full" against 
things that need full, so the default runs fully test the default. You 
only theoretically need one extra run to verify the entire full config, 
and it's something you don't have to do very often (can just be a 
one-off hudson run near a release).

IMO If we have multiple modules it should just be one per functional 
area (approximation of a subsystem). That's it. We don't even need one 
for every subsystem, and even if we did we only have 20 subsystems.

The real advantage is that its just clean organization, clean layout and 
it's common maven behavior. Yes there is some copying of pom.xml files, 
but that's just boilerplate, and you can script updates like we already 
have to do.

Also note, that I'm not arguing for 20 modules right now. What I am 
saying is that we can take the 3 or so that require full and make them 
modules right now, and then later, over time migrate when needed.

-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat