[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Brian Stansberry]

On 12/2/11 10:15 AM, Jason T. Greene wrote:
> On 12/1/11 11:47 PM, Ondřej Žižka wrote:
>> Hi everyone interested,
>>
>> there are these motivations for grouping the tests:
>>
>> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
>> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
>> 3) Running tests with different configs.
>> 4) Running various groups of tests.
>>
>> 4) can be done using e.g.
>> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
>> works fine), so I'll drop it from further considerations).
>>
>> 1-3) is currently done by a combination of modules and Surefire executions.
>> The problem with the later is that a failure in one execution prevents
>> the successive from running.
>>
>> Here are options how that can be solved:
>>
>> *A) Naive solution: Keep status quo, keep tests in the first execution
>> in a good condition.*
>> Pros: * We already have it.
>> Cons: * It will fail now and then, and people would need an extra run
>> for the other group, using various params.
>>
>> *B) Megalomanic solution: Make every combination a maven module.*
>> Pros:
>> * Avaliable right away.
>> Cons:
>> * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
>> basic-full, ...
>
> I don't think we need MxN modules that accounts for [2/3]. That can just
> be an optional property that allows the entire testsuite to use a
> different configuration. Note that the default only runs "full" against
> things that need full, so the default runs fully test the default. You
> only theoretically need one extra run to verify the entire full config,
> and it's something you don't have to do very often (can just be a
> one-off hudson run near a release).
>
> IMO If we have multiple modules it should just be one per functional
> area (approximation of a subsystem). That's it. We don't even need one
> for every subsystem, and even if we did we only have 20 subsystems.
>
> The real advantage is that its just clean organization, clean layout and
> it's common maven behavior. Yes there is some copying of pom.xml files,
> but that's just boilerplate, and you can script updates like we already
> have to do.
>
> Also note, that I'm not arguing for 20 modules right now. What I am
> saying is that we can take the 3 or so that require full and make them
> modules right now, and then later, over time migrate when needed.
>

I argued in a separate message against significant refactors prior to 
CR1. I don't regard what you're described in this last paragraph as a 
significant refactor.

-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat