[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Jason T. Greene]

On 12/2/11 10:20 AM, Brian Stansberry wrote:
> On 12/2/11 10:15 AM, Jason T. Greene wrote:
>> On 12/1/11 11:47 PM, Ondřej Žižka wrote:
>>> Hi everyone interested,
>>>
>>> there are these motivations for grouping the tests:
>>>
>>> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
>>> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
>>> 3) Running tests with different configs.
>>> 4) Running various groups of tests.
>>>
>>> 4) can be done using e.g.
>>> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
>>> works fine), so I'll drop it from further considerations).
>>>
>>> 1-3) is currently done by a combination of modules and Surefire executions.
>>> The problem with the later is that a failure in one execution prevents
>>> the successive from running.
>>>
>>> Here are options how that can be solved:
>>>
>>> *A) Naive solution: Keep status quo, keep tests in the first execution
>>> in a good condition.*
>>> Pros: * We already have it.
>>> Cons: * It will fail now and then, and people would need an extra run
>>> for the other group, using various params.
>>>
>>> *B) Megalomanic solution: Make every combination a maven module.*
>>> Pros:
>>> * Avaliable right away.
>>> Cons:
>>> * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
>>> basic-full, ...
>>
>> I don't think we need MxN modules that accounts for [2/3]. That can just
>> be an optional property that allows the entire testsuite to use a
>> different configuration. Note that the default only runs "full" against
>> things that need full, so the default runs fully test the default. You
>> only theoretically need one extra run to verify the entire full config,
>> and it's something you don't have to do very often (can just be a
>> one-off hudson run near a release).
>>
>> IMO If we have multiple modules it should just be one per functional
>> area (approximation of a subsystem). That's it. We don't even need one
>> for every subsystem, and even if we did we only have 20 subsystems.
>>
>> The real advantage is that its just clean organization, clean layout and
>> it's common maven behavior. Yes there is some copying of pom.xml files,
>> but that's just boilerplate, and you can script updates like we already
>> have to do.
>>
>> Also note, that I'm not arguing for 20 modules right now. What I am
>> saying is that we can take the 3 or so that require full and make them
>> modules right now, and then later, over time migrate when needed.
>>
>
> I argued in a separate message against significant refactors prior to
> CR1. I don't regard what you're described in this last paragraph as a
> significant refactor.
>

BTW I should note that I can live with the two execution status quo. 
It's not great but it does work (mostly).

-- 
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat