[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Darran Lofthouse]

One other topic that has come up recently is interoperability with 
previous AS releases, from the discussions so far I don't think now is 
going to be the time to address it but it could be worth considering how 
it would at least fit.

I wonder if a single module concentrating on the management interactions 
needed for interoperability, that module could then be manually run 
against remote AS instances of previous versions to verify nothing is 
broken.

Regards,
Darran Lofthouse.


On 12/02/2011 07:49 AM, Thomas Diesler wrote:
> Good post - thanks Ondrej
>
> how about creating a maven module per purpose (i.e. smoke, basic,
> stress, etc). Each module would have the ability to create its own
> config via xslt transform. Selecting tests by functional area would work
> via package selection (i.e. -Dtest=jms/**/*TestCase). This means I can
> run a subset of tests in any given module. The subset can be as small as
> one single test.
>
> If a subset of tests must be run with a different config, we could have
> a profile per config, which could be activated by -Dconfig=foo. The
> default profile would be activated by not having a -Dconfig parameter.
> Each profile has the ability to do config transformation.
>
> With this setup you would not have multiple surefire executions. The
> large set of integration tests would live in one module (i.e. basic) and
> be executed together by default. This module could contain any number of
> specialized profiles which select arbitrary subsets with arbitrary configs.
>
> cheers
> -thomas
>
> On 12/02/2011 06:47 AM, Ondřej Žižka wrote:
>> Hi everyone interested,
>>
>> there are these motivations for grouping the tests:
>>
>> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
>> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
>> 3) Running tests with different configs.
>> 4) Running various groups of tests.
>>
>> 4) can be done using e.g.
>> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
>> -Dtest works fine), so I'll drop it from further considerations).
>>
>> 1-3)  is currently done by a combination of modules and Surefire
>> executions.
>> The problem with the later is that a failure in one execution prevents
>> the successive from running.
>>
>> Here are options how that can be solved:
>>
>> *A) Naive solution: Keep status quo, keep tests in the first execution
>> in a good condition.*
>>   Pros:   * We already have it.
>>   Cons:  * It will fail now and then, and people would need an extra
>> run for the other group, using various params.
>>
>> *B) Megalomanic solution: Make every combination a maven module.*
>> Pros:
>>   * Avaliable right away.
>> Cons:
>>   * Will result in many modules - e.g. smoke-web, smoke-full,
>> basic-web, basic-full, ...
>>   * => much harder to maintain, esp. for the product features (need to
>> distribute stuff to too many pom.xml's).
>>   * Slower testsuite and AS7 build.
>>   * Would not scale for the cases when each test needs a different config.
>>
>> *C) Optimistic engineer's solution:  Use JUnit's new @Category  and
>> Arquillians ability to restart a server with other config, based on
>> @ContainerConfig("path-to-config.xml").*
>> Pros:
>>   * Simple, clean approach; no pom.xml hell.
>>   * Easy to maintain.
>>   * Easy to reorganize later.
>>   * Would get 3) and 4) out of maven
>> Cons:
>>   * Arq's impl of @Category is not tested
>>   * We would need to put @Category to all tests
>>   * @ContainerConfig("path-to-config.xml") is not implemented. We
>> could (mis)use @TargetsContainer and @OperatesOnDeployment.
>>
>> *Are there others?*
>> *Did I miss some pros or cons?*
>> *Which one do you like?*
>>
>> Ondra
>>
>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> --
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Thomas Diesler
> JBoss OSGi Lead
> JBoss, a division of Red Hat
> xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev