[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Ondřej Žižka]

Aslak Knutsen píše v Pá 02. 12. 2011 v 04:34 -0500:

> I think you missed the most obvious one.. 
> 
> E) Fix Maven Execution / Surefire

Actually I didn't,  http://jira.codehaus.org/browse/SUREFIRE-803
Discussed with John Casey (CC).

John, do you think this could be fixed until CR1?  (2011-12-21)

Thanks,
Ondra





> 
> -aslak-
> 
> 
> ----- Original Message -----
> > Hi everyone interested,
> > 
> > there are these motivations for grouping the tests:
> > 
> > 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> > 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> > 3) Running tests with different configs.
> > 4) Running various groups of tests.
> > 
> > 4) can be done using e.g.
> > -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
> > -Dtest works fine), so I'll drop it from further considerations).
> > 
> > 1-3) is currently done by a combination of modules and Surefire
> > executions.
> > The problem with the later is that a failure in one execution
> > prevents the successive from running.
> > 
> > Here are options how that can be solved:
> > 
> > A) Naive solution: Keep status quo, keep tests in the first execution
> > in a good condition.
> > Pros: * We already have it.
> > Cons: * It will fail now and then, and people would need an extra run
> > for the other group, using various params.
> > 
> > B) Megalomanic solution: Make every combination a maven module.
> > Pros:
> > * Avaliable right away.
> > Cons:
> > * Will result in many modules - e.g. smoke-web, smoke-full,
> > basic-web, basic-full, ...
> > * => much harder to maintain, esp. for the product features (need to
> > distribute stuff to too many pom.xml's).
> > * Slower testsuite and AS7 build.
> > * Would not scale for the cases when each test needs a different
> > config.
> > 
> > C) Optimistic engineer's solution: Use JUnit's new @Category and
> > Arquillians ability to restart a server with other config, based on
> > @ContainerConfig("path-to-config.xml").
> > Pros:
> > * Simple, clean approach; no pom.xml hell.
> > * Easy to maintain.
> > * Easy to reorganize later.
> > * Would get 3) and 4) out of maven
> > Cons:
> > * Arq's impl of @Category is not tested
> > * We would need to put @Category to all tests
> > * @ContainerConfig("path-to-config.xml") is not implemented. We could
> > (mis)use @TargetsContainer and @OperatesOnDeployment.
> > 
> > Are there others?
> > Did I miss some pros or cons?
> > Which one do you like?
> > 
> > Ondra
> > 
> > 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-as7-dev/attachments/20111209/41a21044/attachment.html