[jboss-as7-dev] Testsuite grouping: Splitting to many modules, or something different?

[Ondřej Žižka]

Ondřej Žižka píše v Pá 09. 12. 2011 v 19:10 +0100:

> Aslak Knutsen píše v Pá 02. 12. 2011 v 04:34 -0500: 
> 
> > I think you missed the most obvious one.. 
> > 
> > E) Fix Maven Execution / Surefire
> 
> Actually I didn't,  http://jira.codehaus.org/browse/SUREFIRE-803

Just to be clear - we would still need some better grouping mechanism,
see my today's new thread on that.



> Discussed with John Casey (CC).
> 
> John, do you think this could be fixed until CR1?  (2011-12-21)
> 
> Thanks,
> Ondra
> 
> 
> 
> 
> 
> > 
> > -aslak-
> > 
> > 
> > ----- Original Message -----
> > > Hi everyone interested,
> > > 
> > > there are these motivations for grouping the tests:
> > > 
> > > 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> > > 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> > > 3) Running tests with different configs.
> > > 4) Running various groups of tests.
> > > 
> > > 4) can be done using e.g.
> > > -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
> > > -Dtest works fine), so I'll drop it from further considerations).
> > > 
> > > 1-3) is currently done by a combination of modules and Surefire
> > > executions.
> > > The problem with the later is that a failure in one execution
> > > prevents the successive from running.
> > > 
> > > Here are options how that can be solved:
> > > 
> > > A) Naive solution: Keep status quo, keep tests in the first execution
> > > in a good condition.
> > > Pros: * We already have it.
> > > Cons: * It will fail now and then, and people would need an extra run
> > > for the other group, using various params.
> > > 
> > > B) Megalomanic solution: Make every combination a maven module.
> > > Pros:
> > > * Avaliable right away.
> > > Cons:
> > > * Will result in many modules - e.g. smoke-web, smoke-full,
> > > basic-web, basic-full, ...
> > > * => much harder to maintain, esp. for the product features (need to
> > > distribute stuff to too many pom.xml's).
> > > * Slower testsuite and AS7 build.
> > > * Would not scale for the cases when each test needs a different
> > > config.
> > > 
> > > C) Optimistic engineer's solution: Use JUnit's new @Category and
> > > Arquillians ability to restart a server with other config, based on
> > > @ContainerConfig("path-to-config.xml").
> > > Pros:
> > > * Simple, clean approach; no pom.xml hell.
> > > * Easy to maintain.
> > > * Easy to reorganize later.
> > > * Would get 3) and 4) out of maven
> > > Cons:
> > > * Arq's impl of @Category is not tested
> > > * We would need to put @Category to all tests
> > > * @ContainerConfig("path-to-config.xml") is not implemented. We could
> > > (mis)use @TargetsContainer and @OperatesOnDeployment.
> > > 
> > > Are there others?
> > > Did I miss some pros or cons?
> > > Which one do you like?
> > > 
> > > Ondra
> > > 
> > > 
> 
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-as7-dev/attachments/20111209/4516113b/attachment.html