[jboss-as7-dev] Testsuite execution

[Thomas Diesler]

Using skipTests as a flag has a negative side effect such that you cannot do

 > mvn -Dtest=SomeTestCase test

in an integration module any more. Instead you have to do

 > mvn -DskipTests=false -Dtest=SomeTestCase test

This is non-intuitive and non-standard. Instead I propose to use an 
alternate switch like 'allTests' to turn on profiles which run the 
complete set of tests. IMHO its ok to exclude entire (testsuite) modules 
from the build (i.e. not to compile the tests) because compilation 
errors show up in the IDE and Hudson anyway.

I propose

1) By default, all normal modules and testsuite/smoke will be built in 
the standard way
(i.e. failing tests will fail the build)

2) By adding -DskipTests=true tests that would normally be executed are skipped

3) By adding -DallTests all modules that contain tests will be executed. In Hudson you could add -fae
which fails the build at the end - so catch all failing tests in a single run

It boils down to whether you want to compile tests that you don't want to execute.
IMHO that's not necessary in day to day work - Hudson can catch this.

cheers
-thomas



On 05/15/2011 04:16 AM, Brian Stansberry wrote:
> I'd like to get feedback on changes to the testsuite found in
> https://github.com/bstansberry/jboss-as/commit/efe2051753f6b79788d05156784665e02672613c
>
> I'll be adding a bunch of integration tests that launch a domain, but
> before pushing those I wanted to ensure they weren't run every time
> someone did a build. That led to a general effort to simplify how we can
> execute the 3 main build cases -- a standard build with some smoke
> tests, a no-test fast build, and a full test run. The above commit
> results in the following behavior:
>
> 1) By default, all normal modules and testsuite modules will be built
> (so test compilation issues introduced by code changes will be caught)
> but the only tests that will be executed are:
>
> a) the tests in non-testsuite modules
> b) the tests in testsuite/smoke
>
> 2) By adding -DskipTests=true, no tests will be executed (although
> they'll still compile).
>
> 3) By adding -DskipTests=false the integration tests in testsuite2,
> testsuite/integration and the webservices module will be enabled. So
> that results in a full testsuite run (except for the currently empty
> benchmark and stress test modules).
>
> The testsuite2, testsuite/integration and the webservices module also
> have testFailureIgnore set to true, which means failures in one of the
> many modules will not fail the build, allowing the full set of tests to
> run. A test failure in a non-testsuite module or in testsuite/smoke will
> fail the build.
>

-- 
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx