Sounds good. For 3) something like -DallTests=true vaguely sounds better but I don't know if maven has any equivalent of the horrible ant conditional properties stuff On 15 May 2011, at 03:16, Brian Stansberry wrote: > I'd like to get feedback on changes to the testsuite found in > https://github.com/bstansberry/jboss-as/commit/efe2051753f6b79788d0515678... > > I'll be adding a bunch of integration tests that launch a domain, but > before pushing those I wanted to ensure they weren't run every time > someone did a build. That led to a general effort to simplify how we can > execute the 3 main build cases -- a standard build with some smoke > tests, a no-test fast build, and a full test run. The above commit > results in the following behavior: > > 1) By default, all normal modules and testsuite modules will be built > (so test compilation issues introduced by code changes will be caught) > but the only tests that will be executed are: > > a) the tests in non-testsuite modules > b) the tests in testsuite/smoke > > 2) By adding -DskipTests=true, no tests will be executed (although > they'll still compile). > > 3) By adding -DskipTests=false the integration tests in testsuite2, > testsuite/integration and the webservices module will be enabled. So > that results in a full testsuite run (except for the currently empty > benchmark and stress test modules). > > The testsuite2, testsuite/integration and the webservices module also > have testFailureIgnore set to true, which means failures in one of the > many modules will not fail the build, allowing the full set of tests to > run. A test failure in a non-testsuite module or in testsuite/smoke will > fail the build. > > -- > Brian Stansberry > Principal Software Engineer > JBoss by Red Hat > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

mvn -Dtest=SomeTestCase test

mvn -DskipTests=false -Dtest=SomeTestCase test

I'd like to get feedback on changes to the testsuite found in https://github.com/bstansberry/jboss-as/commit/efe2051753f6b79788d0515678... I'll be adding a bunch of integration tests that launch a domain, but before pushing those I wanted to ensure they weren't run every time someone did a build. That led to a general effort to simplify how we can execute the 3 main build cases -- a standard build with some smoke tests, a no-test fast build, and a full test run. The above commit results in the following behavior: 1) By default, all normal modules and testsuite modules will be built (so test compilation issues introduced by code changes will be caught) but the only tests that will be executed are: a) the tests in non-testsuite modules b) the tests in testsuite/smoke 2) By adding -DskipTests=true, no tests will be executed (although they'll still compile). 3) By adding -DskipTests=false the integration tests in testsuite2, testsuite/integration and the webservices module will be enabled. So that results in a full testsuite run (except for the currently empty benchmark and stress test modules). The testsuite2, testsuite/integration and the webservices module also have testFailureIgnore set to true, which means failures in one of the many modules will not fail the build, allowing the full set of tests to run. A test failure in a non-testsuite module or in testsuite/smoke will fail the build.

That's fine by me. I don't care all that much about compiling the integration tests. I didn't want -P allTests because that turns off default profiles, but using -DallTests to activate a profile shouldn't have that problem. On 5/18/11 10:13 AM, Thomas Diesler wrote: > Using skipTests as a flag has a negative side effect such that you > cannot do > > > mvn -Dtest=SomeTestCase test > > in an integration module any more. Instead you have to do > > > mvn -DskipTests=false -Dtest=SomeTestCase test > > This is non-intuitive and non-standard. Instead I propose to use an > alternate switch like 'allTests' to turn on profiles which run the > complete set of tests. IMHO its ok to exclude entire (testsuite) modules > from the build (i.e. not to compile the tests) because compilation > errors show up in the IDE and Hudson anyway. > > I propose > > 1) By default, all normal modules and testsuite/smoke will be built in > the standard way > (i.e. failing tests will fail the build) > > 2) By adding -DskipTests=true tests that would normally be executed are > skipped > > 3) By adding -DallTests all modules that contain tests will be executed. > In Hudson you could add -fae > which fails the build at the end - so catch all failing tests in a > single run > > It boils down to whether you want to compile tests that you don't want > to execute. > IMHO that's not necessary in day to day work - Hudson can catch this. > > cheers > -thomas > > > > On 05/15/2011 04:16 AM, Brian Stansberry wrote: >> I'd like to get feedback on changes to the testsuite found in >> https://github.com/bstansberry/jboss-as/commit/efe2051753f6b79788d0515678... >> >> >> I'll be adding a bunch of integration tests that launch a domain, but >> before pushing those I wanted to ensure they weren't run every time >> someone did a build. That led to a general effort to simplify how we can >> execute the 3 main build cases -- a standard build with some smoke >> tests, a no-test fast build, and a full test run. The above commit >> results in the following behavior: >> >> 1) By default, all normal modules and testsuite modules will be built >> (so test compilation issues introduced by code changes will be caught) >> but the only tests that will be executed are: >> >> a) the tests in non-testsuite modules >> b) the tests in testsuite/smoke >> >> 2) By adding -DskipTests=true, no tests will be executed (although >> they'll still compile). >> >> 3) By adding -DskipTests=false the integration tests in testsuite2, >> testsuite/integration and the webservices module will be enabled. So >> that results in a full testsuite run (except for the currently empty >> benchmark and stress test modules). >> >> The testsuite2, testsuite/integration and the webservices module also >> have testFailureIgnore set to true, which means failures in one of the >> many modules will not fail the build, allowing the full set of tests to >> run. A test failure in a non-testsuite module or in testsuite/smoke will >> fail the build. >> >

No new tests should be going into testsuite/integration; existing ones should be moved out. On 5/18/11 11:58 AM, Jaikiran Pai wrote: > By the way, are we still adding tests to the (outdated?) > testsuite/integration module? I thought we were going to start using > testsuite2/spec and testsuite2/internals going forward. Where should any > new tests go? > > -Jaikiran > On Wednesday 18 May 2011 09:35 PM, Brian Stansberry wrote: >> That's fine by me. I don't care all that much about compiling the >> integration tests. I didn't want -P allTests because that turns off >> default profiles, but using -DallTests to activate a profile shouldn't >> have that problem. >> >> On 5/18/11 10:13 AM, Thomas Diesler wrote: >>> Using skipTests as a flag has a negative side effect such that you >>> cannot do >>> >>> > mvn -Dtest=SomeTestCase test >>> >>> in an integration module any more. Instead you have to do >>> >>> > mvn -DskipTests=false -Dtest=SomeTestCase test >>> >>> This is non-intuitive and non-standard. Instead I propose to use an >>> alternate switch like 'allTests' to turn on profiles which run the >>> complete set of tests. IMHO its ok to exclude entire (testsuite) modules >>> from the build (i.e. not to compile the tests) because compilation >>> errors show up in the IDE and Hudson anyway. >>> >>> I propose >>> >>> 1) By default, all normal modules and testsuite/smoke will be built in >>> the standard way >>> (i.e. failing tests will fail the build) >>> >>> 2) By adding -DskipTests=true tests that would normally be executed are >>> skipped >>> >>> 3) By adding -DallTests all modules that contain tests will be executed. >>> In Hudson you could add -fae >>> which fails the build at the end - so catch all failing tests in a >>> single run >>> >>> It boils down to whether you want to compile tests that you don't want >>> to execute. >>> IMHO that's not necessary in day to day work - Hudson can catch this. >>> >>> cheers >>> -thomas >>> >>> >>> >>> On 05/15/2011 04:16 AM, Brian Stansberry wrote: >>>> I'd like to get feedback on changes to the testsuite found in >>>> https://github.com/bstansberry/jboss-as/commit/efe2051753f6b79788d0515678... >>>> >>>> >>>> I'll be adding a bunch of integration tests that launch a domain, but >>>> before pushing those I wanted to ensure they weren't run every time >>>> someone did a build. That led to a general effort to simplify how we can >>>> execute the 3 main build cases -- a standard build with some smoke >>>> tests, a no-test fast build, and a full test run. The above commit >>>> results in the following behavior: >>>> >>>> 1) By default, all normal modules and testsuite modules will be built >>>> (so test compilation issues introduced by code changes will be caught) >>>> but the only tests that will be executed are: >>>> >>>> a) the tests in non-testsuite modules >>>> b) the tests in testsuite/smoke >>>> >>>> 2) By adding -DskipTests=true, no tests will be executed (although >>>> they'll still compile). >>>> >>>> 3) By adding -DskipTests=false the integration tests in testsuite2, >>>> testsuite/integration and the webservices module will be enabled. So >>>> that results in a full testsuite run (except for the currently empty >>>> benchmark and stress test modules). >>>> >>>> The testsuite2, testsuite/integration and the webservices module also >>>> have testFailureIgnore set to true, which means failures in one of the >>>> many modules will not fail the build, allowing the full set of tests to >>>> run. A test failure in a non-testsuite module or in testsuite/smoke will >>>> fail the build. >>>> >> > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 05/15/2011 04:16 AM, Brian Stansberry wrote: > The testsuite2, testsuite/integration and the webservices module also > have testFailureIgnore set to true, which means failures in one of the > many modules will not fail the build, allowing the full set of tests to > run. A test failure in a non-testsuite module or in testsuite/smoke will > fail the build. > That's bad. We had this before and it gives false positives. Why not mvn -fae ?