6:47 a.m.
Hi everyone interested,
there are these motivations for grouping the tests:
1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
3) Running tests with different configs.
4) Running various groups of tests.
4) can be done using e.g.
-Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
works fine), so I'll drop it from further considerations).
1-3) is currently done by a combination of modules and Surefire
executions.
The problem with the later is that a failure in one execution prevents
the successive from running.
Here are options how that can be solved:
A) Naive solution: Keep status quo, keep tests in the first execution in
a good condition.
Pros: * We already have it.
Cons: * It will fail now and then, and people would need an extra run
for the other group, using various params.
B) Megalomanic solution: Make every combination a maven module.
Pros:
* Avaliable right away.
Cons:
* Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
basic-full, ...
* => much harder to maintain, esp. for the product features (need to
distribute stuff to too many pom.xml's).
* Slower testsuite and AS7 build.
* Would not scale for the cases when each test needs a different
config.
C) Optimistic engineer's solution: Use JUnit's new @Category and
Arquillians ability to restart a server with other config, based on
@ContainerConfig("path-to-config.xml").
Pros:
* Simple, clean approach; no pom.xml hell.
* Easy to maintain.
* Easy to reorganize later.
* Would get 3) and 4) out of maven
Cons:
* Arq's impl of @Category is not tested
* We would need to put @Category to all tests
* @ContainerConfig("path-to-config.xml") is not implemented. We could
(mis)use @TargetsContainer and @OperatesOnDeployment.
Are there others?
Did I miss some pros or cons?
Which one do you like?
Ondra
8:49 a.m.
Good post - thanks Ondrej
how about creating a maven module per purpose (i.e. smoke, basic,
stress, etc). Each module would have the ability to create its own
config via xslt transform. Selecting tests by functional area would work
via package selection (i.e. -Dtest=jms/**/*TestCase). This means I can
run a subset of tests in any given module. The subset can be as small as
one single test.
If a subset of tests must be run with a different config, we could have
a profile per config, which could be activated by -Dconfig=foo. The
default profile would be activated by not having a -Dconfig parameter.
Each profile has the ability to do config transformation.
With this setup you would not have multiple surefire executions. The
large set of integration tests would live in one module (i.e. basic) and
be executed together by default. This module could contain any number of
specialized profiles which select arbitrary subsets with arbitrary configs.
cheers
-thomas
On 12/02/2011 06:47 AM, Ondřej Žižka wrote:
Hi everyone interested,
there are these motivations for grouping the tests:
1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
3) Running tests with different configs.
4) Running various groups of tests.
4) can be done using e.g.
-Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
-Dtest works fine), so I'll drop it from further considerations).
1-3) is currently done by a combination of modules and Surefire
executions.
The problem with the later is that a failure in one execution prevents
the successive from running.
Here are options how that can be solved:
*A) Naive solution: Keep status quo, keep tests in the first execution
in a good condition.*
Pros: * We already have it.
Cons: * It will fail now and then, and people would need an extra
run for the other group, using various params.
*B) Megalomanic solution: Make every combination a maven module.*
Pros:
* Avaliable right away.
Cons:
* Will result in many modules - e.g. smoke-web, smoke-full,
basic-web, basic-full, ...
* => much harder to maintain, esp. for the product features (need to
distribute stuff to too many pom.xml's).
* Slower testsuite and AS7 build.
* Would not scale for the cases when each test needs a different config.
*C) Optimistic engineer's solution: Use JUnit's new @Category and
Arquillians ability to restart a server with other config, based on
@ContainerConfig("path-to-config.xml").*
Pros:
* Simple, clean approach; no pom.xml hell.
* Easy to maintain.
* Easy to reorganize later.
* Would get 3) and 4) out of maven
Cons:
* Arq's impl of @Category is not tested
* We would need to put @Category to all tests
* @ContainerConfig("path-to-config.xml") is not implemented. We
could (mis)use @TargetsContainer and @OperatesOnDeployment.
*Are there others?*
*Did I miss some pros or cons?*
*Which one do you like?*
Ondra
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
10:51 a.m.
One other topic that has come up recently is interoperability with
previous AS releases, from the discussions so far I don't think now is
going to be the time to address it but it could be worth considering how
it would at least fit.
I wonder if a single module concentrating on the management interactions
needed for interoperability, that module could then be manually run
against remote AS instances of previous versions to verify nothing is
broken.
Regards,
Darran Lofthouse.
On 12/02/2011 07:49 AM, Thomas Diesler wrote:
Good post - thanks Ondrej
how about creating a maven module per purpose (i.e. smoke, basic,
stress, etc). Each module would have the ability to create its own
config via xslt transform. Selecting tests by functional area would work
via package selection (i.e. -Dtest=jms/**/*TestCase). This means I can
run a subset of tests in any given module. The subset can be as small as
one single test.
If a subset of tests must be run with a different config, we could have
a profile per config, which could be activated by -Dconfig=foo. The
default profile would be activated by not having a -Dconfig parameter.
Each profile has the ability to do config transformation.
With this setup you would not have multiple surefire executions. The
large set of integration tests would live in one module (i.e. basic) and
be executed together by default. This module could contain any number of
specialized profiles which select arbitrary subsets with arbitrary configs.
cheers
-thomas
On 12/02/2011 06:47 AM, Ondřej Žižka wrote:
> Hi everyone interested,
>
> there are these motivations for grouping the tests:
>
> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> 3) Running tests with different configs.
> 4) Running various groups of tests.
>
> 4) can be done using e.g.
> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
> -Dtest works fine), so I'll drop it from further considerations).
>
> 1-3) is currently done by a combination of modules and Surefire
> executions.
> The problem with the later is that a failure in one execution prevents
> the successive from running.
>
> Here are options how that can be solved:
>
> *A) Naive solution: Keep status quo, keep tests in the first execution
> in a good condition.*
> Pros: * We already have it.
> Cons: * It will fail now and then, and people would need an extra
> run for the other group, using various params.
>
> *B) Megalomanic solution: Make every combination a maven module.*
> Pros:
> * Avaliable right away.
> Cons:
> * Will result in many modules - e.g. smoke-web, smoke-full,
> basic-web, basic-full, ...
> * => much harder to maintain, esp. for the product features (need to
> distribute stuff to too many pom.xml's).
> * Slower testsuite and AS7 build.
> * Would not scale for the cases when each test needs a different config.
>
> *C) Optimistic engineer's solution: Use JUnit's new @Category and
> Arquillians ability to restart a server with other config, based on
> @ContainerConfig("path-to-config.xml").*
> Pros:
> * Simple, clean approach; no pom.xml hell.
> * Easy to maintain.
> * Easy to reorganize later.
> * Would get 3) and 4) out of maven
> Cons:
> * Arq's impl of @Category is not tested
> * We would need to put @Category to all tests
> * @ContainerConfig("path-to-config.xml") is not implemented. We
> could (mis)use @TargetsContainer and @OperatesOnDeployment.
>
> *Are there others?*
> *Did I miss some pros or cons?*
> *Which one do you like?*
>
> Ondra
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
--
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Thomas Diesler
JBoss OSGi Lead
JBoss, a division of Red Hat
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
10:34 a.m.
I think you missed the most obvious one..
E) Fix Maven Execution / Surefire
-aslak-
----- Original Message -----
Hi everyone interested,
there are these motivations for grouping the tests:
1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
3) Running tests with different configs.
4) Running various groups of tests.
4) can be done using e.g.
-Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
-Dtest works fine), so I'll drop it from further considerations).
1-3) is currently done by a combination of modules and Surefire
executions.
The problem with the later is that a failure in one execution
prevents the successive from running.
Here are options how that can be solved:
A) Naive solution: Keep status quo, keep tests in the first execution
in a good condition.
Pros: * We already have it.
Cons: * It will fail now and then, and people would need an extra run
for the other group, using various params.
B) Megalomanic solution: Make every combination a maven module.
Pros:
* Avaliable right away.
Cons:
* Will result in many modules - e.g. smoke-web, smoke-full,
basic-web, basic-full, ...
* => much harder to maintain, esp. for the product features (need to
distribute stuff to too many pom.xml's).
* Slower testsuite and AS7 build.
* Would not scale for the cases when each test needs a different
config.
C) Optimistic engineer's solution: Use JUnit's new @Category and
Arquillians ability to restart a server with other config, based on
@ContainerConfig("path-to-config.xml").
Pros:
* Simple, clean approach; no pom.xml hell.
* Easy to maintain.
* Easy to reorganize later.
* Would get 3) and 4) out of maven
Cons:
* Arq's impl of @Category is not tested
* We would need to put @Category to all tests
* @ContainerConfig("path-to-config.xml") is not implemented. We could
(mis)use @TargetsContainer and @OperatesOnDeployment.
Are there others?
Did I miss some pros or cons?
Which one do you like?
Ondra
7:10 p.m.
New subject: Testsuite grouping: Splitting to many modules, or something different?
Aslak Knutsen píše v Pá 02. 12. 2011 v 04:34 -0500:
I think you missed the most obvious one..
E) Fix Maven Execution / Surefire
Actually I didn't, http://jira.codehaus.org/browse/SUREFIRE-803
Discussed with John Casey (CC).
John, do you think this could be fixed until CR1? (2011-12-21)
Thanks,
Ondra
-aslak-
----- Original Message -----
> Hi everyone interested,
>
> there are these motivations for grouping the tests:
>
> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> 3) Running tests with different configs.
> 4) Running various groups of tests.
>
> 4) can be done using e.g.
> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
> -Dtest works fine), so I'll drop it from further considerations).
>
> 1-3) is currently done by a combination of modules and Surefire
> executions.
> The problem with the later is that a failure in one execution
> prevents the successive from running.
>
> Here are options how that can be solved:
>
> A) Naive solution: Keep status quo, keep tests in the first execution
> in a good condition.
> Pros: * We already have it.
> Cons: * It will fail now and then, and people would need an extra run
> for the other group, using various params.
>
> B) Megalomanic solution: Make every combination a maven module.
> Pros:
> * Avaliable right away.
> Cons:
> * Will result in many modules - e.g. smoke-web, smoke-full,
> basic-web, basic-full, ...
> * => much harder to maintain, esp. for the product features (need to
> distribute stuff to too many pom.xml's).
> * Slower testsuite and AS7 build.
> * Would not scale for the cases when each test needs a different
> config.
>
> C) Optimistic engineer's solution: Use JUnit's new @Category and
> Arquillians ability to restart a server with other config, based on
> @ContainerConfig("path-to-config.xml").
> Pros:
> * Simple, clean approach; no pom.xml hell.
> * Easy to maintain.
> * Easy to reorganize later.
> * Would get 3) and 4) out of maven
> Cons:
> * Arq's impl of @Category is not tested
> * We would need to put @Category to all tests
> * @ContainerConfig("path-to-config.xml") is not implemented. We could
> (mis)use @TargetsContainer and @OperatesOnDeployment.
>
> Are there others?
> Did I miss some pros or cons?
> Which one do you like?
>
> Ondra
>
>
7:16 p.m.
New subject: Testsuite grouping: Splitting to many modules, or something different?
Ondřej Žižka píše v Pá 09. 12. 2011 v 19:10 +0100:
Aslak Knutsen píše v Pá 02. 12. 2011 v 04:34 -0500:
> I think you missed the most obvious one..
>
> E) Fix Maven Execution / Surefire
Actually I didn't, http://jira.codehaus.org/browse/SUREFIRE-803
Just to be clear - we would still need some better grouping mechanism,
see my today's new thread on that.
Discussed with John Casey (CC).
John, do you think this could be fixed until CR1? (2011-12-21)
Thanks,
Ondra
>
> -aslak-
>
>
> ----- Original Message -----
> > Hi everyone interested,
> >
> > there are these motivations for grouping the tests:
> >
> > 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> > 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> > 3) Running tests with different configs.
> > 4) Running various groups of tests.
> >
> > 4) can be done using e.g.
> > -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided
> > -Dtest works fine), so I'll drop it from further considerations).
> >
> > 1-3) is currently done by a combination of modules and Surefire
> > executions.
> > The problem with the later is that a failure in one execution
> > prevents the successive from running.
> >
> > Here are options how that can be solved:
> >
> > A) Naive solution: Keep status quo, keep tests in the first execution
> > in a good condition.
> > Pros: * We already have it.
> > Cons: * It will fail now and then, and people would need an extra run
> > for the other group, using various params.
> >
> > B) Megalomanic solution: Make every combination a maven module.
> > Pros:
> > * Avaliable right away.
> > Cons:
> > * Will result in many modules - e.g. smoke-web, smoke-full,
> > basic-web, basic-full, ...
> > * => much harder to maintain, esp. for the product features (need to
> > distribute stuff to too many pom.xml's).
> > * Slower testsuite and AS7 build.
> > * Would not scale for the cases when each test needs a different
> > config.
> >
> > C) Optimistic engineer's solution: Use JUnit's new @Category and
> > Arquillians ability to restart a server with other config, based on
> > @ContainerConfig("path-to-config.xml").
> > Pros:
> > * Simple, clean approach; no pom.xml hell.
> > * Easy to maintain.
> > * Easy to reorganize later.
> > * Would get 3) and 4) out of maven
> > Cons:
> > * Arq's impl of @Category is not tested
> > * We would need to put @Category to all tests
> > * @ContainerConfig("path-to-config.xml") is not implemented. We
could
> > (mis)use @TargetsContainer and @OperatesOnDeployment.
> >
> > Are there others?
> > Did I miss some pros or cons?
> > Which one do you like?
> >
> > Ondra
> >
> >
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
4:57 p.m.
Until 7.1 is out, I want A).
I am tired of testsuite refactors; they have cost me far more
productivity than dealing with failures has.
IMHO we don't have time on the CR1 schedule to be significantly
refactoring the testsuite.
On 12/1/11 11:47 PM, Ondřej Žižka wrote:
Hi everyone interested,
there are these motivations for grouping the tests:
1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
3) Running tests with different configs.
4) Running various groups of tests.
4) can be done using e.g.
-Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
works fine), so I'll drop it from further considerations).
1-3) is currently done by a combination of modules and Surefire executions.
The problem with the later is that a failure in one execution prevents
the successive from running.
Here are options how that can be solved:
*A) Naive solution: Keep status quo, keep tests in the first execution
in a good condition.*
Pros: * We already have it.
Cons: * It will fail now and then, and people would need an extra run
for the other group, using various params.
*B) Megalomanic solution: Make every combination a maven module.*
Pros:
* Avaliable right away.
Cons:
* Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
basic-full, ...
* => much harder to maintain, esp. for the product features (need to
distribute stuff to too many pom.xml's).
* Slower testsuite and AS7 build.
* Would not scale for the cases when each test needs a different config.
*C) Optimistic engineer's solution: Use JUnit's new @Category and
Arquillians ability to restart a server with other config, based on
@ContainerConfig("path-to-config.xml").*
Pros:
* Simple, clean approach; no pom.xml hell.
* Easy to maintain.
* Easy to reorganize later.
* Would get 3) and 4) out of maven
Cons:
* Arq's impl of @Category is not tested
* We would need to put @Category to all tests
* @ContainerConfig("path-to-config.xml") is not implemented. We could
(mis)use @TargetsContainer and @OperatesOnDeployment.
*Are there others?*
*Did I miss some pros or cons?*
*Which one do you like?*
Ondra
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
5:15 p.m.
On 12/1/11 11:47 PM, Ondřej Žižka wrote:
Hi everyone interested,
there are these motivations for grouping the tests:
1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
3) Running tests with different configs.
4) Running various groups of tests.
4) can be done using e.g.
-Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
works fine), so I'll drop it from further considerations).
1-3) is currently done by a combination of modules and Surefire executions.
The problem with the later is that a failure in one execution prevents
the successive from running.
Here are options how that can be solved:
*A) Naive solution: Keep status quo, keep tests in the first execution
in a good condition.*
Pros: * We already have it.
Cons: * It will fail now and then, and people would need an extra run
for the other group, using various params.
*B) Megalomanic solution: Make every combination a maven module.*
Pros:
* Avaliable right away.
Cons:
* Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
basic-full, ...
I don't think we need MxN modules that accounts for [2/3]. That can just
be an optional property that allows the entire testsuite to use a
different configuration. Note that the default only runs "full" against
things that need full, so the default runs fully test the default. You
only theoretically need one extra run to verify the entire full config,
and it's something you don't have to do very often (can just be a
one-off hudson run near a release).
IMO If we have multiple modules it should just be one per functional
area (approximation of a subsystem). That's it. We don't even need one
for every subsystem, and even if we did we only have 20 subsystems.
The real advantage is that its just clean organization, clean layout and
it's common maven behavior. Yes there is some copying of pom.xml files,
but that's just boilerplate, and you can script updates like we already
have to do.
Also note, that I'm not arguing for 20 modules right now. What I am
saying is that we can take the 3 or so that require full and make them
modules right now, and then later, over time migrate when needed.
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
5:20 p.m.
On 12/2/11 10:15 AM, Jason T. Greene wrote:
On 12/1/11 11:47 PM, Ondřej Žižka wrote:
> Hi everyone interested,
>
> there are these motivations for grouping the tests:
>
> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
> 3) Running tests with different configs.
> 4) Running various groups of tests.
>
> 4) can be done using e.g.
> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
> works fine), so I'll drop it from further considerations).
>
> 1-3) is currently done by a combination of modules and Surefire executions.
> The problem with the later is that a failure in one execution prevents
> the successive from running.
>
> Here are options how that can be solved:
>
> *A) Naive solution: Keep status quo, keep tests in the first execution
> in a good condition.*
> Pros: * We already have it.
> Cons: * It will fail now and then, and people would need an extra run
> for the other group, using various params.
>
> *B) Megalomanic solution: Make every combination a maven module.*
> Pros:
> * Avaliable right away.
> Cons:
> * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
> basic-full, ...
I don't think we need MxN modules that accounts for [2/3]. That can just
be an optional property that allows the entire testsuite to use a
different configuration. Note that the default only runs "full" against
things that need full, so the default runs fully test the default. You
only theoretically need one extra run to verify the entire full config,
and it's something you don't have to do very often (can just be a
one-off hudson run near a release).
IMO If we have multiple modules it should just be one per functional
area (approximation of a subsystem). That's it. We don't even need one
for every subsystem, and even if we did we only have 20 subsystems.
The real advantage is that its just clean organization, clean layout and
it's common maven behavior. Yes there is some copying of pom.xml files,
but that's just boilerplate, and you can script updates like we already
have to do.
Also note, that I'm not arguing for 20 modules right now. What I am
saying is that we can take the 3 or so that require full and make them
modules right now, and then later, over time migrate when needed.
I argued in a separate message against significant refactors prior to
CR1. I don't regard what you're described in this last paragraph as a
significant refactor.
--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
5:22 p.m.
On 12/2/11 10:20 AM, Brian Stansberry wrote:
On 12/2/11 10:15 AM, Jason T. Greene wrote:
> On 12/1/11 11:47 PM, Ondřej Žižka wrote:
>> Hi everyone interested,
>>
>> there are these motivations for grouping the tests:
>>
>> 1) Splitting tests by functional area, e.g. jms, jacorb, cmp.
>> 2) Splitting tests by purpose, e.g. smoke, basic, stress etc.
>> 3) Running tests with different configs.
>> 4) Running various groups of tests.
>>
>> 4) can be done using e.g.
>> -Dtest=org.jboss.as.test.integration.ejb.*TestCase.java (provided -Dtest
>> works fine), so I'll drop it from further considerations).
>>
>> 1-3) is currently done by a combination of modules and Surefire executions.
>> The problem with the later is that a failure in one execution prevents
>> the successive from running.
>>
>> Here are options how that can be solved:
>>
>> *A) Naive solution: Keep status quo, keep tests in the first execution
>> in a good condition.*
>> Pros: * We already have it.
>> Cons: * It will fail now and then, and people would need an extra run
>> for the other group, using various params.
>>
>> *B) Megalomanic solution: Make every combination a maven module.*
>> Pros:
>> * Avaliable right away.
>> Cons:
>> * Will result in many modules - e.g. smoke-web, smoke-full, basic-web,
>> basic-full, ...
>
> I don't think we need MxN modules that accounts for [2/3]. That can just
> be an optional property that allows the entire testsuite to use a
> different configuration. Note that the default only runs "full" against
> things that need full, so the default runs fully test the default. You
> only theoretically need one extra run to verify the entire full config,
> and it's something you don't have to do very often (can just be a
> one-off hudson run near a release).
>
> IMO If we have multiple modules it should just be one per functional
> area (approximation of a subsystem). That's it. We don't even need one
> for every subsystem, and even if we did we only have 20 subsystems.
>
> The real advantage is that its just clean organization, clean layout and
> it's common maven behavior. Yes there is some copying of pom.xml files,
> but that's just boilerplate, and you can script updates like we already
> have to do.
>
> Also note, that I'm not arguing for 20 modules right now. What I am
> saying is that we can take the 3 or so that require full and make them
> modules right now, and then later, over time migrate when needed.
>
I argued in a separate message against significant refactors prior to
CR1. I don't regard what you're described in this last paragraph as a
significant refactor.
BTW I should note that I can live with the two execution status quo.
It's not great but it does work (mostly).
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
10:45 p.m.
On 12/2/11 10:22 AM, Jason T. Greene wrote:
On 12/2/11 10:20 AM, Brian Stansberry wrote:
> On 12/2/11 10:15 AM, Jason T. Greene wrote:
>> Also note, that I'm not arguing for 20 modules right now.
What I am
>> saying is that we can take the 3 or so that require full and make them
>> modules right now, and then later, over time migrate when needed.
>>
>
> I argued in a separate message against significant refactors prior to
> CR1. I don't regard what you're described in this last paragraph as a
> significant refactor.
>
BTW I should note that I can live with the two execution status quo.
It's not great but it does work (mostly).
Another problem I just discovered with multiple executions, is that
hudson test results using the maven plugin are masked. So we really
should move off of this at our earliest convenience.
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
1:29 p.m.
Hi Jason.
Use reportNameSuffix [1] for surefire configuration in your module or execution.
General solution would be to define empty property <surefire.default.reportNameSuffix
/> and add
<reportNameSuffix>${surefire.default.reportNameSuffix}</reportNameSuffix> into
maven-surefire-plugin configuration in testsuite/pom.xml. In your module or execution you
can just redefine the property. There is no filename change when reportNameSuffix is null
or empty.
Cheers.
Rosta
[1] http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html#repo...
----- Original Message -----
On 12/2/11 10:22 AM, Jason T. Greene wrote:
> On 12/2/11 10:20 AM, Brian Stansberry wrote:
>> On 12/2/11 10:15 AM, Jason T. Greene wrote:
>>> Also note, that I'm not arguing for 20 modules right now. What I
>>> am
>>> saying is that we can take the 3 or so that require full and make
>>> them
>>> modules right now, and then later, over time migrate when needed.
>>>
>>
>> I argued in a separate message against significant refactors prior
>> to
>> CR1. I don't regard what you're described in this last paragraph
>> as a
>> significant refactor.
>>
>
> BTW I should note that I can live with the two execution status
> quo.
> It's not great but it does work (mostly).
>
Another problem I just discovered with multiple executions, is that
hudson test results using the maven plugin are masked. So we really
should move off of this at our earliest convenience.
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
3:37 p.m.
Hello Rotislav,
I am referring to a different issue than an overwrite condition. The
problem is that jenkins only pays attention to the "default" execution,
so it does not read the additional surefire reports. I can workaround
this by just doing a scripted build like QA does, but the maven
integration is quite a bit nicer.
On 12/7/11 6:29 AM, Rostislav Svoboda wrote:
Hi Jason.
Use reportNameSuffix [1] for surefire configuration in your module or execution.
General solution would be to define empty property<surefire.default.reportNameSuffix
/> and
add<reportNameSuffix>${surefire.default.reportNameSuffix}</reportNameSuffix>
into maven-surefire-plugin configuration in testsuite/pom.xml. In your module or execution
you can just redefine the property. There is no filename change when reportNameSuffix is
null or empty.
Cheers.
Rosta
[1]
http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html#repo...
----- Original Message -----
> On 12/2/11 10:22 AM, Jason T. Greene wrote:
>> On 12/2/11 10:20 AM, Brian Stansberry wrote:
>>> On 12/2/11 10:15 AM, Jason T. Greene wrote:
>
>>>> Also note, that I'm not arguing for 20 modules right now. What I
>>>> am
>>>> saying is that we can take the 3 or so that require full and make
>>>> them
>>>> modules right now, and then later, over time migrate when needed.
>>>>
>>>
>>> I argued in a separate message against significant refactors prior
>>> to
>>> CR1. I don't regard what you're described in this last paragraph
>>> as a
>>> significant refactor.
>>>
>>
>> BTW I should note that I can live with the two execution status
>> quo.
>> It's not great but it does work (mostly).
>>
>
> Another problem I just discovered with multiple executions, is that
> hudson test results using the maven plugin are masked. So we really
> should move off of this at our earliest convenience.
>
> --
> Jason T. Greene
> JBoss AS Lead / EAP Platform Architect
> JBoss, a division of Red Hat
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
4:06 p.m.
Hi Jason,
Jenkins simply takes all reports according to a mask.
So if there are
org.foo.bar.MyTest-fullConfig.xml
org.foo.bar.MyTest-webConfig.xml
then both appear in the reports, statistics etc.
Or did you mean something else?
Ondra
Jason T. Greene píše v St 07. 12. 2011 v 08:37 -0600:
problem is that jenkins only pays attention to the
"default"
execution,
so it does not read the additional surefire reports.
5:19 p.m.
New subject: Experience with Maven Failsafe plugin? // Re: Testsuite grouping: Splitting to many modules, or something different?
Hi all,
Do we have any experience with Maven Failsafe plugin?
It's like surefire, but the test process is split to two goals: running
tests and checking the results.
They're bound to different phases by default. Which leads me to:
Would it solve our "two executions" problem? Are there some gotchas?
Additional info from John Casey:
(17:07:11) OndrejZizka: Is new functionality added to surefire
automatically added to failsafe too?
(17:07:17) OndrejZizka: like, groups, for instance
(17:07:27) OndrejZizka: and support for newer junit etc
(17:07:57) jdcasey: you mean code-wise? yeah, I think so
(17:08:21) jdcasey: they made a huge effort not long ago to synchronize
those plugins, and IIRC they're in the same codebase in SVN now
Regards,
Ondra
Ondřej Žižka píše v Pá 02. 12. 2011 v 06:47 +0100:
1-3) is currently done by a combination of modules and Surefire
executions.
The problem with the later is that a failure in one execution prevents
the successive from running.
5:47 p.m.
New subject: Experience with Maven Failsafe plugin? // Re: Testsuite grouping: Splitting to many modules, or something different?
Ondřej Žižka píše v Po 12. 12. 2011 v 17:19 +0100:
Hi all,
Do we have any experience with Maven Failsafe plugin?
It's like surefire, but the test process is split to two goals:
running tests and checking the results.
To save you from some searching:
http://maven.apache.org/plugins/maven-failsafe-plugin/
http://maven.apache.org/plugins/maven-failsafe-plugin/integration-test-mo...
http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html
They're bound to different phases by default. Which leads me to:
Would it solve our "two executions" problem? Are there some gotchas?
Additional info from John Casey:
(17:07:11) OndrejZizka: Is new functionality added to surefire
automatically added to failsafe too?
(17:07:17) OndrejZizka: like, groups, for instance
(17:07:27) OndrejZizka: and support for newer junit etc
(17:07:57) jdcasey: you mean code-wise? yeah, I think so
(17:08:21) jdcasey: they made a huge effort not long ago to
synchronize those plugins, and IIRC they're in the same codebase in
SVN now
Regards,
Ondra
Ondřej Žižka píše v Pá 02. 12. 2011 v 06:47 +0100:
> 1-3) is currently done by a combination of modules and Surefire
> executions.
> The problem with the later is that a failure in one execution
> prevents the successive from running.
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
6:12 p.m.
New subject: Experience with Maven Failsafe plugin? // Re: Testsuite grouping: Splitting to many modules, or something different?
Just found a related comment on
http://jira.codehaus.org/browse/SUREFIRE-803 :
"The current failsafe verification could be modified to consider
results for multiple executions."
That sounds like it would not work as it is now.
See the discussion at jira. More info later.
Ondra
Ondřej Žižka píše v Po 12. 12. 2011 v 17:47 +0100:
Ondřej Žižka píše v Po 12. 12. 2011 v 17:19 +0100:
> Hi all,
>
> Do we have any experience with Maven Failsafe plugin?
> It's like surefire, but the test process is split to two goals:
> running tests and checking the results.
To save you from some searching:
http://maven.apache.org/plugins/maven-failsafe-plugin/
http://maven.apache.org/plugins/maven-failsafe-plugin/integration-test-mo...
http://maven.apache.org/plugins/maven-surefire-plugin/test-mojo.html
> They're bound to different phases by default. Which leads me to:
>
> Would it solve our "two executions" problem? Are there some gotchas?
>
>
> Additional info from John Casey:
>
> (17:07:11) OndrejZizka: Is new functionality added to surefire
> automatically added to failsafe too?
> (17:07:17) OndrejZizka: like, groups, for instance
> (17:07:27) OndrejZizka: and support for newer junit etc
> (17:07:57) jdcasey: you mean code-wise? yeah, I think so
> (17:08:21) jdcasey: they made a huge effort not long ago to
> synchronize those plugins, and IIRC they're in the same codebase in
> SVN now
>
>
> Regards,
> Ondra
>
>
>
> Ondřej Žižka píše v Pá 02. 12. 2011 v 06:47 +0100:
>
>
> > 1-3) is currently done by a combination of modules and Surefire
> > executions.
> > The problem with the later is that a failure in one execution
> > prevents the successive from running.
>
>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
4512
days inactive
4522
days old
16 comments
7 participants
participants (7)
-
Aslak Knutsen -
Brian Stansberry -
Darran Lofthouse -
Jason T. Greene -
Ondřej Žižka -
Rostislav Svoboda -
Thomas Diesler