From asoldano at redhat.com Fri Jun 13 11:39:24 2014 From: asoldano at redhat.com (Alessio Soldano) Date: Fri, 13 Jun 2014 17:39:24 +0200 Subject: [jbossws-dev] JBossWS testsuite reorganization Message-ID: <539B1B2C.4020105@redhat.com> Folks, with current development on jbossws trunk aimed at version 5, I believe it's time consider rethinking the testsuite a bit. Current status -------------------- Historically, we've been running the integration tests against a JBoss AS container that has to be started before the testsuite. The target container is a vanilla distribution that suits with most of the tests, except for few ones requiring a https connector or a specific security domain to be available. Implementations of the org.jboss.wsf.spi.deployer.Deployer interface are available for each target container and allow deploying test archives as well as setting security domains / https connector by internally relying on the container management API. Different maven profiles trigger different implementations of Deployer (due to different artifact dependencies / classpath), allowing running the testsuite the same way against different target containers. When it comes to continuous integration, the idea is that a QA job first updates a target container with the current jbossws libraries, then it starts the container and runs the jbossws testsuite against it. Issues / reasons for changing --------------------------------------- The current approach is working pretty much fine, but suffers from some limitations: * it needs a container to be already running when the integration test is triggered; it would be great if we could have the container started / stopped by the test itself * we're limited to running the testsuite against a predefined container profile (standalone.xml, standalone-full.xml ): while most of the tests makes sense when run against standalone.xml profile (because the webservices subsystem is on in it), we have some tests requiring subsystems available on different profiles (for instance, the messaging subsystem which is in standalone-full.xml or the xts subsystem which is in standalone-xts.xml). Currently we run the testsuite in Spring mode against standalone-full.xml, basically because till CXF 2.7.x the SOAP-over-JMS implementation has been available with Spring only. That's not the case anymore, on CXF 3.0.0, so we'd likely have to test it in both spring and non-spring mode, but that needs the messaging subsystem which is not in standalone.xml * we have few tests (but might have more in future) that actually require further additional changes to the server; that includes adding users, messaging queues, setting server side system properties at boot, ... The only way to achieve that atm is to have the QA job run a CLI script to modify the server or set a system property on command line when starting the server jvm * we have tests affecting the server management model, for instance changing the way the webservices subsystem model rewrites wsdl soap:address attributes; that makes it impossible to run those tests at the same time of any other tests Proposals / idea --------------------- I'm listening to proposals to revisit the way the testsuite is run and better address the point above. Something I've been thinking about is relying on Arquillian, similarly to what the WFLY testsuite does. What's not trivial is the fact that we still need to have different version of the Arquillian SPI implementation (that is, different WFLY / EAP versions) used depending on a maven profile specified when running the testsuite; moreover we also need to be able to run against the result of building the latest WFLY master (iow, not a released target for which a wildfly-arquillian-container-xyz is available on the repository). We could have custom server profile files (standalone-abc.xml) in the jbossws sources for the supported containers to be used for starting specific versions of the containers through Arquillian. The integration tests could be split into groups meant to run against the various different profiles of the selected target container. Pretty much anything we need to customize on the server that can't be done using the Deployer approach can actually be done through a proper profile file (I would still keep the Deployer approach when working to limit the number of profile files to maintain, as there's a lot of stuff in there). Actually, we could even think about having a prepare phase before the actual testsuite that runs given CLI scripts to obtain the desired profile configuration. Next steps --------------- Please provide feedback / ideas. Then we need a plan and eventually jiras created. Cheers Alessio -- Alessio Soldano Web Service Lead, JBoss From rsearls at redhat.com Fri Jun 13 14:44:36 2014 From: rsearls at redhat.com (Rebecca Searls) Date: Fri, 13 Jun 2014 14:44:36 -0400 (EDT) Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539B1B2C.4020105@redhat.com> References: <539B1B2C.4020105@redhat.com> Message-ID: <1758035775.32112284.1402685076494.JavaMail.zimbra@redhat.com> Apache'a Cargo api is worth investigating. In a past life I evaluate it for use in a different env. I found it easy to use and flexible but not quite what that other env needed. I think it would be worth evaluating for our env. http://cargo.codehaus.org/ Cargo is a thin wrapper that allows you to manipulate various type of application containers (Java EE and others) in a standard way. It has a Java API to start/stop/configure supported containers and ant tasks, Maven2/Maven3 plugins for configuring, starting, stopping and deploying applications to supported containers. ----- Original Message ----- > From: "Alessio Soldano" > To: jbossws-dev at lists.jboss.org > Sent: Friday, June 13, 2014 11:39:24 AM > Subject: [jbossws-dev] JBossWS testsuite reorganization > > Folks, > with current development on jbossws trunk aimed at version 5, I believe > it's time consider rethinking the testsuite a bit. > > Current status > -------------------- > Historically, we've been running the integration tests against a JBoss > AS container that has to be started before the testsuite. The target > container is a vanilla distribution that suits with most of the tests, > except for few ones requiring a https connector or a specific security > domain to be available. Implementations of the > org.jboss.wsf.spi.deployer.Deployer interface are available for each > target container and allow deploying test archives as well as setting > security domains / https connector by internally relying on the > container management API. > Different maven profiles trigger different implementations of Deployer > (due to different artifact dependencies / classpath), allowing running > the testsuite the same way against different target containers. When it > comes to continuous integration, the idea is that a QA job first updates > a target container with the current jbossws libraries, then it starts > the container and runs the jbossws testsuite against it. > > Issues / reasons for changing > --------------------------------------- > The current approach is working pretty much fine, but suffers from some > limitations: > * it needs a container to be already running when the integration test > is triggered; it would be great if we could have the container started / > stopped by the test itself > * we're limited to running the testsuite against a predefined container > profile (standalone.xml, standalone-full.xml ): while most of the tests > makes sense when run against standalone.xml profile (because the > webservices subsystem is on in it), we have some tests requiring > subsystems available on different profiles (for instance, the messaging > subsystem which is in standalone-full.xml or the xts subsystem which is > in standalone-xts.xml). Currently we run the testsuite in Spring mode > against standalone-full.xml, basically because till CXF 2.7.x the > SOAP-over-JMS implementation has been available with Spring only. That's > not the case anymore, on CXF 3.0.0, so we'd likely have to test it in > both spring and non-spring mode, but that needs the messaging subsystem > which is not in standalone.xml > * we have few tests (but might have more in future) that actually > require further additional changes to the server; that includes adding > users, messaging queues, setting server side system properties at boot, > ... The only way to achieve that atm is to have the QA job run a CLI > script to modify the server or set a system property on command line > when starting the server jvm > * we have tests affecting the server management model, for instance > changing the way the webservices subsystem model rewrites wsdl > soap:address attributes; that makes it impossible to run those tests at > the same time of any other tests > > Proposals / idea > --------------------- > I'm listening to proposals to revisit the way the testsuite is run and > better address the point above. > Something I've been thinking about is relying on Arquillian, similarly > to what the WFLY testsuite does. What's not trivial is the fact that we > still need to have different version of the Arquillian SPI > implementation (that is, different WFLY / EAP versions) used depending > on a maven profile specified when running the testsuite; moreover we > also need to be able to run against the result of building the latest > WFLY master (iow, not a released target for which a > wildfly-arquillian-container-xyz is available on the repository). We > could have custom server profile files (standalone-abc.xml) in the > jbossws sources for the supported containers to be used for starting > specific versions of the containers through Arquillian. The integration > tests could be split into groups meant to run against the various > different profiles of the selected target container. Pretty much > anything we need to customize on the server that can't be done using the > Deployer approach can actually be done through a proper profile file (I > would still keep the Deployer approach when working to limit the number > of profile files to maintain, as there's a lot of stuff in there). > Actually, we could even think about having a prepare phase before the > actual testsuite that runs given CLI scripts to obtain the desired > profile configuration. > > Next steps > --------------- > Please provide feedback / ideas. Then we need a plan and eventually > jiras created. > > Cheers > Alessio > > -- > Alessio Soldano > Web Service Lead, JBoss > > _______________________________________________ > jbossws-dev mailing list > jbossws-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jbossws-dev > From rsvoboda at redhat.com Fri Jun 13 15:21:20 2014 From: rsvoboda at redhat.com (Rostislav Svoboda) Date: Fri, 13 Jun 2014 15:21:20 -0400 (EDT) Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539B1C1B.9080002@redhat.com> References: <539B1B2C.4020105@redhat.com> <539B1C1B.9080002@redhat.com> Message-ID: <1476228652.24357977.1402687280566.JavaMail.zimbra@redhat.com> > Folks, > with current development on jbossws trunk aimed at version 5, I believe > it's time consider rethinking the testsuite a bit. > > Current status > -------------------- > Historically, we've been running the integration tests against a JBoss > AS container that has to be started before the testsuite. The target > container is a vanilla distribution that suits with most of the tests, > except for few ones requiring a https connector or a specific security > domain to be available. Implementations of the > org.jboss.wsf.spi.deployer.Deployer interface are available for each > target container and allow deploying test archives as well as setting > security domains / https connector by internally relying on the > container management API. > Different maven profiles trigger different implementations of Deployer > (due to different artifact dependencies / classpath), allowing running > the testsuite the same way against different target containers. When it > comes to continuous integration, the idea is that a QA job first updates > a target container with the current jbossws libraries, then it starts > the container and runs the jbossws testsuite against it. > > Issues / reasons for changing > --------------------------------------- > The current approach is working pretty much fine, but suffers from some > limitations: > * it needs a container to be already running when the integration test > is triggered; it would be great if we could have the container started / > stopped by the test itself ARQ solves that. I think ARQ starts application server per module for multi-module maven based TS and per surefire execution. > * we're limited to running the testsuite against a predefined container > profile (standalone.xml, standalone-full.xml ): while most of the tests > makes sense when run against standalone.xml profile (because the > webservices subsystem is on in it), we have some tests requiring > subsystems available on different profiles (for instance, the messaging > subsystem which is in standalone-full.xml or the xts subsystem which is > in standalone-xts.xml). You would still need modules or executions with different arquillian.xml > Currently we run the testsuite in Spring mode > against standalone-full.xml, basically because till CXF 2.7.x the > SOAP-over-JMS implementation has been available with Spring only. That's > not the case anymore, on CXF 3.0.0, so we'd likely have to test it in > both spring and non-spring mode, but that needs the messaging subsystem > which is not in standalone.xml This case is great for 2 executions, but you will need to differentiate xml result file names -- but it's already solved for surefire -- http://jira.codehaus.org/browse/SUREFIRE-750 > * we have few tests (but might have more in future) that actually > require further additional changes to the server; that includes adding > users, messaging queues, setting server side system properties at boot, > ... The only way to achieve that atm is to have the QA job run a CLI > script to modify the server or set a system property on command line > when starting the server jvm ARQ helps here, you will have to switch default start/stop behavior to apply on every TestCase > * we have tests affecting the server management model, for instance > changing the way the webservices subsystem model rewrites wsdl > soap:address attributes; that makes it impossible to run those tests at > the same time of any other tests I thing this still applies even with ARQ usage - server is running all the time, it's not started for every test. So such tests should be isolated in separate execution or submodule and parallel test execution disabled. * BOM file I would like to see definition of versions in BOM file and not in parent. This could help me a lot to run JBossWS TS with productized bits. Now we have to prepare patches for JBossWS codebase to have productized bits on client side too. * ARQ direct usage Still confused about dropping/not dropping old way and moving fully to ARQ. Have feeling that this proposal is on the half way to the full usage of ARQ. I think it's quite safe approach, you still have fallback plan. * integration tests using surefire or failsafe Wouldn't be failsafe more appropriate for integration tests ? http://maven.apache.org/surefire/maven-failsafe-plugin/ The Failsafe Plugin is designed to run integration tests while the Surefire Plugins is designed to run unit tests. > Proposals / idea > --------------------- > I'm listening to proposals to revisit the way the testsuite is run and > better address the point above. > Something I've been thinking about is relying on Arquillian, similarly > to what the WFLY testsuite does. What's not trivial is the fact that we > still need to have different version of the Arquillian SPI > implementation (that is, different WFLY / EAP versions) used depending +1 > on a maven profile specified when running the testsuite; moreover we > also need to be able to run against the result of building the latest > WFLY master (iow, not a released target for which a > wildfly-arquillian-container-xyz is available on the repository). So first build latest WF to have latest wildfly-arquillian-container-xyz ? > We could have custom server profile files (standalone-abc.xml) in the > jbossws sources for the supported containers to be used for starting > specific versions of the containers through Arquillian. The integration Different "configurations" will be definitely necessary (meaning arquillian.xml files), some things can be provided by parameters, but not sure if everything would be configurable this way. About standalone-abc.xml files - why would you have them ? Maintaining them isn't fun. I hope everything could be configurable using ARQ from default profiles/configurations, no custom confing files. > tests could be split into groups meant to run against the various > different profiles of the selected target container. Pretty much > anything we need to customize on the server that can't be done using the > Deployer approach can actually be done through a proper profile file (I > would still keep the Deployer approach when working to limit the number > of profile files to maintain, as there's a lot of stuff in there). Meaning to let ARQ to start WF/EAP with standalone.xml and execute current tests with prepared test libs? ==> in the beginning you won't need to rewrite test archives using ShrinkWrap ? > Actually, we could even think about having a prepare phase before the > actual testsuite that runs given CLI scripts to obtain the desired > profile configuration. I would rather avoid custom standalone-abc.xml files. Cheers. Rostislav > Next steps > --------------- > Please provide feedback / ideas. Then we need a plan and eventually > jiras created. > > Cheers > Alessio > > -- > Alessio Soldano > Web Service Lead, JBoss > > _______________________________________________ > jbossws-dev mailing list > jbossws-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jbossws-dev > > > > From ema at redhat.com Mon Jun 16 06:19:10 2014 From: ema at redhat.com (Jim Ma) Date: Mon, 16 Jun 2014 18:19:10 +0800 Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539B1B2C.4020105@redhat.com> References: <539B1B2C.4020105@redhat.com> Message-ID: <539EC49E.40905@redhat.com> On 06/13/2014 11:39 PM, Alessio Soldano wrote: > Folks, > with current development on jbossws trunk aimed at version 5, I believe > it's time consider rethinking the testsuite a bit. > > Current status > -------------------- > Historically, we've been running the integration tests against a JBoss > AS container that has to be started before the testsuite. The target > container is a vanilla distribution that suits with most of the tests, > except for few ones requiring a https connector or a specific security > domain to be available. Implementations of the > org.jboss.wsf.spi.deployer.Deployer interface are available for each > target container and allow deploying test archives as well as setting > security domains / https connector by internally relying on the > container management API. > Different maven profiles trigger different implementations of Deployer > (due to different artifact dependencies / classpath), allowing running > the testsuite the same way against different target containers. When it > comes to continuous integration, the idea is that a QA job first updates > a target container with the current jbossws libraries, then it starts > the container and runs the jbossws testsuite against it. > > Issues / reasons for changing > --------------------------------------- > The current approach is working pretty much fine, but suffers from some > limitations: > * it needs a container to be already running when the integration test > is triggered; it would be great if we could have the container started / > stopped by the test itself > * we're limited to running the testsuite against a predefined container > profile (standalone.xml, standalone-full.xml ): while most of the tests > makes sense when run against standalone.xml profile (because the > webservices subsystem is on in it), we have some tests requiring > subsystems available on different profiles (for instance, the messaging > subsystem which is in standalone-full.xml or the xts subsystem which is > in standalone-xts.xml). Currently we run the testsuite in Spring mode > against standalone-full.xml, basically because till CXF 2.7.x the > SOAP-over-JMS implementation has been available with Spring only. That's > not the case anymore, on CXF 3.0.0, so we'd likely have to test it in > both spring and non-spring mode, but that needs the messaging subsystem > which is not in standalone.xml > * we have few tests (but might have more in future) that actually > require further additional changes to the server; that includes adding > users, messaging queues, setting server side system properties at boot, > ... The only way to achieve that atm is to have the QA job run a CLI > script to modify the server or set a system property on command line > when starting the server jvm > * we have tests affecting the server management model, for instance > changing the way the webservices subsystem model rewrites wsdl > soap:address attributes; that makes it impossible to run those tests at > the same time of any other tests As Rosta suggested, the ARQ can help resolve some test issues. We also need to write some base or helper class to work better with ARQ to make all these things (configure the standalone.xml and change management model in runtime) much easier. > > Proposals / idea > --------------------- > I'm listening to proposals to revisit the way the testsuite is run and > better address the point above. > Something I've been thinking about is relying on Arquillian, similarly > to what the WFLY testsuite does. What's not trivial is the fact that we > still need to have different version of the Arquillian SPI > implementation (that is, different WFLY / EAP versions) used depending > on a maven profile specified when running the testsuite; moreover we > also need to be able to run against the result of building the latest > WFLY master (iow, not a released target for which a > wildfly-arquillian-container-xyz is available on the repository). Adding profiles for different AS/WF versions looks the thing what we have to do. If WFLY master can publish all these snapshot artifacts during the teamcity build, we can consume it from the snapshot repo. > We > could have custom server profile files (standalone-abc.xml) in the > jbossws sources for the supported containers to be used for starting > specific versions of the containers through Arquillian. The integration > tests could be split into groups meant to run against the various > different profiles of the selected target container. Pretty much > anything we need to customize on the server that can't be done using the > Deployer approach can actually be done through a proper profile file (I > would still keep the Deployer approach when working to limit the number > of profile files to maintain, as there's a lot of stuff in there). > Actually, we could even think about having a prepare phase before the > actual testsuite that runs given CLI scripts to obtain the desired > profile configuration. What about we creating more maven modules/directory for different server profile like WFLY's basic/domain/xts test maven module ? > > Next steps > --------------- > Please provide feedback / ideas. Then we need a plan and eventually > jiras created. There could be many work to convert the test archive built with old ant script to new shrinkwrap. Is there any tool exists help us with this ? We can consider to create some tool to allow us migrate to ARQ much easier and faster ? Cheers, Jim > > Cheers > Alessio > From asoldano at redhat.com Mon Jun 16 07:01:31 2014 From: asoldano at redhat.com (Alessio Soldano) Date: Mon, 16 Jun 2014 13:01:31 +0200 Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <1758035775.32112284.1402685076494.JavaMail.zimbra@redhat.com> References: <539B1B2C.4020105@redhat.com> <1758035775.32112284.1402685076494.JavaMail.zimbra@redhat.com> Message-ID: <539ECE8B.3060905@redhat.com> Thanks Rebecca. I've given a look at Cargo website; it's definitely interesting, but I was not able to figure out the advantage it would offer us when compare to Arquillian. In other words, afaics Arquillian offers anything Cargo does, plus a better testsuite integration. Alessio On 13/06/14 20:44, Rebecca Searls wrote: > Apache'a Cargo api is worth investigating. In a past life I evaluate > it for use in a different env. I found it easy to use and flexible but not > quite what that other env needed. I think it would be worth evaluating for > our env. > > http://cargo.codehaus.org/ > Cargo is a thin wrapper that allows you to manipulate various type of > application containers (Java EE and others) in a standard way. It has > a Java API to start/stop/configure supported containers and ant tasks, > Maven2/Maven3 plugins for configuring, starting, stopping and deploying > applications to supported containers. > > > > ----- Original Message ----- >> From: "Alessio Soldano" >> To: jbossws-dev at lists.jboss.org >> Sent: Friday, June 13, 2014 11:39:24 AM >> Subject: [jbossws-dev] JBossWS testsuite reorganization >> >> Folks, >> with current development on jbossws trunk aimed at version 5, I believe >> it's time consider rethinking the testsuite a bit. >> >> Current status >> -------------------- >> Historically, we've been running the integration tests against a JBoss >> AS container that has to be started before the testsuite. The target >> container is a vanilla distribution that suits with most of the tests, >> except for few ones requiring a https connector or a specific security >> domain to be available. Implementations of the >> org.jboss.wsf.spi.deployer.Deployer interface are available for each >> target container and allow deploying test archives as well as setting >> security domains / https connector by internally relying on the >> container management API. >> Different maven profiles trigger different implementations of Deployer >> (due to different artifact dependencies / classpath), allowing running >> the testsuite the same way against different target containers. When it >> comes to continuous integration, the idea is that a QA job first updates >> a target container with the current jbossws libraries, then it starts >> the container and runs the jbossws testsuite against it. >> >> Issues / reasons for changing >> --------------------------------------- >> The current approach is working pretty much fine, but suffers from some >> limitations: >> * it needs a container to be already running when the integration test >> is triggered; it would be great if we could have the container started / >> stopped by the test itself >> * we're limited to running the testsuite against a predefined container >> profile (standalone.xml, standalone-full.xml ): while most of the tests >> makes sense when run against standalone.xml profile (because the >> webservices subsystem is on in it), we have some tests requiring >> subsystems available on different profiles (for instance, the messaging >> subsystem which is in standalone-full.xml or the xts subsystem which is >> in standalone-xts.xml). Currently we run the testsuite in Spring mode >> against standalone-full.xml, basically because till CXF 2.7.x the >> SOAP-over-JMS implementation has been available with Spring only. That's >> not the case anymore, on CXF 3.0.0, so we'd likely have to test it in >> both spring and non-spring mode, but that needs the messaging subsystem >> which is not in standalone.xml >> * we have few tests (but might have more in future) that actually >> require further additional changes to the server; that includes adding >> users, messaging queues, setting server side system properties at boot, >> ... The only way to achieve that atm is to have the QA job run a CLI >> script to modify the server or set a system property on command line >> when starting the server jvm >> * we have tests affecting the server management model, for instance >> changing the way the webservices subsystem model rewrites wsdl >> soap:address attributes; that makes it impossible to run those tests at >> the same time of any other tests >> >> Proposals / idea >> --------------------- >> I'm listening to proposals to revisit the way the testsuite is run and >> better address the point above. >> Something I've been thinking about is relying on Arquillian, similarly >> to what the WFLY testsuite does. What's not trivial is the fact that we >> still need to have different version of the Arquillian SPI >> implementation (that is, different WFLY / EAP versions) used depending >> on a maven profile specified when running the testsuite; moreover we >> also need to be able to run against the result of building the latest >> WFLY master (iow, not a released target for which a >> wildfly-arquillian-container-xyz is available on the repository). We >> could have custom server profile files (standalone-abc.xml) in the >> jbossws sources for the supported containers to be used for starting >> specific versions of the containers through Arquillian. The integration >> tests could be split into groups meant to run against the various >> different profiles of the selected target container. Pretty much >> anything we need to customize on the server that can't be done using the >> Deployer approach can actually be done through a proper profile file (I >> would still keep the Deployer approach when working to limit the number >> of profile files to maintain, as there's a lot of stuff in there). >> Actually, we could even think about having a prepare phase before the >> actual testsuite that runs given CLI scripts to obtain the desired >> profile configuration. >> >> Next steps >> --------------- >> Please provide feedback / ideas. Then we need a plan and eventually >> jiras created. >> >> Cheers >> Alessio >> >> -- >> Alessio Soldano >> Web Service Lead, JBoss >> >> _______________________________________________ >> jbossws-dev mailing list >> jbossws-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jbossws-dev >> -- Alessio Soldano Web Service Lead, JBoss From asoldano at redhat.com Mon Jun 16 07:28:16 2014 From: asoldano at redhat.com (Alessio Soldano) Date: Mon, 16 Jun 2014 13:28:16 +0200 Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <1476228652.24357977.1402687280566.JavaMail.zimbra@redhat.com> References: <539B1B2C.4020105@redhat.com> <539B1C1B.9080002@redhat.com> <1476228652.24357977.1402687280566.JavaMail.zimbra@redhat.com> Message-ID: <539ED4D0.5000908@redhat.com> Hi Rostislav, On 13/06/14 21:21, Rostislav Svoboda wrote: >> * we have few tests (but might have more in future) that actually >> require further additional changes to the server; that includes adding >> users, messaging queues, setting server side system properties at boot, >> ... The only way to achieve that atm is to have the QA job run a CLI >> script to modify the server or set a system property on command line >> when starting the server jvm > ARQ helps here, you will have to switch default start/stop behavior to apply on every TestCase > >> * we have tests affecting the server management model, for instance >> changing the way the webservices subsystem model rewrites wsdl >> soap:address attributes; that makes it impossible to run those tests at >> the same time of any other tests > I thing this still applies even with ARQ usage - server is running all the time, it's not started for every test. > So such tests should be isolated in separate execution or submodule and parallel test execution disabled. Basically, I'm thinking about solving this the same as for the previous item (start a container for each of these tests and stop it when the test is over). > * BOM file > > I would like to see definition of versions in BOM file and not in parent. > This could help me a lot to run JBossWS TS with productized bits. > Now we have to prepare patches for JBossWS codebase to have productized bits on client side too. OK, this is however an independent task (a general project build/structure change), isn't it? > * ARQ direct usage > > Still confused about dropping/not dropping old way and moving fully to ARQ. > Have feeling that this proposal is on the half way to the full usage of ARQ. > I think it's quite safe approach, you still have fallback plan. Not sure what you mean here above; in any case, the move to use ARQ is still something being evaluated, not set in stone (yet). > * integration tests using surefire or failsafe > > Wouldn't be failsafe more appropriate for integration tests ? > http://maven.apache.org/surefire/maven-failsafe-plugin/ > The Failsafe Plugin is designed to run integration tests while the Surefire Plugins is designed to run unit tests. Possibly yes; to be honest, though, given we don't run anything after the integration-test phase atm, there would be no change. But you're right, failsafe is probably more appropriate. > >> on a maven profile specified when running the testsuite; moreover we >> also need to be able to run against the result of building the latest >> WFLY master (iow, not a released target for which a >> wildfly-arquillian-container-xyz is available on the repository). > So first build latest WF to have latest wildfly-arquillian-container-xyz ? Right, I didn't notice those artifacts actually come the WFLY build. So no problem here. >> We could have custom server profile files (standalone-abc.xml) in the >> jbossws sources for the supported containers to be used for starting >> specific versions of the containers through Arquillian. The integration > Different "configurations" will be definitely necessary (meaning arquillian.xml files), some things can be provided by parameters, but not sure if everything would be configurable this way. > About standalone-abc.xml files - why would you have them ? Maintaining them isn't fun. I hope everything could be configurable using ARQ from default profiles/configurations, no custom confing files. I share your concern; that's why I said I'd probably try something like a preliminary phase in which an existing server config file is copied, the server started, some cli commands sent to modify the config, the server stopped and eventually the server config referenced and used by arquillian. Maintaining standalone-abc.xml files should be the last option. > >> tests could be split into groups meant to run against the various >> different profiles of the selected target container. Pretty much >> anything we need to customize on the server that can't be done using the >> Deployer approach can actually be done through a proper profile file (I >> would still keep the Deployer approach when working to limit the number >> of profile files to maintain, as there's a lot of stuff in there). > Meaning to let ARQ to start WF/EAP with standalone.xml and execute current tests with prepared test libs? > ==> in the beginning you won't need to rewrite test archives using ShrinkWrap ? I was actually still reasoning on the stanalone-abc.xml thing above. However, the migration to SW is something we wanted to do anyway (there's a jra for it). What you propose could be a interesting idea, the migration of ~600 tests to SW is not going to be a few hours tasks ;-) Thanks Alessio -- Alessio Soldano Web Service Lead, JBoss From asoldano at redhat.com Mon Jun 16 08:19:35 2014 From: asoldano at redhat.com (Alessio Soldano) Date: Mon, 16 Jun 2014 14:19:35 +0200 Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539EC49E.40905@redhat.com> References: <539B1B2C.4020105@redhat.com> <539EC49E.40905@redhat.com> Message-ID: <539EE0D7.2020907@redhat.com> Hi Jim, On 16/06/14 12:19, Jim Ma wrote: > As Rosta suggested, the ARQ can help resolve some test issues. We also > need to write some base or helper class to work better with ARQ to make > all these things (configure the standalone.xml and change management > model > in runtime) much easier. Right... >> >> Proposals / idea >> --------------------- >> I'm listening to proposals to revisit the way the testsuite is run and >> better address the point above. >> Something I've been thinking about is relying on Arquillian, similarly >> to what the WFLY testsuite does. What's not trivial is the fact that we >> still need to have different version of the Arquillian SPI >> implementation (that is, different WFLY / EAP versions) used depending >> on a maven profile specified when running the testsuite; moreover we >> also need to be able to run against the result of building the latest >> WFLY master (iow, not a released target for which a >> wildfly-arquillian-container-xyz is available on the repository). > Adding profiles for different AS/WF versions looks the thing what we > have to do. If WFLY master can publish all these snapshot > artifacts during the teamcity build, we can consume it from the > snapshot repo. We build WFLY master before running the testsuite on hudson, and build wfly locally. That should be enough :-) >> We >> could have custom server profile files (standalone-abc.xml) in the >> jbossws sources for the supported containers to be used for starting >> specific versions of the containers through Arquillian. The integration >> tests could be split into groups meant to run against the various >> different profiles of the selected target container. Pretty much >> anything we need to customize on the server that can't be done using the >> Deployer approach can actually be done through a proper profile file (I >> would still keep the Deployer approach when working to limit the number >> of profile files to maintain, as there's a lot of stuff in there). >> Actually, we could even think about having a prepare phase before the >> actual testsuite that runs given CLI scripts to obtain the desired >> profile configuration. > What about we creating more maven modules/directory for different > server profile like WFLY's basic/domain/xts test maven module ? Yes, that's required too. >> >> Next steps >> --------------- >> Please provide feedback / ideas. Then we need a plan and eventually >> jiras created. > There could be many work to convert the test archive built with old > ant script to new shrinkwrap. Is there any tool exists help us with > this ? Not that I know > We can consider to create some tool to allow us migrate to ARQ much > easier and faster ? Not sure, but something might be doable for the Shrinkwrap conversion; we should basically parse the Ant build files actually containing the libs creation, create a map deployment-archive-name <--> file-contents and then go through all the testcases and paste the new code using SW for the corresponding deployment-archive. Cheers Alessio -- Alessio Soldano Web Service Lead, JBoss From asoldano at redhat.com Mon Jun 16 10:12:00 2014 From: asoldano at redhat.com (Alessio Soldano) Date: Mon, 16 Jun 2014 16:12:00 +0200 Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539EE0D7.2020907@redhat.com> References: <539B1B2C.4020105@redhat.com> <539EC49E.40905@redhat.com> <539EE0D7.2020907@redhat.com> Message-ID: <539EFB30.3000209@redhat.com> On 16/06/14 14:19, Alessio Soldano wrote: >> There could be many work to convert the test archive built with old >> ant script to new shrinkwrap. Is there any tool exists help us with >> this ? > Not that I know > >> We can consider to create some tool to allow us migrate to ARQ much >> easier and faster ? > Not sure, but something might be doable for the Shrinkwrap conversion; > we should basically parse the Ant build files actually containing the > libs creation, create a map deployment-archive-name <--> file-contents > and then go through all the testcases and paste the new code using SW > for the corresponding deployment-archive. I'll do some experiment on this as part of https://issues.jboss.org/browse/JBWS-3711 Alessio -- Alessio Soldano Web Service Lead, JBoss From rsvoboda at redhat.com Thu Jun 26 03:28:50 2014 From: rsvoboda at redhat.com (Rostislav Svoboda) Date: Thu, 26 Jun 2014 03:28:50 -0400 (EDT) Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539ED4D0.5000908@redhat.com> References: <539B1B2C.4020105@redhat.com> <539B1C1B.9080002@redhat.com> <1476228652.24357977.1402687280566.JavaMail.zimbra@redhat.com> <539ED4D0.5000908@redhat.com> Message-ID: <351036786.30699354.1403767730366.JavaMail.zimbra@redhat.com> Hi Alessio, > Hi Rostislav, > > On 13/06/14 21:21, Rostislav Svoboda wrote: > >> * we have few tests (but might have more in future) that actually > >> require further additional changes to the server; that includes adding > >> users, messaging queues, setting server side system properties at boot, > >> ... The only way to achieve that atm is to have the QA job run a CLI > >> script to modify the server or set a system property on command line > >> when starting the server jvm > > ARQ helps here, you will have to switch default start/stop behavior to > > apply on every TestCase > > > >> * we have tests affecting the server management model, for instance > >> changing the way the webservices subsystem model rewrites wsdl > >> soap:address attributes; that makes it impossible to run those tests at > >> the same time of any other tests > > I thing this still applies even with ARQ usage - server is running all the > > time, it's not started for every test. > > So such tests should be isolated in separate execution or submodule and > > parallel test execution disabled. > Basically, I'm thinking about solving this the same as for the previous > item (start a container for each of these tests and stop it when the > test is over). Ok > > > * BOM file > > > > I would like to see definition of versions in BOM file and not in parent. > > This could help me a lot to run JBossWS TS with productized bits. > > Now we have to prepare patches for JBossWS codebase to have productized > > bits on client side too. > OK, this is however an independent task (a general project > build/structure change), isn't it? For me it's 50/50 :) I think it's related to testsuite too. It could be possible to quickly change client/testuite configuration to run on upstream or productized bits for example. > > > * ARQ direct usage > > > > Still confused about dropping/not dropping old way and moving fully to ARQ. > > Have feeling that this proposal is on the half way to the full usage of > > ARQ. > > I think it's quite safe approach, you still have fallback plan. > Not sure what you mean here above; in any case, the move to use ARQ is > still something being evaluated, not set in stone (yet). Ok > > > * integration tests using surefire or failsafe > > > > Wouldn't be failsafe more appropriate for integration tests ? > > http://maven.apache.org/surefire/maven-failsafe-plugin/ > > The Failsafe Plugin is designed to run integration tests while the > > Surefire Plugins is designed to run unit tests. > Possibly yes; to be honest, though, given we don't run anything after > the integration-test phase atm, there would be no change. But you're > right, failsafe is probably more appropriate. +1 > > > > >> on a maven profile specified when running the testsuite; moreover we > >> also need to be able to run against the result of building the latest > >> WFLY master (iow, not a released target for which a > >> wildfly-arquillian-container-xyz is available on the repository). > > So first build latest WF to have latest wildfly-arquillian-container-xyz ? > Right, I didn't notice those artifacts actually come the WFLY build. So > no problem here. +1 > > >> We could have custom server profile files (standalone-abc.xml) in the > >> jbossws sources for the supported containers to be used for starting > >> specific versions of the containers through Arquillian. The integration > > Different "configurations" will be definitely necessary (meaning > > arquillian.xml files), some things can be provided by parameters, but not > > sure if everything would be configurable this way. > > About standalone-abc.xml files - why would you have them ? Maintaining them > > isn't fun. I hope everything could be configurable using ARQ from default > > profiles/configurations, no custom confing files. > I share your concern; that's why I said I'd probably try something like > a preliminary phase in which an existing server config file is copied, > the server started, some cli commands sent to modify the config, the > server stopped and eventually the server config referenced and used by > arquillian. Maintaining standalone-abc.xml files should be the last option. again +1 :) > > > >> tests could be split into groups meant to run against the various > >> different profiles of the selected target container. Pretty much > >> anything we need to customize on the server that can't be done using the > >> Deployer approach can actually be done through a proper profile file (I > >> would still keep the Deployer approach when working to limit the number > >> of profile files to maintain, as there's a lot of stuff in there). > > Meaning to let ARQ to start WF/EAP with standalone.xml and execute current > > tests with prepared test libs? > > ==> in the beginning you won't need to rewrite test archives using > > ShrinkWrap ? > I was actually still reasoning on the stanalone-abc.xml thing above. > However, the migration to SW is something we wanted to do anyway > (there's a jra for it). What you propose could be a interesting idea, > the migration of ~600 tests to SW is not going to be a few hours tasks ;-) > > Thanks > Alessio Reorganization plan is more clear to now. Thanks for sharing details. Cheers. Rostislav > -- > Alessio Soldano > Web Service Lead, JBoss > > From rsvoboda at redhat.com Thu Jun 26 03:30:18 2014 From: rsvoboda at redhat.com (Rostislav Svoboda) Date: Thu, 26 Jun 2014 03:30:18 -0400 (EDT) Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <539EE0D7.2020907@redhat.com> References: <539B1B2C.4020105@redhat.com> <539EC49E.40905@redhat.com> <539EE0D7.2020907@redhat.com> Message-ID: <177181653.30699830.1403767818007.JavaMail.zimbra@redhat.com> > >> > >> Next steps > >> --------------- > >> Please provide feedback / ideas. Then we need a plan and eventually > >> jiras created. > > There could be many work to convert the test archive built with old > > ant script to new shrinkwrap. Is there any tool exists help us with > > this ? > Not that I know I did some searching and found nothing for this use-case ... > > > We can consider to create some tool to allow us migrate to ARQ much > > easier and faster ? > Not sure, but something might be doable for the Shrinkwrap conversion; > we should basically parse the Ant build files actually containing the > libs creation, create a map deployment-archive-name <--> file-contents > and then go through all the testcases and paste the new code using SW > for the corresponding deployment-archive. +1 Cheers. Rostislav