From: "Alessio Soldano" <asoldano(a)redhat.com&gt; To: jbossws-dev(a)lists.jboss.org Sent: Friday, June 13, 2014 11:39:24 AM Subject: [jbossws-dev] JBossWS testsuite reorganization Folks, with current development on jbossws trunk aimed at version 5, I believe it's time consider rethinking the testsuite a bit. Current status -------------------- Historically, we've been running the integration tests against a JBoss AS container that has to be started before the testsuite. The target container is a vanilla distribution that suits with most of the tests, except for few ones requiring a https connector or a specific security domain to be available. Implementations of the org.jboss.wsf.spi.deployer.Deployer interface are available for each target container and allow deploying test archives as well as setting security domains / https connector by internally relying on the container management API. Different maven profiles trigger different implementations of Deployer (due to different artifact dependencies / classpath), allowing running the testsuite the same way against different target containers. When it comes to continuous integration, the idea is that a QA job first updates a target container with the current jbossws libraries, then it starts the container and runs the jbossws testsuite against it. Issues / reasons for changing --------------------------------------- The current approach is working pretty much fine, but suffers from some limitations: * it needs a container to be already running when the integration test is triggered; it would be great if we could have the container started / stopped by the test itself * we're limited to running the testsuite against a predefined container profile (standalone.xml, standalone-full.xml ): while most of the tests makes sense when run against standalone.xml profile (because the webservices subsystem is on in it), we have some tests requiring subsystems available on different profiles (for instance, the messaging subsystem which is in standalone-full.xml or the xts subsystem which is in standalone-xts.xml). Currently we run the testsuite in Spring mode against standalone-full.xml, basically because till CXF 2.7.x the SOAP-over-JMS implementation has been available with Spring only. That's not the case anymore, on CXF 3.0.0, so we'd likely have to test it in both spring and non-spring mode, but that needs the messaging subsystem which is not in standalone.xml * we have few tests (but might have more in future) that actually require further additional changes to the server; that includes adding users, messaging queues, setting server side system properties at boot, ... The only way to achieve that atm is to have the QA job run a CLI script to modify the server or set a system property on command line when starting the server jvm * we have tests affecting the server management model, for instance changing the way the webservices subsystem model rewrites wsdl soap:address attributes; that makes it impossible to run those tests at the same time of any other tests Proposals / idea --------------------- I'm listening to proposals to revisit the way the testsuite is run and better address the point above. Something I've been thinking about is relying on Arquillian, similarly to what the WFLY testsuite does. What's not trivial is the fact that we still need to have different version of the Arquillian SPI implementation (that is, different WFLY / EAP versions) used depending on a maven profile specified when running the testsuite; moreover we also need to be able to run against the result of building the latest WFLY master (iow, not a released target for which a wildfly-arquillian-container-xyz is available on the repository). We could have custom server profile files (standalone-abc.xml) in the jbossws sources for the supported containers to be used for starting specific versions of the containers through Arquillian. The integration tests could be split into groups meant to run against the various different profiles of the selected target container. Pretty much anything we need to customize on the server that can't be done using the Deployer approach can actually be done through a proper profile file (I would still keep the Deployer approach when working to limit the number of profile files to maintain, as there's a lot of stuff in there). Actually, we could even think about having a prepare phase before the actual testsuite that runs given CLI scripts to obtain the desired profile configuration. Next steps --------------- Please provide feedback / ideas. Then we need a plan and eventually jiras created. Cheers Alessio -- Alessio Soldano Web Service Lead, JBoss _______________________________________________ jbossws-dev mailing list jbossws-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbossws-dev

Folks, with current development on jbossws trunk aimed at version 5, I believe it's time consider rethinking the testsuite a bit. Current status -------------------- Historically, we've been running the integration tests against a JBoss AS container that has to be started before the testsuite. The target container is a vanilla distribution that suits with most of the tests, except for few ones requiring a https connector or a specific security domain to be available. Implementations of the org.jboss.wsf.spi.deployer.Deployer interface are available for each target container and allow deploying test archives as well as setting security domains / https connector by internally relying on the container management API. Different maven profiles trigger different implementations of Deployer (due to different artifact dependencies / classpath), allowing running the testsuite the same way against different target containers. When it comes to continuous integration, the idea is that a QA job first updates a target container with the current jbossws libraries, then it starts the container and runs the jbossws testsuite against it. Issues / reasons for changing --------------------------------------- The current approach is working pretty much fine, but suffers from some limitations: * it needs a container to be already running when the integration test is triggered; it would be great if we could have the container started / stopped by the test itself * we're limited to running the testsuite against a predefined container profile (standalone.xml, standalone-full.xml ): while most of the tests makes sense when run against standalone.xml profile (because the webservices subsystem is on in it), we have some tests requiring subsystems available on different profiles (for instance, the messaging subsystem which is in standalone-full.xml or the xts subsystem which is in standalone-xts.xml). Currently we run the testsuite in Spring mode against standalone-full.xml, basically because till CXF 2.7.x the SOAP-over-JMS implementation has been available with Spring only. That's not the case anymore, on CXF 3.0.0, so we'd likely have to test it in both spring and non-spring mode, but that needs the messaging subsystem which is not in standalone.xml * we have few tests (but might have more in future) that actually require further additional changes to the server; that includes adding users, messaging queues, setting server side system properties at boot, ... The only way to achieve that atm is to have the QA job run a CLI script to modify the server or set a system property on command line when starting the server jvm * we have tests affecting the server management model, for instance changing the way the webservices subsystem model rewrites wsdl soap:address attributes; that makes it impossible to run those tests at the same time of any other tests

Proposals / idea --------------------- I'm listening to proposals to revisit the way the testsuite is run and better address the point above. Something I've been thinking about is relying on Arquillian, similarly to what the WFLY testsuite does. What's not trivial is the fact that we still need to have different version of the Arquillian SPI implementation (that is, different WFLY / EAP versions) used depending on a maven profile specified when running the testsuite; moreover we also need to be able to run against the result of building the latest WFLY master (iow, not a released target for which a wildfly-arquillian-container-xyz is available on the repository).

We could have custom server profile files (standalone-abc.xml) in the jbossws sources for the supported containers to be used for starting specific versions of the containers through Arquillian. The integration tests could be split into groups meant to run against the various different profiles of the selected target container. Pretty much anything we need to customize on the server that can't be done using the Deployer approach can actually be done through a proper profile file (I would still keep the Deployer approach when working to limit the number of profile files to maintain, as there's a lot of stuff in there). Actually, we could even think about having a prepare phase before the actual testsuite that runs given CLI scripts to obtain the desired profile configuration.

Next steps --------------- Please provide feedback / ideas. Then we need a plan and eventually jiras created.

Cheers Alessio

> There could be many work to convert the test archive built with old > ant script to new shrinkwrap. Is there any tool exists help us with > this ? Not that I know > We can consider to create some tool to allow us migrate to ARQ much > easier and faster ? Not sure, but something might be doable for the Shrinkwrap conversion; we should basically parse the Ant build files actually containing the libs creation, create a map deployment-archive-name <--> file-contents and then go through all the testcases and paste the new code using SW for the corresponding deployment-archive.

----- Original Message ----- > From: "Alessio Soldano" <asoldano(a)redhat.com&gt; > To: jbossws-dev(a)lists.jboss.org > Sent: Monday, November 17, 2014 6:04:41 AM > Subject: Re: [jbossws-dev] JBossWS testsuite reorganization > > ... > - we'll have a convention requiring deployment names to be unique (IOW, > no more deployments used by multiple tests); there're are currently few > deployments used in more then a test, but that causes more problems then > benefits (also consider we could now avoid writing the actually jar/war > to the filesystem by default) I find the filesystem archives helpful at times. Particularly when created new tests. Hopefully there is a simple switch that will allow us to turn this facility on and off. If not ,I would prefer this be left on.

Five months after I started this thread, I finally come back to it with the result of some additional Arquillian investigation I did at the end of last week and a plan for dealing with the reorganization: - we'll be using the wildfly-arquillian-container-managed integration - we'll have multiple standalone.xml server configurations for testing; I plan to provide a solution for avoiding maintaining many confs, basically deriving them from the current container one (either by running CLI commands or by applying XSL transformations). The multiple

configurations are required for any test that needs e.g. a custom https connector - we'll be using the 'class' Arquillian container mode for any test requiring a custom server configuration (each test is going to have its own server configuration); all remaining tests will use the default (suite) container mode - all @Deployment will have 'managed=true', except for very few and special cases - all @Deployment will initially have 'testable=false' and all @Test methods will also be @RunAsClient annotated; later we will see if we can re-write some of the tests meant to run in-container which are currently using custom mechanisms (ejb3 / servlet client deployments). - we'll have a convention requiring deployment names to be unique (IOW, no more deployments used by multiple tests); there're are currently few deployments used in more then a test, but that causes more problems then benefits (also consider we could now avoid writing the actually jar/war to the filesystem by default) - we'll probably have a mechanism in the build to automatically retrieve the specified version of WildFly, clone it, generate the various standalone.xml, update the ws stack on it and finally have the testsuite run. No more need for 2 builds (ant + maven) for deploying the stack first and testing it afterwards - we won't need the RemoteDeployer stuff anymore ;-)

Cheers Alessio -- Alessio Soldano Web Service Lead, JBoss _______________________________________________ jbossws-dev mailing list jbossws-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbossws-dev

Five months after I started this thread, I finally come back to it with the result of some additional Arquillian investigation I did at the end of last week and a plan for dealing with the reorganization: - we'll be using the wildfly-arquillian-container-managed integration - we'll have multiple standalone.xml server configurations for testing; I plan to provide a solution for avoiding maintaining many confs, basically deriving them from the current container one (either by running CLI commands or by applying XSL transformations). The multiple configurations are required for any test that needs e.g. a custom https connector

- we'll be using the 'class' Arquillian container mode for any test requiring a custom server configuration (each test is going to have its own server configuration); all remaining tests will use the default (suite) container mode - all @Deployment will have 'managed=true', except for very few and special cases - all @Deployment will initially have 'testable=false' and all @Test methods will also be @RunAsClient annotated; later we will see if we can re-write some of the tests meant to run in-container which are currently using custom mechanisms (ejb3 / servlet client deployments). - we'll have a convention requiring deployment names to be unique (IOW, no more deployments used by multiple tests); there're are currently few deployments used in more then a test, but that causes more problems then benefits (also consider we could now avoid writing the actually jar/war to the filesystem by default) - we'll probably have a mechanism in the build to automatically retrieve the specified version of WildFly, clone it, generate the various standalone.xml, update the ws stack on it and finally have the testsuite run. No more need for 2 builds (ant + maven) for deploying the stack first and testing it afterwards

- we won't need the RemoteDeployer stuff anymore ;-)

Cheers Alessio

mvn -PwildflyXYZ integration-test

mvn -PwildflyXYZ -Dserver.home=/foo/bar package

mvn -PwildflyXYZ package

mvn -Pdist,testsuite clean

Can we make wildfly900 profile as the default configuration ? If there is no any profile or system property specified, with "mvn clean install" it downloads wildfly900 dist ,install jbossws-cxf and run all the tests ? On 12/12/2014 12:44 AM, Alessio Soldano wrote: > The first initial prototype for the new testsuite and build of JBossWS 5 > is available at > https://svn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian > I have converted the cxf-specific testuite to use Arquillian and > temporarly disabled the shared-testsuite (it will be re-enabled once it > will be converted too). > A basic README file has been added to the root of the project with build > instructions; here is the content anyway: > > Building and running the testsuite > ------------------------------------ > The build follows the usual Maven flow; a wilflyXYZ profile has to be > specified to tell the project which target container to use for > integration tests; if no wildflyXYZ profile is specified, the > integration tests are skipped. > > > mvn -PwildflyXYZ integration-test > > The -Dserver.home=/foo/bar option can be used to run the testsuite > against a given local server instance; the server does not need to be > already running, as the build will create various standalone server > configurations and start multiple instances. > > The 'fast' profile can also be used to run tests concurrently. > > Updating WS stack > ------------------- > In some cases it might be needed to build the ws stack and install it on > a specified server instance without running the integration testsuite; > this is achieved as follows: > > > mvn -PwildflyXYZ -Dserver.home=/foo/bar package > > If a server.home property is not provided, the build creates a zip > archive with a vanilla WildFly server patched with the current WS stack: > > > mvn -PwildflyXYZ package > > the zip file path is > modules/dist/target/jbossws-cxf-dist-${project.version}-test-server.zip > > > Cleaning up > ------------- > The project is cleaned up as follows: > > > mvn -Pdist,testsuite clean > > > > > As you can see, it is way simpler then what we used to have in previous > JBossWS versions. The idea is to completely discontinue the binary and > source distributions and distribute a simple zip file of the svn/git > project, which the user will simply build like we do as explained above. > > One of the major differences with the past is that having a local and > running WildFly server instance is not strictly required anymore, as the > JBossWS build goes and fetch the WildFly distros, self-installs on them, > start them and run the integration testsuites. Few different server > configurations are actually started (3 at the moment, I've been grouping > together all the server setup requirements that can live together: > security domains, system properties, etc.) > The server configurations are obtained by copying the standalone.xml and > patching it using gmaven plugin (Groovy), which seems quite simple and > easy to maintain. > > Any comment, let me know. There's still quite a lot to do, to cleanup a > bunch of stuff that we don't use anymore in the build, convert the > shared and spring testsuites and nail down some transient test failures > that can be reproduced when using the -Pfast profile (likely Arquillian > related issues). > > Cheers > Alessio >

On 12/16/2014 05:03 PM, Alessio Soldano wrote: > Hi Jim, > actually no, we can't, because when the latest moving target is used > (wfly900 here, which comes from wfly master), the server.home prop is > actually required, because the maven repo does not have the > distribution artifacts. wildfly upstream is our jbossws project consumer, and we need to first make sure everything works, like we already did in past. I know wildfly upstream changes very quick even in each day, deploy the snapshot artifacts will be time consuming work for their build machine. But jbossws requires the latest wildfly dist snapshot and make sure jbossws can work well with wildfly integration layer. To resolve the wildfly dist snapshot is missing, we can look at download it directly from our hudson machine. This would be helpful for our user to try our code, we don't require them to pull whole wildfly upstream and build all snapshot artifacts before running.

> Generally speaking, I'm not sure we should single out a container and > run the integration testsuite against that; if we actually decide to > do this, a released target would need to be used (also for future > build reproducibility, we don't want to default testing against a > snapshot version) IMO, this is not a problem when both jbossws and wildfly are snapshots and in development phrase. When we do release, we only need to release the deploy stuff against wildfly9's future formal release or wildfly810/wildfly800.

What I want to propose is to simplify our build more. wildfly900 or wildfly810 profile is required to run test suite, and we can take one as the default setting. My goal is to use one simple "mvn clean install" to build jbossws-cxf, deploy to wildfly and run the tests. If wildfly900 is not a good option here, I am not against take wildfly810 profile as the default test container. Your thoughts ?

On 17/12/14 11:17, Alessio Soldano wrote: > On 17/12/14 10:54, Jim Ma wrote: >> What I want to propose is to simplify our build more. wildfly900 or >> wildfly810 profile is required to run test suite, and we can take one >> as the default setting. My goal is to use one simple "mvn clean >> install" to build jbossws-cxf, deploy to wildfly and run the tests. If >> wildfly900 is not a good option here, I am not against take wildfly810 >> profile as the default test container. Your thoughts ? > I see your point. I can have a look at enabling by default the most > recent released target container (wfly 810 currently). I've had a look and it's not easily doable, basically because there's no way in Maven (AFAIK) to have a profile enabled unless another one (in a set of 2 or 3) is already enabled. The only solution is to have the wildfly800,wildfly810,wildfly900 profiles be controlled by the value of a command line property (something like -DintegrationTarget=wildfly810). In the main pom.xml, we'd need an additional profile for the default scenario, to be enabled when the property is not provided. We'd have to duplicate the whole wildfly810 profile in modules/testsuite/pom.xml too, which is quite a big one (additional dependencies, additional plugins, etc in it), making the pom even harder to read.

I'm starting thinking this creates more problems then it actually solves. An alternative approach could be to have a profile using a antrun plugin to check if there's one of the wildfly800,wildfly810,wildfly900 profiles enabled (e.g. by checking the jboss.version property that's set by those profiles) and make the build fail with a nice message explaining what to do (either use one of those profiles, or set a property for preventing the check). WDYT?

Cheers Alessio

On 17/12/14 11:17, Alessio Soldano wrote: > On 17/12/14 10:54, Jim Ma wrote: >> What I want to propose is to simplify our build more. wildfly900 or >> wildfly810 profile is required to run test suite, and we can take one >> as the default setting. My goal is to use one simple "mvn clean >> install" to build jbossws-cxf, deploy to wildfly and run the tests. If >> wildfly900 is not a good option here, I am not against take wildfly810 >> profile as the default test container. Your thoughts ? > I see your point. I can have a look at enabling by default the most > recent released target container (wfly 810 currently). I've had a look and it's not easily doable, basically because there's no way in Maven (AFAIK) to have a profile enabled unless another one (in a set of 2 or 3) is already enabled. The only solution is to have the wildfly800,wildfly810,wildfly900 profiles be controlled by the value of a command line property (something like -DintegrationTarget=wildfly810). In the main pom.xml, we'd need an additional profile for the default scenario, to be enabled when the property is not provided. We'd have to duplicate the whole wildfly810 profile in modules/testsuite/pom.xml too, which is quite a big one (additional dependencies, additional plugins, etc in it), making the pom even harder to read. I'm starting thinking this creates more problems then it actually

solves. An alternative approach could be to have a profile using a antrun plugin to check if there's one of the wildfly800,wildfly810,wildfly900 profiles enabled (e.g. by checking the jboss.version property that's set by those profiles) and make the build fail with a nice message explaining what to do (either use one of those profiles, or set a property for preventing the check). WDYT?

Cheers Alessio -- Alessio Soldano Web Service Lead, JBoss _______________________________________________ jbossws-dev mailing list jbossws-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbossws-dev

The first initial prototype for the new testsuite and build of JBossWS 5 is available at https://svn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian I have converted the cxf-specific testuite to use Arquillian and temporarly disabled the shared-testsuite (it will be re-enabled once it will be converted too). A basic README file has been added to the root of the project with build instructions; here is the content anyway: Building and running the testsuite ------------------------------------ The build follows the usual Maven flow; a wilflyXYZ profile has to be specified to tell the project which target container to use for integration tests; if no wildflyXYZ profile is specified, the integration tests are skipped. > mvn -PwildflyXYZ integration-test The -Dserver.home=/foo/bar option can be used to run the testsuite against a given local server instance; the server does not need to be already running, as the build will create various standalone server configurations and start multiple instances. The 'fast' profile can also be used to run tests concurrently. Updating WS stack ------------------- In some cases it might be needed to build the ws stack and install it on a specified server instance without running the integration testsuite; this is achieved as follows: > mvn -PwildflyXYZ -Dserver.home=/foo/bar package If a server.home property is not provided, the build creates a zip archive with a vanilla WildFly server patched with the current WS stack: > mvn -PwildflyXYZ package the zip file path is modules/dist/target/jbossws-cxf-dist-${project.version}-test-server.zip Cleaning up ------------- The project is cleaned up as follows: > mvn -Pdist,testsuite clean As you can see, it is way simpler then what we used to have in previous JBossWS versions. The idea is to completely discontinue the binary and source distributions and distribute a simple zip file of the svn/git project, which the user will simply build like we do as explained above. One of the major differences with the past is that having a local and running WildFly server instance is not strictly required anymore, as the JBossWS build goes and fetch the WildFly distros, self-installs on them, start them and run the integration testsuites. Few different server configurations are actually started (3 at the moment, I've been grouping together all the server setup requirements that can live together: security domains, system properties, etc.) The server configurations are obtained by copying the standalone.xml and patching it using gmaven plugin (Groovy), which seems quite simple and easy to maintain. Any comment, let me know. There's still quite a lot to do, to cleanup a bunch of stuff that we don't use anymore in the build, convert the shared and spring testsuites and nail down some transient test failures that can be reproduced when using the -Pfast profile (likely Arquillian related issues). Cheers Alessio

This does not run locally for me. Looks like I'm missing some mvn settings repo ref.

[WARNING] Could not transfer metadata org.eclipse.wst.common:uriresolver/maven-metadata.xml from/to maven-repository.dev.java.net (http://download.java.net/maven/1): No connector available to access repository maven-repository.dev.java.net (http://download.java.net/maven/1) of type legacy using the available factories WagonRepositoryConnectorFactory [WARNING] Failure to transfer org.eclipse.wst.sse:core/maven-metadata.xml from http://download.java.net/maven/1 was cached in the local repository, resolution will not be reattempted until the update interval of maven-repository.dev.java.net has elapsed or updates are forced. Original error: Could not transfer metadata org.eclipse.wst.sse:core/maven-metadata.xml from/to maven-repository.dev.java.net (http://download.java.net/maven/1): No connector available to access repository maven-repository.dev.java.net (http://download.java.net/maven/1) of type legacy using the available factories WagonRepositoryConnectorFactory ----- Original Message ----- > From: "Alessio Soldano" <asoldano(a)redhat.com&gt; > To: jbossws-dev(a)lists.jboss.org > Sent: Tuesday, December 16, 2014 8:26:55 AM > Subject: Re: [jbossws-dev] JBossWS testsuite reorganization > > I've set up a Hudson instance testing the 'arquillian' branch; the build > is working fine and all (converted) tests are currently passing :-) > See http://jbossws-qa.jboss.org:8280/hudson/ > The hudson setup is simpler / smaller. > Cheers > Alessio > > On 11/12/14 17:44, Alessio Soldano wrote: >> The first initial prototype for the new testsuite and build of JBossWS 5 >> is available at >> https://svn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian >> I have converted the cxf-specific testuite to use Arquillian and >> temporarly disabled the shared-testsuite (it will be re-enabled once it >> will be converted too). >> A basic README file has been added to the root of the project with build >> instructions; here is the content anyway: >> >> Building and running the testsuite >> ------------------------------------ >> The build follows the usual Maven flow; a wilflyXYZ profile has to be >> specified to tell the project which target container to use for >> integration tests; if no wildflyXYZ profile is specified, the >> integration tests are skipped. >> >> > mvn -PwildflyXYZ integration-test >> >> The -Dserver.home=/foo/bar option can be used to run the testsuite >> against a given local server instance; the server does not need to be >> already running, as the build will create various standalone server >> configurations and start multiple instances. >> >> The 'fast' profile can also be used to run tests concurrently. >> >> Updating WS stack >> ------------------- >> In some cases it might be needed to build the ws stack and install it on >> a specified server instance without running the integration testsuite; >> this is achieved as follows: >> >> > mvn -PwildflyXYZ -Dserver.home=/foo/bar package >> >> If a server.home property is not provided, the build creates a zip >> archive with a vanilla WildFly server patched with the current WS stack: >> >> > mvn -PwildflyXYZ package >> >> the zip file path is >> modules/dist/target/jbossws-cxf-dist-${project.version}-test-server.zip >> >> >> Cleaning up >> ------------- >> The project is cleaned up as follows: >> >> > mvn -Pdist,testsuite clean >> >> >> >> >> As you can see, it is way simpler then what we used to have in previous >> JBossWS versions. The idea is to completely discontinue the binary and >> source distributions and distribute a simple zip file of the svn/git >> project, which the user will simply build like we do as explained above. >> >> One of the major differences with the past is that having a local and >> running WildFly server instance is not strictly required anymore, as the >> JBossWS build goes and fetch the WildFly distros, self-installs on them, >> start them and run the integration testsuites. Few different server >> configurations are actually started (3 at the moment, I've been grouping >> together all the server setup requirements that can live together: >> security domains, system properties, etc.) >> The server configurations are obtained by copying the standalone.xml and >> patching it using gmaven plugin (Groovy), which seems quite simple and >> easy to maintain. >> >> Any comment, let me know. There's still quite a lot to do, to cleanup a >> bunch of stuff that we don't use anymore in the build, convert the >> shared and spring testsuites and nail down some transient test failures >> that can be reproduced when using the -Pfast profile (likely Arquillian >> related issues). >> >> Cheers >> Alessio >> > > -- > Alessio Soldano > Web Service Lead, JBoss > > _______________________________________________ > jbossws-dev mailing list > jbossws-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jbossws-dev >

Below I'm sharing some notes on the actual testsuite migration to Arquillian, so that anybody can contribute / help on converting the remaining tests: * Server configurations The idea is to keep the set as small as possible, that is to start the minimum number of containers (to save memory); the cxf-tests testsuite currently starts 3 different instances, defined in src/test/etc/arquillian.xml. The first one, with qualifier 'jboss', is the one most of the tests are run against, the second and the third ones are for tests that require a specific instances (in these cases because they need a custom https connector and because they modify the default ws client/endpoint predefined configurations). Keep in mind that tests can be run concurrently (unless they're named -Forked, in which case they're run in a separate process, one at a time). Each server configuration (besides for the first one) needs a port offset, to avoid port clashes; I've been using 10000, 20000, etc. as offsets. I've also set tight memory parameters for the instances that are meant to be used for few or just a single test. The first container instance has the default 'mode', which is 'suite' and means the server is automatically started by Arquillian at the beginning of the testsuite and stopped at the end of the testsuites; the other containers have 'mode = manual' meaning they're not automatically started by Arquillian but the testcases are expected to do that; I've been doing this to defer the memory allocation to when the server is actually needed. The pom.xml for the testsuite project also needs a 'fast' profile to start the same container instances defined in arquillian.xml using the wildfly-maven-plugin; this is needed because Arquillian does not seem to be able to start the containers properly when surefire runs in parallel mode; for this mechanism to work, the 'allowConnectingToRunningServer' option is set to true in arquillian.xml. As far as the standalone.xml setup is concerned, the gmaven-plugin is used to produce a specific standalone configuration for each of the instances that are to be started by Arquillian. We have a groovy script for each configuration that parses the standalone xml tree, modifies it and saves it in a different file. Have a look at the scripts I did for the cxf-tests testsuite to get an idea; the customizations in there are meant to replace the work previously done at runtime by the RemoteDeployer's addSecurityDomain, addHttpsConnector, setSystemProperty methods. * Deployments I've managed to have unique deployment archive names and a 1-1 relationship between archive deployments and testcases. This makes things easier and prevents issues with Arquillian deployment. As we used to have tests relying on the same deployment, I had to either merge tests into a single test class (when it made sense) or duplicate the deployment and use two different names, one for each test. For the -client.jar deployments, that are used to tune the client classpath, I've added a getClientJarPaths() method to JBossWSTest which has to be overridden to return the string path to client deployment on the filesystem. Test methods which need the classpath to be tuned are to be annotated with the @org.jboss.wsf.test.WrapThreadContextClassLoader annotation. See [1] for an example. * Tests Tests are of course to be annotated with @RunWith(Arquillian.class); note that all test methods have to be annotated with @Test. I've used @Deployment(testable=false) for methods providing the deployment most of the times; as a matter of fact we're running all the tests out-of-container so far [3], with @RunAsClient annotation, hence the 'testable = false'. The managed flag is usually not specified (default is true) to let Arquillian manage the deploy/undeploy; the only cases in which we use 'managed = false' is when we need to actually control the deploy/undeploy; a notable example is [2] which uses a special container instance ("ssl-mutual-auth"), starts it in a @Before annotated method and manually deploy the archive on it. Please try to make the less assumption as possible on the deployment publish location, as that really depends on the Arquillian container setup. So, use the @ArquillianResource annotation whenever possible to have the deployment base URL injected in the tests and avoid hard-coding port numbers in the tests. That's all for now, I'll possibly write more if I remember of anything additional worth sharing. Cheers Alessio [1] http://anonsvn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian/modu... [2] http://anonsvn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian/modu... [3] Actually, we do have in-container tests relying on servlets that will be refactored into in-container Arquillian tests in a future effort. On 16/12/14 14:26, Alessio Soldano wrote: > I've set up a Hudson instance testing the 'arquillian' branch; the build > is working fine and all (converted) tests are currently passing :-) > See http://jbossws-qa.jboss.org:8280/hudson/ > The hudson setup is simpler / smaller. > Cheers > Alessio > > On 11/12/14 17:44, Alessio Soldano wrote: >> The first initial prototype for the new testsuite and build of JBossWS 5 >> is available at >> https://svn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian >> I have converted the cxf-specific testuite to use Arquillian and >> temporarly disabled the shared-testsuite (it will be re-enabled once it >> will be converted too). >> A basic README file has been added to the root of the project with build >> instructions; here is the content anyway: >> >> Building and running the testsuite >> ------------------------------------ >> The build follows the usual Maven flow; a wilflyXYZ profile has to be >> specified to tell the project which target container to use for >> integration tests; if no wildflyXYZ profile is specified, the >> integration tests are skipped. >> >> > mvn -PwildflyXYZ integration-test >> >> The -Dserver.home=/foo/bar option can be used to run the testsuite >> against a given local server instance; the server does not need to be >> already running, as the build will create various standalone server >> configurations and start multiple instances. >> >> The 'fast' profile can also be used to run tests concurrently. >> >> Updating WS stack >> ------------------- >> In some cases it might be needed to build the ws stack and install it on >> a specified server instance without running the integration testsuite; >> this is achieved as follows: >> >> > mvn -PwildflyXYZ -Dserver.home=/foo/bar package >> >> If a server.home property is not provided, the build creates a zip >> archive with a vanilla WildFly server patched with the current WS stack: >> >> > mvn -PwildflyXYZ package >> >> the zip file path is >> modules/dist/target/jbossws-cxf-dist-${project.version}-test-server.zip >> >> >> Cleaning up >> ------------- >> The project is cleaned up as follows: >> >> > mvn -Pdist,testsuite clean >> >> >> >> >> As you can see, it is way simpler then what we used to have in previous >> JBossWS versions. The idea is to completely discontinue the binary and >> source distributions and distribute a simple zip file of the svn/git >> project, which the user will simply build like we do as explained above. >> >> One of the major differences with the past is that having a local and >> running WildFly server instance is not strictly required anymore, as the >> JBossWS build goes and fetch the WildFly distros, self-installs on them, >> start them and run the integration testsuites. Few different server >> configurations are actually started (3 at the moment, I've been grouping >> together all the server setup requirements that can live together: >> security domains, system properties, etc.) >> The server configurations are obtained by copying the standalone.xml and >> patching it using gmaven plugin (Groovy), which seems quite simple and >> easy to maintain. >> >> Any comment, let me know. There's still quite a lot to do, to cleanup a >> bunch of stuff that we don't use anymore in the build, convert the >> shared and spring testsuites and nail down some transient test failures >> that can be reproduced when using the -Pfast profile (likely Arquillian >> related issues). >> >> Cheers >> Alessio >>

Some notes from my experiments with jbossws-arquillian-branch: I tried to build everything with JDK 1.6, the only problem was with containers: jbossws-wildfly810 -- javac: invalid target release: 1.7 http://anonsvn.jboss.org/repos/jbossws/maven/parent/tags/jbossws-parent-1... says <source>1.6</source> <target>1.6</target> ==> AI: new parent with 1.7 as source and target ? ==> AI: add info into README to use JDK 1.7+

mvn -Dmaven.test.failure.ignore=true -Pwildfly810 package requires org.jboss.ws:jbossws-wildfly800-server-integration:jar:5.0.0-SNAPSHOT <dependency> <groupId>org.jboss.ws</groupId> <artifactId>jbossws-wildfly800-server-integration</artifactId> <scope>provided</scope> </dependency> <dependency> <groupId>org.jboss.ws</groupId> <artifactId>jbossws-wildfly810-server-integration</artifactId> <scope>provided</scope> </dependency> ==> AI: put these dependencies into separate profiles - wildfly810, wildfly800, etc. ?

http://anonsvn.jboss.org/repos/jbossws/container/wildfly81/branches/jboss... <wildfly.version>8.1.0.CR2</wildfly.version> ==> AI: update to GA

anonsvn.jboss.org/repos/jbossws/container/wildfly82 ... not available ... ==> AI: create wildfly82 container or add note (where ?) wildfly81 can be used with WildFly 8.2.0.Final

mvn -Dmaven.test.failure.ignore=true -Pwildfly810 package among others produces: ROOT/target/antrun/build-main.xml ==> remove ? build-main.xml present in other modules too, afaik ant makes sense in dist module for WF preparation ==> AI: clean unnecessary ant related stuff

modules/dist/target/wildfly-8.1.0.Final/bin/standalone.sh is not executable ==> AI: make it and all .sh scripts executable ?

I see following warnings at the end of TS execution - mvn -Dmaven.test.failure.ignore=true -Pwildfly810 clean integration-test: Warning: org.apache.xerces.parsers.SAXParser: Property 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized. Warning: org.apache.xerces.parsers.SAXParser: Property 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not recognized. ==> AI: cleanup, but how ... upgrade xerces ?

Following tests failed - mvn -Dmaven.test.failure.ignore=true -Pwildfly810 clean integration-test: testProbeAndResolve(org.jboss.test.ws.jaxws.samples.wsdd.WSDiscoveryTestCase): Number of matches can not be 0. testClientSide(org.jboss.test.ws.jaxws.cxf.udp.UDPEndpointAPITestCase): Could not send Message. testBroadcastUDP(org.jboss.wsf.stack.cxf.addons.transports.udp.UDPTransportTest): Could not send Message. "Problem" is in firewall ... ==> AI: README -- info about proper environment setup ?

> As you can see, it is way simpler then what we used to have in previous > JBossWS versions. The idea is to completely discontinue the binary and > source distributions and distribute a simple zip file of the svn/git > project, which the user will simply build like we do as explained above. This must be ensured to work (with empty local maven repo) ... all the time :)

Makes kind of sense to drop maintenance of ant based installation procedures, this will push users to take a look at JBossWS code. There is possibility that this step will discourage some people. But I guess it's acceptable risk.

Maybe for latest JBossWS on latest WildFly we could provide zip directly - from modules/dist/target/jbossws-cxf-dist-5.0.0-SNAPSHOT-test-server.zip It's already generated in jobs in http://jbossws-qa.jboss.org:8280/hudson/ ... so archiving the artifact shouldn't be a problem ...

>> modules/dist/target/wildfly-8.1.0.Final/bin/standalone.sh is not executable >> ==> AI: make it and all .sh scripts executable ? > Well, this is what comes from the wildfly distribution zip, perhaps the > files in there should already be marked as executable (assuming that's > doable..?) ? Depends how the wildfly distribution zip is generated ... I downloaded and unzipped several archives and scripts were executable. For example Maven zip - http://www.eng.lsu.edu/mirrors/apache/maven/maven-3/3.2.3/binaries/apache... Rostislav _______________________________________________ jbossws-dev mailing list jbossws-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jbossws-dev