[jbossws-dev] JBossWS testsuite reorganization

[Alessio Soldano]

Below I'm sharing some notes on the actual testsuite migration to 
Arquillian, so that anybody can contribute / help on converting the 
remaining tests:

* Server configurations

The idea is to keep the set as small as possible, that is to start the 
minimum number of containers (to save memory); the cxf-tests testsuite 
currently starts 3 different instances, defined in 
src/test/etc/arquillian.xml. The first one, with qualifier 'jboss', is 
the one most of the tests are run against, the second and the third ones 
are for tests that require a specific instances (in these cases because 
they need a custom https connector and because they modify the default 
ws client/endpoint predefined configurations). Keep in mind that tests 
can be run concurrently (unless they're named -Forked, in which case 
they're run in a separate process, one at a time). Each server 
configuration (besides for the first one) needs a port offset, to avoid 
port clashes; I've been using 10000, 20000, etc. as offsets. I've also 
set tight memory parameters for the instances that are meant to be used 
for few or just a single test. The first container instance has the 
default 'mode', which is 'suite' and means the server is automatically 
started by Arquillian at the beginning of the testsuite and stopped at 
the end of the testsuites; the other containers have 'mode = manual' 
meaning they're not automatically started by Arquillian but the 
testcases are expected to do that; I've been doing this to defer the 
memory allocation to when the server is actually needed.
The pom.xml for the testsuite project also needs a 'fast' profile to 
start the same container instances defined in arquillian.xml using the 
wildfly-maven-plugin; this is needed because Arquillian does not seem to 
be able to start the containers properly when surefire runs in parallel 
mode; for this mechanism to work, the 'allowConnectingToRunningServer' 
option is set to true in arquillian.xml.
As far as the standalone.xml setup is concerned, the gmaven-plugin is 
used to produce a specific standalone configuration for each of the 
instances that are to be started by Arquillian. We have a groovy script 
for each configuration that parses the standalone xml tree, modifies it 
and saves it in a different file.
Have a look at the scripts I did for the cxf-tests testsuite to get an 
idea; the customizations in there are meant to replace the work 
previously done at runtime by the RemoteDeployer's addSecurityDomain, 
addHttpsConnector, setSystemProperty methods.

* Deployments

I've managed to have unique deployment archive names and a 1-1 
relationship between archive deployments and testcases. This makes 
things easier and prevents issues with Arquillian deployment. As we used 
to have tests relying on the same deployment, I had to either merge 
tests into a single test class (when it made sense) or duplicate the 
deployment and use two different names, one for each test.
For the -client.jar deployments, that are used to tune the client 
classpath, I've added a getClientJarPaths() method to JBossWSTest which 
has to be overridden to return the string path to client deployment on 
the filesystem. Test methods which need the classpath to be tuned are to 
be annotated with the @org.jboss.wsf.test.WrapThreadContextClassLoader 
annotation. See [1] for an example.

* Tests

Tests are of course to be annotated with @RunWith(Arquillian.class); 
note that all test methods have to be annotated with @Test.
I've used @Deployment(testable=false) for methods providing the 
deployment most of the times; as a matter of fact we're running all the 
tests out-of-container so far [3], with @RunAsClient annotation, hence 
the 'testable = false'. The managed flag is usually not specified 
(default is true) to let Arquillian manage the deploy/undeploy; the only 
cases in which we use 'managed = false' is when we need to actually 
control the deploy/undeploy; a notable example is [2] which uses a 
special container instance ("ssl-mutual-auth"), starts it in a @Before 
annotated method and manually deploy the archive on it.
Please try to make the less assumption as possible on the deployment 
publish location, as that really depends on the Arquillian container 
setup. So, use the @ArquillianResource annotation whenever possible to 
have the deployment base URL injected in the tests and avoid hard-coding 
port numbers in the tests.

That's all for now, I'll possibly write more if I remember of anything 
additional worth sharing.
Cheers
Alessio


[1] 
http://anonsvn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian/modules/testsuite/cxf-tests/src/test/java/org/jboss/test/ws/jaxws/samples/wsse/policy/basic/AnnotatedSignEncryptTestCase.java
[2] 
http://anonsvn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian/modules/testsuite/cxf-tests/src/test/java/org/jboss/test/ws/jaxws/samples/wsse/policy/oasis/WSSecurityPolicyExamples23xTestCase.java
[3] Actually, we do have in-container tests relying on servlets that 
will be refactored into in-container Arquillian tests in a future effort.

On 16/12/14 14:26, Alessio Soldano wrote:
> I've set up a Hudson instance testing the 'arquillian' branch; the build
> is working fine and all (converted) tests are currently passing :-)
> See http://jbossws-qa.jboss.org:8280/hudson/
> The hudson setup is simpler / smaller.
> Cheers
> Alessio
>
> On 11/12/14 17:44, Alessio Soldano wrote:
>> The first initial prototype for the new testsuite and build of JBossWS 5
>> is available at
>> https://svn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian
>> I have converted the cxf-specific testuite to use Arquillian and
>> temporarly disabled the shared-testsuite (it will be re-enabled once it
>> will be converted too).
>> A basic README file has been added to the root of the project with build
>> instructions; here is the content anyway:
>>
>>     Building and running the testsuite
>> ------------------------------------
>> The build follows the usual Maven flow; a wilflyXYZ profile has to be
>> specified to tell the project which target container to use for
>> integration tests; if no wildflyXYZ profile is specified, the
>> integration tests are skipped.
>>
>>    > mvn -PwildflyXYZ integration-test
>>
>> The -Dserver.home=/foo/bar option can be used to run the testsuite
>> against a given local server instance; the server does not need to be
>> already running, as the build will create various standalone server
>> configurations and start multiple instances.
>>
>> The 'fast' profile can also be used to run tests concurrently.
>>
>>     Updating WS stack
>> -------------------
>> In some cases it might be needed to build the ws stack and install it on
>> a specified server instance without running the integration testsuite;
>> this is achieved as follows:
>>
>>    > mvn -PwildflyXYZ -Dserver.home=/foo/bar package
>>
>> If a server.home property is not provided, the build creates a zip
>> archive with a vanilla WildFly server patched with the current WS stack:
>>
>>    > mvn -PwildflyXYZ package
>>
>> the zip file path is
>> modules/dist/target/jbossws-cxf-dist-${project.version}-test-server.zip
>>
>>
>>     Cleaning up
>> -------------
>> The project is cleaned up as follows:
>>
>>    > mvn -Pdist,testsuite clean
>>
>>
>>
>>
>> As you can see, it is way simpler then what we used to have in previous
>> JBossWS versions. The idea is to completely discontinue the binary and
>> source distributions and distribute a simple zip file of the svn/git
>> project, which the user will simply build like we do as explained above.
>>
>> One of the major differences with the past is that having a local and
>> running WildFly server instance is not strictly required anymore, as the
>> JBossWS build goes and fetch the WildFly distros, self-installs on them,
>> start them and run the integration testsuites. Few different server
>> configurations are actually started (3 at the moment, I've been grouping
>> together all the server setup requirements that can live together:
>> security domains, system properties, etc.)
>> The server configurations are obtained by copying the standalone.xml and
>> patching it using gmaven plugin (Groovy), which seems quite simple and
>> easy to maintain.
>>
>> Any comment, let me know. There's still quite a lot to do, to cleanup a
>> bunch of stuff that we don't use anymore in the build, convert the
>> shared and spring testsuites and nail down some transient test failures
>> that can be reproduced when using the -Pfast profile (likely Arquillian
>> related issues).
>>
>> Cheers
>> Alessio
>>
>


-- 
Alessio Soldano
Web Service Lead, JBoss