[jbossws-dev] JBossWS testsuite reorganization

[Alessio Soldano]

Folks,
with current development on jbossws trunk aimed at version 5, I believe 
it's time consider rethinking the testsuite a bit.

Current status
--------------------
Historically, we've been running the integration tests against a JBoss 
AS container that has to be started before the testsuite. The target 
container is a vanilla distribution that suits with most of the tests, 
except for few ones requiring a https connector or a specific security 
domain to be available. Implementations of the 
org.jboss.wsf.spi.deployer.Deployer interface are available for each 
target container and allow deploying test archives as well as setting 
security domains / https connector by internally relying on the 
container management API.
Different maven profiles trigger different implementations of Deployer 
(due to different artifact dependencies / classpath), allowing running 
the testsuite the same way against different target containers. When it 
comes to continuous integration, the idea is that a QA job first updates 
a target container with the current jbossws libraries, then it starts 
the container and runs the jbossws testsuite against it.

Issues / reasons for changing
---------------------------------------
The current approach is working pretty much fine, but suffers from some 
limitations:
* it needs a container to be already running when the integration test 
is triggered; it would be great if we could have the container started / 
stopped by the test itself
* we're limited to running the testsuite against a predefined container 
profile (standalone.xml, standalone-full.xml ): while most of the tests 
makes sense when run against standalone.xml profile (because the 
webservices subsystem is on in it), we have some tests requiring 
subsystems available on different profiles (for instance, the messaging 
subsystem which is in standalone-full.xml or the xts subsystem which is 
in standalone-xts.xml). Currently we run the testsuite in Spring mode 
against standalone-full.xml, basically because till CXF 2.7.x the 
SOAP-over-JMS implementation has been available with Spring only. That's 
not the case anymore, on CXF 3.0.0, so we'd likely have to test it in 
both spring and non-spring mode, but that needs the messaging subsystem 
which is not in standalone.xml
* we have few tests (but might have more in future) that actually 
require further additional changes to the server; that includes adding 
users, messaging queues, setting server side system properties at boot, 
... The only way to achieve that atm is to have the QA job run a CLI 
script to modify the server or set a system property on command line 
when starting the server jvm
* we have tests affecting the server management model, for instance 
changing the way the webservices subsystem model rewrites wsdl 
soap:address attributes; that makes it impossible to run those tests at 
the same time of any other tests

Proposals / idea
---------------------
I'm listening to proposals to revisit the way the testsuite is run and 
better address the point above.
Something I've been thinking about is relying on Arquillian, similarly 
to what the WFLY testsuite does. What's not trivial is the fact that we 
still need to have different version of the Arquillian SPI 
implementation (that is, different WFLY / EAP versions) used depending 
on a maven profile specified when running the testsuite; moreover we 
also need to be able to run against the result of building the latest 
WFLY master (iow, not a released target for which a 
wildfly-arquillian-container-xyz is available on the repository). We 
could have custom server profile files (standalone-abc.xml) in the 
jbossws sources for the supported containers to be used for starting 
specific versions of the containers through Arquillian. The integration 
tests could be split into groups meant to run against the various 
different profiles of the selected target container. Pretty much 
anything we need to customize on the server that can't be done using the 
Deployer approach can actually be done through a proper profile file (I 
would still keep the Deployer approach when working to limit the number 
of profile files to maintain, as there's a lot of stuff in there). 
Actually, we could even think about having a prepare phase before the 
actual testsuite that runs given CLI scripts to obtain the desired 
profile configuration.

Next steps
---------------
Please provide feedback / ideas. Then we need a plan and eventually 
jiras created.

Cheers
Alessio

-- 
Alessio Soldano
Web Service Lead, JBoss