[jbossws-dev] JBossWS testsuite reorganization

[Jim Ma]

On 06/13/2014 11:39 PM, Alessio Soldano wrote:
> Folks,
> with current development on jbossws trunk aimed at version 5, I believe
> it's time consider rethinking the testsuite a bit.
>
> Current status
> --------------------
> Historically, we've been running the integration tests against a JBoss
> AS container that has to be started before the testsuite. The target
> container is a vanilla distribution that suits with most of the tests,
> except for few ones requiring a https connector or a specific security
> domain to be available. Implementations of the
> org.jboss.wsf.spi.deployer.Deployer interface are available for each
> target container and allow deploying test archives as well as setting
> security domains / https connector by internally relying on the
> container management API.
> Different maven profiles trigger different implementations of Deployer
> (due to different artifact dependencies / classpath), allowing running
> the testsuite the same way against different target containers. When it
> comes to continuous integration, the idea is that a QA job first updates
> a target container with the current jbossws libraries, then it starts
> the container and runs the jbossws testsuite against it.
>
> Issues / reasons for changing
> ---------------------------------------
> The current approach is working pretty much fine, but suffers from some
> limitations:
> * it needs a container to be already running when the integration test
> is triggered; it would be great if we could have the container started /
> stopped by the test itself
> * we're limited to running the testsuite against a predefined container
> profile (standalone.xml, standalone-full.xml ): while most of the tests
> makes sense when run against standalone.xml profile (because the
> webservices subsystem is on in it), we have some tests requiring
> subsystems available on different profiles (for instance, the messaging
> subsystem which is in standalone-full.xml or the xts subsystem which is
> in standalone-xts.xml). Currently we run the testsuite in Spring mode
> against standalone-full.xml, basically because till CXF 2.7.x the
> SOAP-over-JMS implementation has been available with Spring only. That's
> not the case anymore, on CXF 3.0.0, so we'd likely have to test it in
> both spring and non-spring mode, but that needs the messaging subsystem
> which is not in standalone.xml
> * we have few tests (but might have more in future) that actually
> require further additional changes to the server; that includes adding
> users, messaging queues, setting server side system properties at boot,
> ... The only way to achieve that atm is to have the QA job run a CLI
> script to modify the server or set a system property on command line
> when starting the server jvm
> * we have tests affecting the server management model, for instance
> changing the way the webservices subsystem model rewrites wsdl
> soap:address attributes; that makes it impossible to run those tests at
> the same time of any other tests
As Rosta suggested, the ARQ can help resolve some test issues. We also
need to write some base or helper class to work better with ARQ to make
all these things (configure the standalone.xml and change management model
in runtime) much easier.
>
> Proposals / idea
> ---------------------
> I'm listening to proposals to revisit the way the testsuite is run and
> better address the point above.
> Something I've been thinking about is relying on Arquillian, similarly
> to what the WFLY testsuite does. What's not trivial is the fact that we
> still need to have different version of the Arquillian SPI
> implementation (that is, different WFLY / EAP versions) used depending
> on a maven profile specified when running the testsuite; moreover we
> also need to be able to run against the result of building the latest
> WFLY master (iow, not a released target for which a
> wildfly-arquillian-container-xyz is available on the repository).
Adding profiles for different AS/WF versions looks the thing what we 
have to do. If WFLY master can publish all these snapshot
artifacts during the teamcity build, we can consume it from the snapshot 
repo.
> We
> could have custom server profile files (standalone-abc.xml) in the
> jbossws sources for the supported containers to be used for starting
> specific versions of the containers through Arquillian. The integration
> tests could be split into groups meant to run against the various
> different profiles of the selected target container. Pretty much
> anything we need to customize on the server that can't be done using the
> Deployer approach can actually be done through a proper profile file (I
> would still keep the Deployer approach when working to limit the number
> of profile files to maintain, as there's a lot of stuff in there).
> Actually, we could even think about having a prepare phase before the
> actual testsuite that runs given CLI scripts to obtain the desired
> profile configuration.
  What about we creating more maven modules/directory for different 
server profile like WFLY's basic/domain/xts test maven module ?

>
> Next steps
> ---------------
> Please provide feedback / ideas. Then we need a plan and eventually
> jiras created.
  There could be many work to convert the test archive built with old 
ant script to new shrinkwrap. Is there any tool exists help us with this ?
  We can consider to create some tool to allow us migrate to ARQ much 
easier and faster ?


Cheers,
Jim
>
> Cheers
> Alessio
>