From asoldano at redhat.com Mon Feb 9 12:33:08 2015 From: asoldano at redhat.com (Alessio Soldano) Date: Mon, 09 Feb 2015 18:33:08 +0100 Subject: [jbossws-dev] JBossWS testsuite reorganization In-Reply-To: <5494536E.7050404@redhat.com> References: <539B1B2C.4020105@redhat.com> <539EC49E.40905@redhat.com> <539EE0D7.2020907@redhat.com> <177181653.30699830.1403767818007.JavaMail.zimbra@redhat.com> <5469D649.1040205@redhat.com> <5472FE1D.4030306@redhat.com> <54731435.2050605@redhat.com> <5489C9E7.8000201@redhat.com> <5490331F.2090609@redhat.com> <5494536E.7050404@redhat.com> Message-ID: <54D8EF54.4000203@redhat.com> The 'arquillian' branch has been merged to trunk. Any further development on this topic will go on on trunk. I'll delete the arquillian branch tomorrow after having merged the hudson configuration from the corresponding branch to trunk. Cheers Alessio On 19/12/14 17:33, Alessio Soldano wrote: > Below I'm sharing some notes on the actual testsuite migration to > Arquillian, so that anybody can contribute / help on converting the > remaining tests: > > * Server configurations > > The idea is to keep the set as small as possible, that is to start the > minimum number of containers (to save memory); the cxf-tests testsuite > currently starts 3 different instances, defined in > src/test/etc/arquillian.xml. The first one, with qualifier 'jboss', is > the one most of the tests are run against, the second and the third ones > are for tests that require a specific instances (in these cases because > they need a custom https connector and because they modify the default > ws client/endpoint predefined configurations). Keep in mind that tests > can be run concurrently (unless they're named -Forked, in which case > they're run in a separate process, one at a time). Each server > configuration (besides for the first one) needs a port offset, to avoid > port clashes; I've been using 10000, 20000, etc. as offsets. I've also > set tight memory parameters for the instances that are meant to be used > for few or just a single test. The first container instance has the > default 'mode', which is 'suite' and means the server is automatically > started by Arquillian at the beginning of the testsuite and stopped at > the end of the testsuites; the other containers have 'mode = manual' > meaning they're not automatically started by Arquillian but the > testcases are expected to do that; I've been doing this to defer the > memory allocation to when the server is actually needed. > The pom.xml for the testsuite project also needs a 'fast' profile to > start the same container instances defined in arquillian.xml using the > wildfly-maven-plugin; this is needed because Arquillian does not seem to > be able to start the containers properly when surefire runs in parallel > mode; for this mechanism to work, the 'allowConnectingToRunningServer' > option is set to true in arquillian.xml. > As far as the standalone.xml setup is concerned, the gmaven-plugin is > used to produce a specific standalone configuration for each of the > instances that are to be started by Arquillian. We have a groovy script > for each configuration that parses the standalone xml tree, modifies it > and saves it in a different file. > Have a look at the scripts I did for the cxf-tests testsuite to get an > idea; the customizations in there are meant to replace the work > previously done at runtime by the RemoteDeployer's addSecurityDomain, > addHttpsConnector, setSystemProperty methods. > > * Deployments > > I've managed to have unique deployment archive names and a 1-1 > relationship between archive deployments and testcases. This makes > things easier and prevents issues with Arquillian deployment. As we used > to have tests relying on the same deployment, I had to either merge > tests into a single test class (when it made sense) or duplicate the > deployment and use two different names, one for each test. > For the -client.jar deployments, that are used to tune the client > classpath, I've added a getClientJarPaths() method to JBossWSTest which > has to be overridden to return the string path to client deployment on > the filesystem. Test methods which need the classpath to be tuned are to > be annotated with the @org.jboss.wsf.test.WrapThreadContextClassLoader > annotation. See [1] for an example. > > * Tests > > Tests are of course to be annotated with @RunWith(Arquillian.class); > note that all test methods have to be annotated with @Test. > I've used @Deployment(testable=false) for methods providing the > deployment most of the times; as a matter of fact we're running all the > tests out-of-container so far [3], with @RunAsClient annotation, hence > the 'testable = false'. The managed flag is usually not specified > (default is true) to let Arquillian manage the deploy/undeploy; the only > cases in which we use 'managed = false' is when we need to actually > control the deploy/undeploy; a notable example is [2] which uses a > special container instance ("ssl-mutual-auth"), starts it in a @Before > annotated method and manually deploy the archive on it. > Please try to make the less assumption as possible on the deployment > publish location, as that really depends on the Arquillian container > setup. So, use the @ArquillianResource annotation whenever possible to > have the deployment base URL injected in the tests and avoid hard-coding > port numbers in the tests. > > That's all for now, I'll possibly write more if I remember of anything > additional worth sharing. > Cheers > Alessio > > > [1] > http://anonsvn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian/modules/testsuite/cxf-tests/src/test/java/org/jboss/test/ws/jaxws/samples/wsse/policy/basic/AnnotatedSignEncryptTestCase.java > [2] > http://anonsvn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian/modules/testsuite/cxf-tests/src/test/java/org/jboss/test/ws/jaxws/samples/wsse/policy/oasis/WSSecurityPolicyExamples23xTestCase.java > [3] Actually, we do have in-container tests relying on servlets that > will be refactored into in-container Arquillian tests in a future effort. > > On 16/12/14 14:26, Alessio Soldano wrote: >> I've set up a Hudson instance testing the 'arquillian' branch; the build >> is working fine and all (converted) tests are currently passing :-) >> See http://jbossws-qa.jboss.org:8280/hudson/ >> The hudson setup is simpler / smaller. >> Cheers >> Alessio >> >> On 11/12/14 17:44, Alessio Soldano wrote: >>> The first initial prototype for the new testsuite and build of JBossWS 5 >>> is available at >>> https://svn.jboss.org/repos/jbossws/stack/cxf/branches/arquillian >>> I have converted the cxf-specific testuite to use Arquillian and >>> temporarly disabled the shared-testsuite (it will be re-enabled once it >>> will be converted too). >>> A basic README file has been added to the root of the project with build >>> instructions; here is the content anyway: >>> >>> Building and running the testsuite >>> ------------------------------------ >>> The build follows the usual Maven flow; a wilflyXYZ profile has to be >>> specified to tell the project which target container to use for >>> integration tests; if no wildflyXYZ profile is specified, the >>> integration tests are skipped. >>> >>> > mvn -PwildflyXYZ integration-test >>> >>> The -Dserver.home=/foo/bar option can be used to run the testsuite >>> against a given local server instance; the server does not need to be >>> already running, as the build will create various standalone server >>> configurations and start multiple instances. >>> >>> The 'fast' profile can also be used to run tests concurrently. >>> >>> Updating WS stack >>> ------------------- >>> In some cases it might be needed to build the ws stack and install it on >>> a specified server instance without running the integration testsuite; >>> this is achieved as follows: >>> >>> > mvn -PwildflyXYZ -Dserver.home=/foo/bar package >>> >>> If a server.home property is not provided, the build creates a zip >>> archive with a vanilla WildFly server patched with the current WS stack: >>> >>> > mvn -PwildflyXYZ package >>> >>> the zip file path is >>> modules/dist/target/jbossws-cxf-dist-${project.version}-test-server.zip >>> >>> >>> Cleaning up >>> ------------- >>> The project is cleaned up as follows: >>> >>> > mvn -Pdist,testsuite clean >>> >>> >>> >>> >>> As you can see, it is way simpler then what we used to have in previous >>> JBossWS versions. The idea is to completely discontinue the binary and >>> source distributions and distribute a simple zip file of the svn/git >>> project, which the user will simply build like we do as explained above. >>> >>> One of the major differences with the past is that having a local and >>> running WildFly server instance is not strictly required anymore, as the >>> JBossWS build goes and fetch the WildFly distros, self-installs on them, >>> start them and run the integration testsuites. Few different server >>> configurations are actually started (3 at the moment, I've been grouping >>> together all the server setup requirements that can live together: >>> security domains, system properties, etc.) >>> The server configurations are obtained by copying the standalone.xml and >>> patching it using gmaven plugin (Groovy), which seems quite simple and >>> easy to maintain. >>> >>> Any comment, let me know. There's still quite a lot to do, to cleanup a >>> bunch of stuff that we don't use anymore in the build, convert the >>> shared and spring testsuites and nail down some transient test failures >>> that can be reproduced when using the -Pfast profile (likely Arquillian >>> related issues). >>> >>> Cheers >>> Alessio >>> > -- Alessio Soldano Web Service Lead, JBoss