Paul Robinson [http://community.jboss.org/people/paul.robinson] modified the document: "XTSTestingCurrentStatus" To view the document, visit: http://community.jboss.org/docs/DOC-17242 -------------------------------------------------------------- h1. XTS Testing h1. Current Status and Roadmap h1. This Document describes the current status of the XTS tests and what technology they use. There are currently three sets of tests, Unit, Interop and recovery. Each of these is described in turn, and a list of required improvements is presented. All improvements are targeted at Narayana 5.x unless specified otherwise. h2. Unit tests Each XTS component has a set of unit tests. These tests need to be ran within an instance of JBoss AS.  These tests are fully automated by an Ant script ran by Hudson. h3. Improvements *High Priority (EAP 6.0)* 1. *Ensure AS 7 compatibility.* See issue JBTM-900 (https://issues.jboss.org/browse/JBTM-900). 2. *Move into the AS test suite*. This will ensure that changes to components, we depend upon (like JbossWS), that break XTS are spotted at QE time rather than after release. (EAP 6 requirement). https://issues.jboss.org/browse/JBQA-5191 JBQA-5191 3. *Additional Testing*. XTS Demo tests https://issues.jboss.org/browse/JBQA-5194 JBQA-5194 *Medium Priority (Post EAP 6)* 1. *Automate Emma and collate results*. Emma should be used to obtain coverage stats on the test run. Emma, produces individual reports per test, by default. It would be better to have these reports combined as we are concerned with the overall test coverage, rather than the coverage of each test. Ideally we would combine coverage stats over all sets of tests (unit, interop and recovery). Next step would be to improve the coverage where necessary. https://issues.jboss.org/browse/JBTM-952 JBTM-952 *Low Priority* *(Post EAP 6)* 1. *Migrate to Maven*. For consistency with rest of the Narayna project. https://issues.jboss.org/browse/JBTM-945 JBTM-945 2. *Update to use Arquillian*. This would mean that that they can be ran from anywhere that can run JUnit tests, such as an IDE or maven. It would also automate the app server lifecycle and test deployment. 3. *Remove home-brew SOAP stack*. In the past these tests used a mock/simple SOAP stack developed specifically for the tests. This stack is no longer used as the tests run within JBoss AS. This code is redundant and should be removed. https://issues.jboss.org/browse/JBTM-953 JBTM-953 h2. Interop Tests We have two sets of interop tests that live in "XTS/interop". These are built with ant. They are each ran by deploying them as a war to a single JBoss instance which deploys the services needed by the test. This war also exposes a web interface that on request runs the tests and relays the results in the http response. This process is automated by using ant to deploy the war and then making the http request and validating the response. See here in code for scripts: “XTS/localjunit/run-interop-tests.xml” and “XTS/localjunit/run-tests.xml” h3. Improvements *High Priority (EAP 6.0)* 1. *Ensure AS 7 compatibility.* See issue JBTM-905 (https://issues.jboss.org/browse/JBTM-905). 2. *Move into the AS test suite*. https://issues.jboss.org/browse/JBQA-5192 JBQA-5192 *Medium Priority (Post EAP 6.0)* 1. *Automate Emma and collate results*. https://issues.jboss.org/browse/JBTM-952 JBTM-952 *Low Priority (Post EAP 6.0)* 1. *Update to use Arquillian*. 2. *Migrate to Maven*. https://issues.jboss.org/browse/JBTM-943 JBTM-943 h2. Recovery Tests The recovery tests are the tricky ones. At the moment we have a set of test scenarios that run in a single JBoss server invoked by a remote client. There are also a set of Byteman scripts that trigger failure at certain points in a scenario. There are many different permutations of scenario and Byteman scripts that each create a particular test. The test scenarios log their progress through the protocol and recovery. On completion of a test run, a human needs to look over the trace and check that it looks right. This process is hard to automate as the trace produced can have many valid interleavings. This is due to the asynchronous nature of the application. These tests are automated with a bash script, but the output traces must be verified manually by a human. The other problem is that they currently run in a single JBoss server which simulates the situation where every party in the protocol is located in the same app server and crashes & recovers at the same time (not that realistic!). In order to test with multiple servers. we would need a way of combining the traces from each server when verifying the outcome of the test. This could be done by implementing a Byteman helper class, but it would not be trivial. h3. Improvements *High Priority (EAP 6.0)* 1. *Move into AS test suite*. The tests are unlikely to be accepted in their current form, due to the  level of manual intervention required. This improvement can not be made until we have sufficiently automated the process. 2. *Ensure AS 7 compatibility.* https://issues.jboss.org/browse/JBTM-923 JBTM-923 3. *Automate running*. Use Arquillian to automate the tests and generate trace output for human verification. https://issues.jboss.org/browse/JBQA-3926 JBQA-3926 https://issues.jboss.org/browse/JBTM-817 JBTM-817 *Medium priority (Post EAP 6.0)* 1. *Automate Emma and collate results*. https://issues.jboss.org/browse/JBTM-952 JBTM-952 *Low Priority (Post EAP 6.0) * 1. *Migrate to Maven.* https://issues.jboss.org/browse/JBTM-944 JBTM-944* * 2. *Automate verification*. Remove the human verification steps. https://issues.jboss.org/browse/JBTM-949 JBTM-949 3. *Multiple Server Tests*. If step 2) is successful, we could be able to build the more complex scenarios where each party runs in its own JBoss server. https://issues.jboss.org/browse/JBTM-950 JBTM-950 4. *Additional Tests*. Andrew Dinn has provided a set of additional tests for us to consider implementing. These should be considered alongside the Emma coverage data for future work. https://issues.jboss.org/browse/JBTM-951 JBTM-951 h2. h2. Notes * Arquillian supports multiple JBoss servers:* https://docs.jboss.org/author/display/ARQ/Multiple+Containers https://docs.jboss.org/author/display/ARQ/Multiple+Containers * Arquillian doesn't yet support servers that crash. This feature is unlikely to make it into EAP 6.* https://issues.jboss.org/browse/ARQ-336 https://issues.jboss.org/browse/ARQ-336 -------------------------------------------------------------- Comment by going to Community [http://community.jboss.org/docs/DOC-17242] Create a new document in JBoss Transactions Development at Community [http://community.jboss.org/choose-container!input.jspa?contentType=102&am...]