Current Status and Roadmap
This Document describes the current status of the XTS tests and what technology they use. There are currently three sets of tests, Unit, Interop and recovery. Each of these is described in turn, and a list of required improvements is presented.
All improvements are targeted at Narayana 5.x unless specified otherwise.
Unit tests
Each XTS component has a set of unit tests. These tests need to be ran within an instance of JBoss AS. These tests are fully automated by an Ant script ran by Hudson.
Improvements
- Migrate to Maven. For consistency with rest of the Narayna project
- Move into the AS test suite. (https://issues.jboss.org/browse/JBQA-5191). This will ensure that changes to components, we depend upon (like JbossWS), that break XTS are spotted at QE time rather than after release. (EAP 6 requirement)
- Update to use Arquillian. This would mean that that they can be ran from anywhere that can run JUnit tests, such as an IDE or maven. It would also automate the app server lifecycle and test deployment.
- Remove home-brew SOAP stack. In the past these tests used a mock/simple SOAP stack developed specifically for the tests. This stack is no longer used as the tests run within JBoss AS. This code is redundant and should be removed.
- Automate Emma and collate results. Emma should be used to obtain coverage stats on the test run. Emma, produces individual reports per test, by default. It would be better to have these reports combined as we are concerned with the overall test coverage, rather than the coverage of each test. Ideally we would combine coverage stats over all sets of tests (unit, interop and recovery). Next step would be to improve the coverage where necessary.
Interop Tests
We have two sets of interop tests that live in "XTS/interop". These are built with ant. They are each ran by deploying them as a war to a single JBoss instance which deploys the services needed by the test. This war also exposes a web interface that on request runs the tests and relays the results in the http response.
This process is automated by using ant to deploy the war and then making the http request and validating the response. See here in code for script: “XTS/localjunit/run-interop-tests.xml”
Improvements
- Migrate to AS 7.x There is some confusion around what does and doesn't work in AS 7.x. It's possible that these tests have only been tested via the GUI and that it's the automated steps that don't work.
- Move into the AS test suite. (https://issues.jboss.org/browse/JBQA-5192). (EAP 6 requirement)
- Update to use Arquillian. This would mean that that they can be ran from anywhere that can run JUnit tests, such as an IDE or maven. It would also automate the app server lifecycle and test deployment.
- Migrate to Maven. For consistency with rest of the Narayna project.
- Automate Emma and collate results.
Recovery Tests
The recovery tests are the tricky ones. At the moment we have a set of test scenarios that run in a single JBoss server invoked by a remote client. There are also a set of Byteman scripts that trigger failure at certain points in a scenario. There are many different permutations of scenario and Byteman scripts that each create a particular test.
The test scenarios log their progress through the protocol and recovery. On completion of a test run, a human needs to look over the trace and check that it looks right. This process is hard to automate as the trace produced can have many valid interleavings. This is due to the asynchronous nature of the application.
These tests are automated with a bash script, but the output traces must be verified manually by a human. The other problem is that they currently run in a single JBoss server which simulates the situation where every party in the protocol crashes and recovers at the same time. Not that realistic. I assume this is the case because the current tests are hard enough to run without having many JBoss servers to deal with. We would also need a way of combining the traces from each server when verifying the outcome of the test. This could be done by implementing a Byteman helper class, but it would not be trivial.
Improvements
- Migrate to Maven. For consistency with rest of the Narayna project.
- Arquillian support. Use Arquillian to automate the tests and hopefully remove the human verification step.
- Multiple Server Tests. If step 2) is successful, we could be able to build the more complex scenarios where each party runs in its own JBoss server.
- Additional Tests. Andrew Dinn has provided a set of additional tests for us to consider implementing. These should be considered alongside the Emma coverage data for future work.
- Move into AS test suite. The tests are unlikely to be accepted in their current form, due to the level of manual intervention required. This improvement can not be made until we have sufficiently automated the process.
Notes
- Arquillian supports multiple JBoss servers:
- Arquillian doesn't yet support servers that crash. This feature is unlikely to make it into EAP 6.