Paul Robinson [
http://community.jboss.org/people/paul.robinson] created the document:
"XTSTestingCurrentStatus"
To view the document, visit:
http://community.jboss.org/docs/DOC-17242
--------------------------------------------------------------
* XTS Testing
h1. Current Status and Roadmap
h1.
This Document describes the current status of the XTS tests and what technology they use.
There are currently three sets of tests, Unit, Interop and recovery. Each of these is
described in turn, and a list of required improvements is presented.
All improvements are targeted at Narayana 5.x unless specified otherwise.
h2. Unit tests
Each XTS component has a set of unit tests. These tests need to be ran within an instance
of JBoss AS. These tests are fully automated by an Ant script ran by Hudson.
h3. Improvements
1. Migrate to Maven. For consistency with rest of the Narayna project
2. Move into the AS test suite. (
https://issues.jboss.org/browse/JBQA-5191
https://issues.jboss.org/browse/JBQA-5191). This will ensure that changes to components,
we depend upon (like JbossWS), that break XTS are spotted at QE time rather than after
release. (EAP 6 requirement)
3. Update to use Arquillian. This would mean that that they can be ran from anywhere that
can run JUnit tests, such as an IDE or maven. It would also automate the app server
lifecycle and test deployment.
4. Remove home-brew SOAP stack. In the past these tests used a mock/simple SOAP stack
developed specifically for the tests. This stack is no longer used as the tests run within
JBoss AS. This code is redundant and should be removed.
5. Automate Emma and collate results. Emma should be used to obtain coverage stats on the
test run. Emma, produces individual reports per test, by default. It would be better to
have these reports combined as we are concerned with the overall test coverage, rather
than the coverage of each test. Ideally we would combine coverage stats over all sets of
tests (unit, interop and recovery). Next step would be to improve the coverage where
necessary.
h2. Interop Tests
We have two sets of interop tests that live in "XTS/interop". These are built
with ant. They are each ran by deploying them as a war to a single JBoss instance which
deploys the services needed by the test. This war also exposes a web interface that on
request runs the tests and relays the results in the http response.
This process is automated by using ant to deploy the war and then making the http request
and validating the response. See here in code for script:
“XTS/localjunit/run-interop-tests.xml”
h3. Improvements
1. Migrate to AS 7.x There is some confusion around what does and doesn't work in AS
7.x. It's possible that these tests have only been tested via the GUI and that
it's the automated steps that don't work.
2. Move into the AS test suite. (
https://issues.jboss.org/browse/JBQA-5192
https://issues.jboss.org/browse/JBQA-5192). (EAP 6 requirement)
3. Update to use Arquillian. This would mean that that they can be ran from anywhere that
can run JUnit tests, such as an IDE or maven. It would also automate the app server
lifecycle and test deployment.
4. Migrate to Maven. For consistency with rest of the Narayna project.
5. Automate Emma and collate results.
h2. Recovery Tests
The recovery tests are the tricky ones. At the moment we have a set of test scenarios that
run in a single JBoss server invoked by a remote client. There are also a set of Byteman
scripts that trigger failure at certain points in a scenario. There are many different
permutations of scenario and Byteman scripts that each create a particular test.
The test scenarios log their progress through the protocol and recovery. On completion of
a test run, a human needs to look over the trace and check that it looks right. This
process is hard to automate as the trace produced can have many valid interleavings. This
is due to the asynchronous nature of the application.
These tests are automated with a bash script, but the output traces must be verified
manually by a human. The other problem is that they currently run in a single JBoss server
which simulates the situation where every party in the protocol crashes and recovers at
the same time. Not that realistic. I assume this is the case because the current tests are
hard enough to run without having many JBoss servers to deal with. We would also need a
way of combining the traces from each server when verifying the outcome of the test. This
could be done by implementing a Byteman helper class, but it would not be trivial.
h3. Improvements
1. *Migrate to Maven*. For consistency with rest of the Narayna project.
2. *Arquillian support*. Use Arquillian to automate the tests and hopefully remove the
human verification step.
3. *Multiple Server Tests*. If step 2) is successful, we could be able to build the more
complex scenarios where each party runs in its own JBoss server.
4. *Additional Tests*. Andrew Dinn has provided a set of additional tests for us to
consider implementing. These should be considered alongside the Emma coverage data for
future work.
5. *Move into AS test suite*. The tests are unlikely to be accepted in their current form,
due to the level of manual intervention required. This improvement can not be made until
we have sufficiently automated the process.
h2.
h2. Notes
* Arquillian supports multiple JBoss servers:*
https://docs.jboss.org/author/display/ARQ/Multiple+Containers
https://docs.jboss.org/author/display/ARQ/Multiple+Containers
* Arquillian doesn't yet support servers that crash. This feature is unlikely to make
it into EAP 6.*
https://issues.jboss.org/browse/ARQ-336
https://issues.jboss.org/browse/ARQ-336
--------------------------------------------------------------
Comment by going to Community
[
http://community.jboss.org/docs/DOC-17242]
Create a new document in JBoss Transactions Development at Community
[
http://community.jboss.org/choose-container!input.jspa?contentType=102&am...]