Paul Robinson commented on JBTM-817:
I'm in the process of validating the trace output from these tests and I think doing
this automatically is not going to be a trivial process. I have some ideas around how the
byteman scripts can be re-factored to simplify the validation process and to also improve
scalability so that we can introduce more complex, multi-party tests.
Therefore I am removing the requirement to validate the trace output from this issue.
Validation will come in a later release. Hopefully this will turn this issue into a much
more straight forward task.
So to confirm, the requirements for this issue are to:
# Run the tests as part of the AS7 testsuite (as per Ivo's pattern:
# Automate the running of each crash recovery test without any Byteman rules. This
provides some test coverage and ensures we have a valid system before the Byteman rules
# Automate each valid combination of Byteman script to TestCase. The Byteman scripts
specify which tests are available.
# Gather the output from all test runs in separate files for "eyeball
validation". Until this validation is automated, I (with the help of Andrew
initially) will do this validation. The output files should be named "<bytman
script name>:<test class name>". For example:
Amos: does this make sense? Is this in line with how you are currently implementing the
Automate execution of XTS crash recovery tests
Project: JBoss Transaction Manager
Issue Type: Task
Security Level: Public(Everyone can see)
Affects Versions: 4.13.0
Reporter: Andrew Dinn
Assignee: Amos Feng
Fix For: 5.0.0.M2
The XTS codebase includes crash recovery tests in the sar/tests subtree which currently
need to be executed manually. This needs to be changed so that the current tests can be
run automatically via Hudson. After this has been completed it should then be possible to
extend the test suite to exercise scenarios not yet covered by the current range of
The tests themselves employ a web service client and several web services which are
deployed to the app server with the test code. The behaviour of the client and web
services can be scripted so this single deployment is capable of exercising all the
scenarios which are required in order to lead up to a crash. However, automation is not
straightforward for several reasons:
Execution of the tests requires starting up an app server twice, the first time so that
it can be crashed at a specific point during execution then a second time so that recovery
processing can be checked. The JBossTS codebase includes some utilities which can be used
to help manage startup and shutdown of the JBoss AS instance.
Crashing of the app server and tracing of execution during the first and second runs
requires the use of the Byteman agent and Byteman rules. Suitable Byteman rule scripts
exist for all the current tests, However, trace output is currently written to a file and
the output is verified by eyeball. Timing variations mean that this output does not always
have a fixed format. Also, validation requires checking that identifiers printed during
the first and second run match up. It would be worth investigating an alternative way of
collecting and validating this trace information e.g. using the dtest package contributed
to Byteman by Jonathan Halliday. dtest has been used to test similar scenarios in the
JBossTS JTA - XTS bridge code (the latter is in the txbridge source tree).
Timing variations also mean that execution of recovery code may needs to be manipulated
using Byteman rules in order to ensure that the circumstances specified in the test
scenario are actually met. This can involve introducing delays or dropping messages to
ensure that events are handled in a specific order. The existing Bbyteman scripts include
rules to achieve this where needed by the current tests. However, once again, this
complicates automation of the trace validation process since it requires some of the
traced operations to be discounted until an occurrence which has been engineered is
In the longer run the tests will need to be extended in two dimensions.
Firstly, the current test locates the client, web services and transaction coordinator in
one application server. The client, web services and transaction coordinator need to be
tested when they are deployed in different app servers in various possible combinations
and the correct handling of a crash by one or more of these app servers needs to be
Secondly, the current tests only test normal recovery situations. It will also be
necessary to simulate failures in the recovery process, either by scripting the web
service behaviour or by injecting faults using Byteman rules.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
For more information on JIRA, see: http://www.atlassian.com/software/jira