[jbossts-issues] [JBoss JIRA] (JBTM-817) Automate execution of XTS crash recovery tests

[Paul Robinson (Commented) (JIRA)]

    [ https://issues.jboss.org/browse/JBTM-817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635709#comment-12635709 ] 

Paul Robinson commented on JBTM-817:
------------------------------------

I'm in the process of validating the trace output from these tests and I think doing this automatically is not going to be a trivial process. I have some ideas around how the byteman scripts can be re-factored to simplify the validation process and to also improve scalability so that we can introduce more complex, multi-party tests.

Therefore I am removing the requirement to validate the trace output from this issue. Validation will come in a later release. Hopefully this will turn this issue into a much more straight forward task.

So to confirm, the requirements for this issue are to:

# Run the tests as part of the AS7 testsuite (as per Ivo's pattern: https://github.com/istudens/jboss-as/compare/xts_tests#diff-0)
# Automate the running of each crash recovery test without any Byteman rules. This provides some test coverage and ensures we have a valid system before the Byteman rules are applied.
# Automate each valid combination of Byteman script to TestCase. The Byteman scripts specify which tests are available.
# Gather the output from all test runs in separate files for "eyeball validation". Until this validation is automated, I (with the help of Andrew initially) will do this validation. The output files should be named "<bytman script name>:<test class name>". For example: "ATCrashDuringCommit.MultiParticipantPrepareAndCommitTest".

Amos: does this make sense? Is this in line with how you are currently implementing the feature?
                
> Automate execution of XTS crash recovery tests
> ----------------------------------------------
>
>                 Key: JBTM-817
>                 URL: https://issues.jboss.org/browse/JBTM-817
>             Project: JBoss Transaction Manager
>          Issue Type: Task
>      Security Level: Public(Everyone can see) 
>          Components: XTS
>    Affects Versions: 4.13.0
>            Reporter: Andrew Dinn
>            Assignee: Amos Feng
>             Fix For: 5.0.0.M2
>
>
> The XTS codebase includes crash recovery tests in the sar/tests subtree which currently need to be executed manually. This needs to be changed so that the current tests can be run automatically via  Hudson. After this has been completed it should then be possible to extend the test suite to exercise scenarios not yet covered by the current range  of tests.
> The tests themselves employ a  web service client and several web services which are deployed to the app server with the test code. The behaviour of the client and web services can be scripted so this single deployment is capable of exercising all the scenarios which are required in order to lead up to a crash. However, automation is not straightforward for several reasons:
> Execution of the tests requires starting up an app server twice, the first time so that it can be crashed at a specific point during execution then a second time so that recovery processing can be checked. The JBossTS codebase includes some utilities which can be used to help manage startup and shutdown of the JBoss AS instance.
> Crashing of the app server and tracing of execution during the first and second runs requires the use of the Byteman agent and Byteman rules. Suitable Byteman rule scripts exist for all the current tests, However, trace output is currently written to a file and the output is verified by eyeball. Timing variations mean that this output does not always have a fixed format. Also, validation requires checking that identifiers printed during the first and second run match up. It would be worth investigating an alternative way of collecting and validating this trace information e.g. using the dtest package contributed to Byteman by Jonathan Halliday. dtest has been used to test similar scenarios in the JBossTS JTA - XTS bridge code (the latter is in the  txbridge source tree).
> Timing variations also mean that execution of recovery code may needs to be manipulated using Byteman rules in order to ensure that the circumstances specified in the test scenario are actually met. This can involve introducing delays or dropping messages to ensure that events are handled in a specific order. The existing Bbyteman scripts include rules to achieve this where needed by the current tests. However, once again, this complicates automation of the trace validation process since it requires some of  the traced operations to be discounted until an occurrence which has been engineered is identified.
> In the longer run the tests will need  to be extended in two dimensions.
> Firstly, the current test locates the client, web services and transaction coordinator in one application server. The client, web services and transaction coordinator need to be tested when they are deployed in different app servers in various possible combinations and the correct handling of a crash by one or more of these app servers needs to be validated.
> Secondly, the current tests only test normal recovery situations. It will also be necessary to simulate failures in the recovery process, either by scripting  the web service behaviour or by injecting faults using Byteman rules.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira