[jboss-svn-commits] JBL Code SVN: r32122 - labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US.
jboss-svn-commits at lists.jboss.org
jboss-svn-commits at lists.jboss.org
Wed Mar 17 02:01:06 EDT 2010
Author: misty at redhat.com
Date: 2010-03-17 02:01:06 -0400 (Wed, 17 Mar 2010)
New Revision: 32122
Added:
labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/Failure_Recovery.xml
Modified:
labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/JTS_Programmers_Guide.xml
Log:
Added Failure and Recovery chapter
Added: labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/Failure_Recovery.xml
===================================================================
--- labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/Failure_Recovery.xml (rev 0)
+++ labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/Failure_Recovery.xml 2010-03-17 06:01:06 UTC (rev 32122)
@@ -0,0 +1,892 @@
+<?xml version='1.0' encoding='utf-8' ?>
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
+]>
+<chapter>
+ <title>Failure Recovery</title>
+ <para>
+ The failure recovery subsystem of <application>JBoss Transaction Service</application> will ensure that results of a transaction are applied consistently to all resources affected by the transaction, even if any of the application processes or the machine hosting them crash or lose network connectivity. In the case of machine (system) crash or network failure, the recovery will not take place until the system or network is restored, but the original application does not need to be restarted – recovery responsibility is delegated to the Recovery Manager process (see below). Recovery after failure requires that information about the transaction and the resources involved survive the failure and is accessible afterward: this information is held in the ActionStore, which is part of the ObjectStore. If the ObjectStore is destroyed or modified, recovery may not be possible.
+ </para>
+ <para>
+ Until the recovery procedures are complete, resources affected by a transaction that was in progress at the time of the failure may be inaccessible. For database resources, this may be reported as tables or rows held by “in-doubt transactions”. For TransactionalObjects for Java resources, an attempt to activate the Transactional Object (as when trying to get a lock) will fail.
+ </para>
+ <warning>
+ <para>
+ Because of limitations in the ORB which ships with the JDK 1.3, it is not possible to provide crash recovery. We therefore do not recommend using this ORB for mission critical applications.
+ </para>
+ </warning>
+
+ <section>
+ <title>Configuring the Failure Recovery Subsystem for your ORB</title>
+ <para>
+ The failure recovery subsystem of <application>JBoss Transaction Service</application> is an area where complete ORB-independence cannot be achieved. However, the basic configuration of <application>JBoss Transaction Service</application> for failure recovery is ORB-independent. The configuration applicable to applications using <application>JBoss Transaction Service</application> is achieved using the RecoveryManager-properties.xml file, and the orportability-properties.xml, which should contain, respectively, the entry:
+ <screen>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.recoveryActivator_1"
+ value="com.arjuna.ats.internal.jts.orbspecific.recovery.RecoveryEnablement"/>
+ </screen>
+ and the entry:
+ <screen>
+ <property
+ name="com.arjuna.orbportability.orb.PostInit2"
+ value="com.arjuna.ats.internal.jts.recovery.RecoveryInit"/>
+ </screen>
+ </para>
+ <para>
+ These entries cause the loading of instances of the named classes, which in turn load the ORB-specific classes needed and performs other initialisation. This enables failure recovery for transactions initiated by or involving applications using this property file. The default <filename>RecoveryManager-properties.xml</filename> file and the orportability-properties.xml with the distribution include these entries.
+ </para>
+ <note>
+ <para>
+ Failure recovery is <emphasis>not</emphasis> supported with the <application>JavaIDL ORB</application> that is part of JDK. Failure recovery is supported for <application>JacOrb</application> only.
+ </para>
+ </note>
+ <para>
+ If the <property>RecoveryEnablement</property> line in the property file is removed (or commented out), no recovery will be enabled.
+ </para>
+ </section>
+
+ <section>
+ <title>The Recovery Manager</title>
+ <para>
+ The failure recovery subsystem of <application>JBoss Transaction Service</application> requires that the stand-alone Recovery Manager process be running for each ObjectStore (typically one for each node on the network that is running <application>JBoss Transaction Service</application> applications). The RecoveryManager file is located in the arjuna.jar file within the package com.arjuna.ats.arjuna.recovery.RecoveryManager. To start the Recovery Manager issue the following command:<code>java com.arjuna.ats.arjuna.recovery.RecoveryManager</code>. If the <option>-test</option> flag is used with the Recovery Manager then it will display a “Ready” message when initialised, i.e.,
+ <screen>
+ java com.arjuna.ats.arjuna.recovery.RecoveryManager -test
+ </screen>
+ </para>
+
+ <important>
+ <para>
+ To allow successful registration of Resource objects, when the ORB is used is <application>JacOrb</application>, the Recovery Manager will be started.
+ </para>
+ </important>
+
+ <section>
+ <title>Configuring the Recovery Manager</title>
+
+ <para>
+ The RecoveryManager reads the properties defined in the <filename>jbossts-properties.xml</filename> file and then also reads the property file <filename>RecoveryManager-properties.xml</filename>, from the same directory as it found the arjuna properties file. An entry for a property in the RecoveryManager properties file will override an entry for the same property in the main TransactionService properties file. Most of the entries are specific to the Recovery Manager.
+ </para>
+ <para>
+ A default version of <filename>RecoveryManager-properties.xml</filename> is supplied with the distribution – this can be used without modification, except possibly the debug tracing fields (see below, Output). The rest of this section discusses the issues relevant in setting the properties to other values (in the order of their appearance in the default version of the file)
+ </para>
+
+ <!-- This section has been reworked -->
+ <section>
+ <title>Output</title>
+ <para>
+ The RecoveryManager provides output detailing the recovery activities that have taken place. RecoveryManager uses the logging and tracing mechanism provided by the Arjuna Common Logging Framework (CLF), which provides an abstract logging mechanism which can be used by existing logging APIs such as Jakarta log4j and the JDK 1.4 logging API.
+ </para>
+ <para>
+ The CLF parses and publishes messages from commonLogger objects. These objects pass messages to the Handler, which publishes them to a logging facility. Both commonLoggers and Handlers are able to filter messages based on the logging level. Log levels indicate the importance and urgency of the message. These are the logging levels, ordered from lowest to highest priority: DEBUG, INFO, WARN, ERROR and FATAL.
+ </para>
+ <para>
+ The CLF provides an extension to filter logging messages with a finer granularity. When a log message is provided to the commonLogger with the DEBUG level, additional conditions can be specified to determine whether the message is enabled.
+ </para>
+ <note>
+ <para>
+ These conditions are only applied if the DEBUG level is enabled and the log request performed by the application specifies debugging granularity.
+ </para>
+ </note>
+
+ <para>
+ When enabled, Debugging is filtered conditionally on three variables:
+ </para>
+ <variablelist>
+ <varlistentry>
+ <term>Debugging level</term>
+ <listitem><para>this is the facility which will generate the log request with the DEBUG Level is generated, for example, constructors or basic methods. </para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Visibility level</term>
+ <listitem><para>the visibility of the constructor, method, etc. that generates the debugging. </para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Facility code</term>
+ <listitem><para>the package or sub-module within which debugging is generated, e.g., the object store. </para></listitem>
+ </varlistentry>
+ </variablelist>
+ <para>
+ The CLF defines three interfaces for these variables. A particular product may implement its own classes, defining its own granularity. <application>JBoss Transaction Service</application> uses the default Debugging level and the default Visibility level provided by CLF, but it defines its own Facility Code. ArjunaJTS uses the default level assigned to its commonLoggers objects (DEBUG), as described below:
+ </para>
+ <section>
+ <title>Debugging Level</title>
+ <para>
+ <application>JBoss Transaction Service</application> uses the default values defined in the class <classname>com.hp.mw.common.util.logging.CommonDebugLevel</classname>.
+ </para>
+
+ <table>
+ <title>Debugging Levels</title>
+ <tgroup cols="4">
+ <thead>
+ <row>
+ <entry><para>Debugging Level</para></entry>
+ <entry><para>Value</para></entry>
+ <entry><para>Description</para></entry>
+ <entry><para>Additional Info</para></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><para>NO_DEBUGGING</para></entry>
+ <entry><para>0x0000</para></entry>
+ <entry><para>No diagnostics. </para></entry>
+ <entry><para>A commonLogger object assigned with this values discard all debug requests </para></entry>
+ </row>
+ <row>
+ <entry><para>FULL_DEBUGGING</para></entry>
+ <entry><para>0xffff</para></entry>
+ <entry><para>Full diagnostics.</para></entry>
+ <entry><para>A CommonLogger object assigned with this value allows all debug requests if the facility code and the visibility level match those allowed by the commonLogger.</para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>
+ Additional Debugging Values are:
+ </para>
+
+ <table>
+ <title>Additional Debugging Levels</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry><para>Debugging Level</para></entry>
+ <entry><para>Value</para></entry>
+ <entry><para>Description</para></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><para>CONSTRUCTORS</para></entry>
+ <entry><para>0x0001</para></entry>
+ <entry><para>Diagnostics from constructors. </para></entry>
+ </row>
+ <row>
+ <entry><para>DESTRUCTORS</para></entry>
+ <entry><para>0x0002</para></entry>
+ <entry><para>Diagnostics from finalizers. </para></entry>
+ </row>
+ <row>
+ <entry><para>CONSTRUCT_AND_DESTRUCT</para></entry>
+ <entry><para>CONSTRUCTORS|DESTRUCTORS</para></entry>
+ <entry><para>Diagnostics from constructors and finalizers. </para></entry>
+ </row>
+ <row>
+ <entry><para>FUNCTIONS</para></entry>
+ <entry><para>0x0010</para></entry>
+ <entry><para>Diagnostics from functions.</para></entry>
+ </row>
+ <row>
+ <entry><para>OPERATORS</para></entry>
+ <entry><para>0x0020</para></entry>
+ <entry><para>Diagnostics from operators, such as equals.</para></entry>
+ </row>
+ <row>
+ <entry><para>FUNCS_AND_OPS</para></entry>
+ <entry><para>FUNCTIONS|OPERATORS</para></entry>
+ <entry><para>Diagnostics from functions and operations.</para></entry>
+ </row>
+ <row>
+ <entry><para>ALL_NON_TRIVIAL</para></entry>
+ <entry><para>CONSTRUCT_AND_DESTRUCT| FUNCTIONS|OPERATORS</para></entry>
+ <entry><para>Diagnostics from all non-trivial operations.</para></entry>
+ </row>
+ <row>
+ <entry><para>TRIVIAL_FUNCS</para></entry>
+ <entry><para>0x0100</para></entry>
+ <entry><para>Diagnostics from trivial functions.</para></entry>
+ </row>
+ <row>
+ <entry><para>TRIVIAL_OPERATORS</para></entry>
+ <entry><para>0x0200</para></entry>
+ <entry><para>Diagnostics from trivial operations, and operators.</para></entry>
+ </row>
+ <row>
+ <entry><para>ALL_TRIVIAL</para></entry>
+ <entry><para>TRIVIAL_FUNCS|TRIVIAL_OPERATORS</para></entry>
+ <entry><para>Diagnostics from all trivial operations.</para></entry>
+ </row>
+ <row>
+ <entry><para>ERROR_MESSAGES</para></entry>
+ <entry><para>0x400</para></entry>
+ <entry><para>only output from debugging error/warning messages.</para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>
+ Visibiliy level – <application>JBoss Transaction Service</application> uses the default values defined in the class <classname>com.hp.mw.common.util.logging.CommonVisibilityLevel</classname>.
+ </para>
+
+ <table>
+ <title>Visibility Levels</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry><para>Visibility Level</para></entry>
+ <entry><para>Value</para></entry>
+ <entry><para>Description</para></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><para>VIS_NONE</para></entry>
+ <entry><para>0x0000</para></entry>
+ <entry><para>No Diagnostic</para></entry>
+ </row>
+ <row>
+ <entry><para>VIS_PRIVATE</para></entry>
+ <entry><para>0x0001</para></entry>
+ <entry><para>only from private methods. </para></entry>
+ </row>
+ <row>
+ <entry><para>VIS_PROTECTED</para></entry>
+ <entry><para>0x0002</para></entry>
+ <entry><para>only from protected methods.</para></entry>
+ </row>
+ <row>
+ <entry><para>VIS_PUBLIC</para></entry>
+ <entry><para>0x0004</para></entry>
+ <entry><para>only from public methods. </para></entry>
+ </row>
+ <row>
+ <entry><para>VIS_PACKAGE</para></entry>
+ <entry><para>0x0008</para></entry>
+ <entry><para>only from package methods.</para></entry>
+ </row>
+ <row>
+ <entry><para>VIS_ALL</para></entry>
+ <entry><para>0xffff</para></entry>
+ <entry><para>Full Diagnostic </para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+ <para>
+ Facility Code – <application>JBoss Transaction Service</application> uses the following values:
+ </para>
+
+ <table>
+ <title>Facility Codes</title>
+ <tgroup cols="3">
+ <thead>
+ <row>
+ <entry><para>Facility Code</para></entry>
+ <entry><para>Value</para></entry>
+ <entry><para>Description</para></entry>
+ </row>
+ </thead>
+ <tbody>
+ <row>
+ <entry><para>FAC_ATOMIC_ACTION</para></entry>
+ <entry><para>0x0000001</para></entry>
+ <entry><para>atomic action core module.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_BUFFER_MAN</para></entry>
+ <entry><para>0x00000004</para></entry>
+ <entry><para>state management (buffer) classes.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_ABSTRACT_REC</para></entry>
+ <entry><para>0x00000008</para></entry>
+ <entry><para>abstract records.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_OBJECT_STORE</para></entry>
+ <entry><para>0x00000010</para></entry>
+ <entry><para>object store implementations.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_STATE_MAN</para></entry>
+ <entry><para>0x00000020</para></entry>
+ <entry><para>state management and StateManager.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_SHMEM</para></entry>
+ <entry><para>0x00000040</para></entry>
+ <entry><para>shared memory implementation classes.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_GENERAL</para></entry>
+ <entry><para>0x00000080</para></entry>
+ <entry><para>general classes.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_CRASH_RECOVERY</para></entry>
+ <entry><para>0x00000800</para></entry>
+ <entry><para>detailed trace of crash recovery module and classes.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_THREADING</para></entry>
+ <entry><para>0x00002000</para></entry>
+ <entry><para>threading classes.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_JDBC</para></entry>
+ <entry><para>0x00008000</para></entry>
+ <entry><para>JDBC 1.0 and 2.0 support.</para></entry>
+ </row>
+ <row>
+ <entry><para>FAC_RECOVERY_NORMAL</para></entry>
+ <entry><para>0x00040000</para></entry>
+ <entry><para>normal output for crash recovery manager.</para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+ </table>
+
+ <para>
+ To ensure appropriate output, you must set some of the finer debug properties explicitly in the properties file, as described below, where default values are given.
+ </para>
+ <screen>
+ <properties>
+ <!-- CLF 2.0 properties -->
+ <property
+ name="com.arjuna.common.util.logging.DebugLevel"
+ value="0x00000000"/>
+ <property
+ name="com.arjuna.common.util.logging.FacilityLevel"
+ value="0xffffffff"/>
+ <property
+ name="com.arjuna.common.util.logging.VisibilityLevel"
+ value="0xffffffff"/>
+ <property
+ name="com.arjuna.common.util.logger"
+ value="log4j"/>
+ </properties>
+ </screen>
+
+ <para>
+ If particular debug messages are required, the finer debug properties can be set to the appropriate level described earlier. For instance, to enable all messages related to recovery after a crash or CRASH RECOVERY but only those raised within functions (see Facility level), and public (see Visibility level) the finer debugging could be set as followz:
+ </para>
+
+ <screen>
+ <properties>
+ <!-- CLF 2.0 properties -->
+ <property
+ name="com.arjuna.common.util.logging.DebugLevel"
+ value="0x0010"/>
+ <property
+ name="com.arjuna.common.util.logging.FacilityLevel"
+ value="0x00000800"/>
+ <property
+ name="com.arjuna.common.util.logging.VisibilityLevel"
+ value="0x0004"/>
+ <property
+ name="com.arjuna.common.util.logger"
+ value="log4j"/>
+ </properties>
+ </screen>
+
+ <para>
+ A user may have an interest in other messages, as well as Crash Recovery. For instance the following combination logs Crash Recovery messages and atomic action messages produced by the TxCore module.
+ </para>
+ <screen>
+ <properties>
+ …
+ <property
+ name="com.arjuna.common.util.logging.FacilityLevel"
+ value="0x00000801"/>
+ ….
+ </properties>
+ </screen>
+
+ <note>
+ <para>
+ Finer debug messages are enabled only if the logging level is set to a DEBUG value as defined by the underlying logging configuration (see below example for “log4j”)
+ </para>
+ </note>
+
+ <para>
+ The choice of the underlying logging infrastructure is defined by the property <property>com.arjuna.common.util.logger</property>, which is set by default to “log4j”. Possible values are described below and more details could be found in the CLF2.0 Programmer’s Guide.
+ </para>
+
+ <variablelist>
+ <varlistentry>
+ <term>log4j</term>
+ <listitem><para>Log4j logging (log4j classes must be available in the classpath); configuration through the <filename>log4j.properties</filename> file, which is picked up from the <systemitem>CLASSPATH</systemitem> or given through a System property: <property>log4j.configuration</property></para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>jdk14</term>
+ <listitem><para>JDK 1.4 logging API (only supported on JVMs of version 1.4 or higher). Configuration is done through a file <filename>logging.properties</filename> in the jre/lib directory.</para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>simple</term>
+ <listitem><para>Selects the simple JDK 1.1 compatible console-based logger provided by Jakarta Commons Logging</para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Csf</term>
+ <listitem><para>Selects CSF-based logging (CSF embeddor must be available)</para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>dotnet</term>
+ <listitem>
+ <para>Selects a .net logging implementation </para>
+ <para>Since a dotnet logger is not currently implemented, this is currently identical to simple. Simple is a purely JDK1.1 console-based log implementation.</para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>avalon</term>
+ <listitem><para>Uses the Avalon Logkit implementation</para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>noop</term>
+ <listitem><para>Disables all logging </para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>jakarta</term>
+ <listitem><para>Uses the default log system discovery mechanism of the Jakarta Commons Logging framework </para></listitem>
+ </varlistentry>
+ </variablelist>
+
+ <para>
+ The properties of the underlying log system are configured in a manner specific to that log system, e.g., a log4j.properties file in the case that log4j logging is used. The default configuration file for log4j used by <application>JBoss Transaction Service</application> is:
+ </para>
+ <screen>
+ # Default LOG4J Configuration
+ # Arjuna Technologies Ltd.
+ # $Id: log4j.properties,v 1.1 2003/09/28 11:38:24 rbegg Exp $
+ log4j.rootLogger=WARN, stdout
+ log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+ log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+ log4j.appender.stdout.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
+ </screen>
+
+ <para>
+ To set the logging level to DEBUG in order to log finer debug messages, the log4j.rootLogger should have a value as follows:
+ </para>
+
+ <screen>
+ log4j.rootLogger=DEBUG, ….
+ </screen>
+ <para>
+ Messages describing the start and the periodical behavior made by the RecoveryManager are set the INFO level. To see them, the logging level should be set to INFO, as described above for DEBUG. Setting the normal recovery messages to the INFO level allows the RecoveryManager to produce a moderate level of reporting. If there is no work to be done, it only reports the entry into each module for each periodic pass. To disable the INFO messages produced by the Recovery Manager, the logging level can be set to ERROR, which is one level higher than INFO.
+ </para>
+ <note>
+ <para>
+ The Logging level should be set at least to the WARN value to enable the report of warning messages.
+ </para>
+ </note>
+ </section>
+ </section>
+ </section>
+
+ <section>
+ <title>Periodic Recovery</title>
+ <para>
+ The RecoveryManager scans the ObjectStore and other locations of information, looking for transactions and resources that require, or may require recovery. The scans and recovery processing are performed by recovery modules, (instances of classes that implement the com.arjuna.ats.arjuna.recovery.RecoveryModule interface), each with responsibility for a particular category of transaction or resource. The set of recovery modules used are dynamically loaded, using properties found in the RecoveryManager property file.
+ </para>
+ <para>
+ The interface has two methods: <methodname>periodicWorkFirstPass</methodname> and <methodname>periodicWorkSecondPass</methodname>. At an interval (defined by property <property>PERIODIC_RECOVERY_PERIOD</property>), the RecoveryManager calls the first pass method on each property, then waits for a brief period (defined by <property>RECOVERY_BACKOFF_PERIOD</property>), then call the second pass of each module. Typically, in the first pass, the module scans the relevant part of the ObjectStore to find transactions or resources that are in-doubt (i.e. are part way through the commitment process). On the second pass, if any of the same items are still in-doubt, it is possible the original application process has crashed and the item is a candidate for recovery.
+ </para>
+ <para>
+ An attempt by the RecoveryManager to recover a transaction that is still progressing in the original process(es) is likely to break the consistency. Accordingly, the recovery modules use a mechanism (implemented in the <classname>com.arjuna.ats.internal.jts.recovery.contact</classname> package) to check whether the original process is still alive, and if the transaction is still in progress. The RecoveryManager only proceeds with recovery if the original process no longer exists or the transaction is completed. If a server process or machine crashes, but the transaction-initiating process survives, the transaction will complete, usually generating a warning. Recovery of such a transaction is the RecoveryManager’s responsibility.
+ </para>
+ <para>
+ It is important to set the interval periods appropriately. The total iteration time will be the sum of <varname>PERIODIC_RECOVERY_PERIOD</varname>, <varname>RECOVERY_BACKOFF_PERIOD</varname>, and the length of time it takes to scan the stores and to attempt recovery of any in-doubt transactions found, for all the recovery modules. The recovery attempt time may include connection timeouts while trying to communicate (via the ORB) with processes or machines that have crashed or are inaccessible (which is why there are mechanisms in the recovery system to avoid trying to recover the same transaction indefinitely). The total iteration time will affect how long a resource remain inaccessible after a failure. <varname>PERIODIC_RECOVERY_PERIOD</varname> should be set accordingly (default is 120 seconds). The <varname>RECOVERY_BACKOFF_PERIOD</varname> can be comparatively short (default is 10 seconds). Its purpose is mainly to reduce the number of transactions that are candidates!
for recovery and which thus require a “contact” call to the original process to see if they are still in progress.
+ </para>
+ <note>
+ <para>In <application>JBoss Transaction Service</application> 2.0, there was no contact mechanism, and the backoff period had to be long enough to avoid catching transactions in flight at all. From 2.1, there is no such risk.</para>
+ </note>
+ <para>
+ Several recovery modules (implementations of the <classname>com.arjuna.ats.arjuna.recovery.RecoveryModule interface</classname>) are supplied with <application>JBoss Transaction Service</application>, supporting various aspects of transaction recovery including JDBC recovery. It is possible for advanced users to create their own recovery modules and register them with the Recovery Manager, using properties set in <filename>RecoveryManager-properties.xml</filename> that begin with <wordasword>RecoveryExtension</wordasword>. These settings will be invoked on each pass of the periodic recovery in the sort-order of the property names. It is therefore possible to predict the processing order. ordering. Note that a failure in an application process might occur while a periodic recovery pass is in progress. The default RecoveryExtension settings are:
+ </para>
+ <screen>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.recoveryExtension1"
+ value="com.arjuna.ats.internal.arjuna.
+ recovery.AtomicActionRecoveryModule"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.recoveryExtension2"
+ value="com.arjuna.ats.internal.txoj.recovery.TORecoveryModule"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.recoveryExtension3"
+ value="com.arjuna.ats.internal.jts.recovery.transactions.
+ TopLevelTransactionRecoveryModule"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.recoveryExtension4"
+ value="com.arjuna.ats.internal.jts.recovery.transactions.
+ ServerTransactionRecoveryModule"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.recoveryExtension5”
+ value="com.arjuna.ats.internal.jta.recovery.arjunacore.
+ XARecoveryModule"/>
+ </screen>
+ </section>
+
+ <section>
+ <title>XA Resource Recovery</title>
+ <para>
+ Recovery of XA resources (databases etc.) accessed via JDBC is handled by the XARecoveryModule. This has two aspects: transaction-initiated and resource-initiated recovery. Transaction-initiated recovery is possible where the particular transaction branch had progressed far enough for a JTA_ResourceRecord to be written in the ObjectStore. The record contains the information needed to link the transaction, as known to the rest of <application>JBoss Transaction Service</application> to the database. Resource-initiated recovery is necessary for branches where a failure occurred after the database had made a persistent record of the transaction, but before the JTA_ResourceRecord was persisted. Resource-initiated recovery is also necessary for datasources for which it is not possible to hold information in the JTA_ResourceRecord that allows the recreation in the RecoveryManager of the XAConnection/XAResource that was used in the original application.
+ </para>
+ <para>
+ Transaction-initiated recovery is automatic. The XARecoveryModule finds the JTA_ResourceRecord that need recovery (using the two-pass mechanism described above), then uses the normal recovery mechanisms to find the status of the transaction it was involved in (i.e. it calls replay_completion on the RecoveryCoordinator for the transaction branch), (re)creates the appropriate XAResource and issues commit or rollback on it as appropriate. The XAResource creation will use the same information, database name, username, password etc., as the original application.
+ </para>
+ <para>
+ Resource-initiated recovery must be specifically configured by supplying the RecoveryManager with the appropriate information for it to query each database (XADataSources) that has been accessed by any <application>JBoss Transaction Service</application> application. The access to each XADataSource is handled by a class that implements the <classname>com.arjuna.ats.jta.recovery.XAConnectionRecovery</classname> interface. Instances are dynamically loaded, as controlled by properties with names beginning “XAConnectionRecovery”.
+ </para>
+ <para>
+ The XARecoveryModule will use the XAConnectionRecovery implementation to get an XAResource to the target datasource. On each invocation of <methodname>periodicWorkSecondPass</methodname>, the recovery module will issue an <methodname>XAResource.recover</methodname> request. This will (as described in the XA specification) return a list of the transaction identifiers (Xid’s) that are known to the datasource and are in an indeterminate (in-doubt) state. The list of in-doubt Xid’s received on successive passes (i.e. periodicWorkSecondPass-es) is compared. Any Xid appearing in both lists, and for which no JTA_ResourceRecord was found by the intervening transaction-initiated recovery, is assumed to belong to a transaction that was involved in a crash before any JTA_Resource_Record was written, and a rollback is issued for that transaction on the XAResource.
+ </para>
+ <para>
+ This double-scan mechanism is used because it is possible the Xid was obtained from the datasource just as the original application process was about to create the corresponding JTA_ResourceRecord. The interval between the scans should allow time for the record to be written unless the application crashes. If it does crash, rollback is the right answer.
+ </para>
+ <para>
+ An XAConnectionRecovery implementation class can be written to contain all the information needed to perform recovery to some datasource. Alternatively, a single class can handle multiple datasources (with some similar features, presumably). The constructor of the implementation class must have an empty parameter list (because it is loaded dynamically), but the interface includes an <methodname>initialise</methodname> method which passes in additional information as a string. The content of the string is taken from the property value that provides the class name: everything after the first semicolon is passed as the value of the string. <remark>Explain this better.</remark>The use made of this string is determined by the XAConnectionRecovery implementation class. In the example below, the value of the string is <code>2;OraRecoveryInfo</code>.
+ </para>
+ <screen>XAConnectionRecoveryEmpay=com.arjuna.ats.internal.jdbc.recovery.BasicXARecovery;2;OraRecoveryInfo</screen>
+
+ <para>
+ An XAConnectionRecovery implementation class, <classname>com.arjuna.ats.internal.jdbc.recovery.BasicXARecovery</classname> is provided to support resource-initiated recovery for any XADataSource (see <application>JBoss Transaction Service</application> JTA manual). For this class, the string received in <methodname>initialise</methodname> is expected to contain the number of connections to recover, as well as the name of the properties file containing the dynamic class name, the database username, the database password and the URL (Uniform Resource Locator) needed to access the database. The following example is for an Oracle 8.1.6 database accessed via the Sequelink 5.1 driver:
+ </para>
+ <screen>XAConnectionRecoveryEmpay=com.arjuna.ats.internal.jdbc.recovery.BasicXARecovery;2;OraRecoveryInfo</screen>
+
+ <para>
+ This implementation is only provided as an example, because it relies upon user names and passwords appearing in plain text properties files. Users may create their own implementations of <methodname>XAConnectionRecovery</methodname>. See the javadocs and the <classname>example com.arjuna.ats.internal.jdbc.recovery.BasicXARecovery</classname>.
+ </para>
+
+ <programlisting role="JAVA" language="Java">
+ /*
+ * Copyright (C) 2000, 2001,
+ *
+ * Hewlett-Packard,
+ * Arjuna Labs,
+ * Newcastle upon Tyne,
+ * Tyne and Wear,
+ * UK.
+ *
+ */
+ package com.arjuna.ats.internal.jdbc.recovery;
+
+ import com.arjuna.ats.jdbc.TransactionalDriver;
+ import com.arjuna.ats.jdbc.common.jdbcPropertyManager;
+ import com.arjuna.ats.jdbc.logging.jdbcLogger;
+
+ import com.arjuna.ats.internal.jdbc.*;
+ import com.arjuna.ats.jta.recovery.XAConnectionRecovery;
+ import com.arjuna.ats.arjuna.common.*;
+ import com.arjuna.common.util.logging.*;
+
+ import java.sql.*;
+ import javax.sql.*;
+ import javax.transaction.*;
+ import javax.transaction.xa.*;
+ import java.util.*;
+
+ import java.lang.NumberFormatException;
+
+ /**
+ * This class implements the XAConnectionRecovery interface for XAResources.
+ * The parameter supplied in setParameters can contain arbitrary information
+ * necessary to initialise the class once created. In this instance it contains
+ * the name of the property file in which the db connection information is
+ * specified, as well as the number of connections that this file contains
+ * information on (separated by ;).
+ *
+ * IMPORTANT: this is only an *example* of the sorts of things an
+ * XAConnectionRecovery implementor could do. This implementation uses
+ * a property file which is assumed to contain sufficient information to
+ * recreate connections used during the normal run of an application so that
+ * we can perform recovery on them. It is not recommended that information such
+ * as user name and password appear in such a raw text format as it opens up
+ * a potential security hole.
+ *
+ * The db parameters specified in the property file are assumed to be
+ * in the format:
+ *
+ * DB_x_DatabaseURL=
+ * DB_x_DatabaseUser=
+ * DB_x_DatabasePassword=
+ * DB_x_DatabaseDynamicClass=
+ *
+ * DB_JNDI_x_DatabaseURL=
+ * DB_JNDI_x_DatabaseUser=
+ * DB_JNDI_x_DatabasePassword=
+ *
+ * where x is the number of the connection information.
+ *
+ * @since JTS 2.1.
+ */
+
+ public class BasicXARecovery implements XAConnectionRecovery
+ {
+ /*
+ * Some XAConnectionRecovery implementations will do their startup work
+ * here, and then do little or nothing in setDetails. Since this one needs
+ * to know dynamic class name, the constructor does nothing.
+ */
+ public BasicXARecovery () throws SQLException
+ {
+ numberOfConnections = 1;
+ connectionIndex = 0;
+ props = null;
+ }
+
+ /**
+ * The recovery module will have chopped off this class name already.
+ * The parameter should specify a property file from which the url,
+ * user name, password, etc. can be read.
+ */
+
+ public boolean initialise (String parameter) throws SQLException
+ {
+ int breakPosition = parameter.indexOf(BREAKCHARACTER);
+ String fileName = parameter;
+
+ if (breakPosition != -1)
+ {
+ fileName = parameter.substring(0, breakPosition -1);
+
+ try
+ {
+ numberOfConnections = Integer.parseInt(parameter.substring(breakPosition +1));
+ }
+ catch (NumberFormatException e)
+ {
+ //Produce a Warning Message
+ return false;
+ }
+ }
+
+ PropertyManager.addPropertiesFile(fileName);
+
+ try
+ {
+ PropertyManager.loadProperties(true);
+
+ props = PropertyManager.getProperties();
+ }
+ catch (Exception e)
+ {
+ //Produce a Warning Message
+
+ return false;
+ }
+
+ return true;
+ }
+
+ public synchronized XAConnection getConnection () throws SQLException
+ {
+ JDBC2RecoveryConnection conn = null;
+
+ if (hasMoreConnections())
+ {
+ connectionIndex++;
+
+ conn = getStandardConnection();
+
+ if (conn == null)
+ conn = getJNDIConnection();
+
+ if (conn == null)
+ //Produce a Warning message
+ }
+
+ return conn;
+ }
+
+ public synchronized boolean hasMoreConnections ()
+ {
+ if (connectionIndex == numberOfConnections)
+ return false;
+ else
+ return true;
+ }
+
+ private final JDBC2RecoveryConnection getStandardConnection () throws SQLException
+ {
+ String number = new String(""+connectionIndex);
+ String url = new String(dbTag+number+urlTag);
+ String password = new String(dbTag+number+passwordTag);
+ String user = new String(dbTag+number+userTag);
+ String dynamicClass = new String(dbTag+number+dynamicClassTag);
+ Properties dbProperties = new Properties();
+ String theUser = props.getProperty(user);
+ String thePassword = props.getProperty(password);
+
+ if (theUser != null)
+ {
+ dbProperties.put(ArjunaJDBC2Driver.userName, theUser);
+ dbProperties.put(ArjunaJDBC2Driver.password, thePassword);
+
+ String dc = props.getProperty(dynamicClass);
+
+ if (dc != null)
+ dbProperties.put(ArjunaJDBC2Driver.dynamicClass, dc);
+
+ return new JDBC2RecoveryConnection(url, dbProperties);
+ }
+ else
+ return null;
+ }
+
+ private final JDBC2RecoveryConnection getJNDIConnection () throws SQLException
+ {
+ String number = new String(""+connectionIndex);
+ String url = new String(dbTag+jndiTag+number+urlTag);
+ String password = new String(dbTag+jndiTag+number+passwordTag);
+ String user = new String(dbTag+jndiTag+number+userTag);
+ Properties dbProperties = new Properties();
+ String theUser = props.getProperty(user);
+ String thePassword = props.getProperty(password);
+
+ if (theUser != null)
+ {
+ dbProperties.put(ArjunaJDBC2Driver.userName, theUser);
+ dbProperties.put(ArjunaJDBC2Driver.password, thePassword);
+ return new JDBC2RecoveryConnection(url, dbProperties);
+ }
+ else
+ return null;
+ }
+ private int numberOfConnections;
+ private int connectionIndex;
+ private Properties props;
+ private static final String dbTag = "DB_";
+ private static final String urlTag = "_DatabaseURL";
+ private static final String passwordTag = "_DatabasePassword";
+ private static final String userTag = "_DatabaseUser";
+ private static final String dynamicClassTag = "_DatabaseDynamicClass";
+ private static final String jndiTag = "JNDI_";
+
+ /*
+ * Example:
+ *
+ * DB2_DatabaseURL=jdbc\:arjuna\:sequelink\://qa02\:20001
+ * DB2_DatabaseUser=tester2
+ * DB2_DatabasePassword=tester
+ * DB2_DatabaseDynamicClass=
+ * com.arjuna.ats.internal.jdbc.drivers.sequelink_5_1
+ *
+ * DB_JNDI_DatabaseURL=jdbc\:arjuna\:jndi
+ * DB_JNDI_DatabaseUser=tester1
+ * DB_JNDI_DatabasePassword=tester
+ * DB_JNDI_DatabaseName=empay
+ * DB_JNDI_Host=qa02
+ * DB_JNDI_Port=20000
+ */
+
+ private static final char BREAKCHARACTER = ';'; // delimiter for parameters
+ }
+ </programlisting>
+
+ <warning>
+ <para>
+ Oracle usernames for Oracle 8.0 to 8.1.4: it is necessary for any database user that will use distributed transactions (e.g., <application>JBoss Transaction Service</application> and JDBC) to have select privilege on the SYS via DBA_PENDING_TRANSACTIONS. For 8.1.5 and higher, this is not (apparently) necessary for normal transaction access. However, this privilege is needed for the database user given when creating an XAConnection that provides an XAResource that is then used for XAResource.recover. (XAResource.commit, rollback etc. do not require the privilege). Accordingly, administrators may wish to create a special database username for the <application>JBoss Transaction Service</application> RecoveryManager, which has this privilege, which need not be granted to users in general. An implication of this is that access to the RecoveryManager_2_2.properties file needs to be appropriately controlled, if the password for the RecoveryManager user is contained in it.
+ </para>
+ </warning>
+
+ <note>
+ <para>
+ Note: Multiple recovery domains and resource-initiated recovery: XAResource.recover returns the list of all transactions that are in-doubt with in the datasource. If multiple recovery domains (see below) are used with a single datasource, resource-initiated recovery will “see” transactions from other domains. Since it will not have a JTA_ResourceRecord available, it will rollback the transaction in the database, if the Xid appears in successive recover calls. Administrators may wish to suppress resource-initiated recovery (by not supplying an XAConnectionRecovery property) in such cases, or confine it to one (current) recovery domain.
+ </para>
+ </note>
+ </section>
+
+ <section>
+ <title>Recovery Behavior</title>
+ <para>
+ A property, <varname>OTS_ISSUE_RECOVERY_ROLLBACK</varname>, is provided to control whether the RecoveryManager explicitly issues a rollback request when asked (by <methodname>replay_completion</methodname>) for the status of a transaction that is unknown. According to the presume-abort mechanism used by OTS and JTS, it can be inferred that the transaction has rolledback, and this response will be returned to the Resource (including a subordinate coordinator) in this case. The Resource can (and should) then apply that result to the underlying resources. However, it is also legitimate for the superior to issue a rollback: this is done if <varname>OTS_ISSUE_RECOVERY_ROLLBACK</varname> is set to <option>YES</option>.
+ </para>
+ <para>
+ A property of the OTS transaction identification mechanism is that transaction coordinator is able to hold a Resource reference that will never be usable. This can occur in two cases:
+ </para>
+
+ <itemizedlist>
+ <listitem><para>the process holding the Resource crashes before receiving the commit or rollback request from the coordinator</para></listitem>
+ <listitem><para>the Resource received and responded to the commit or rollback request, but the message was lost, or the coordinator process crashed.</para></listitem>
+ </itemizedlist>
+
+ <para>
+ In the first case, the RecoveryManager for the Resource ObjectStore will eventually reconstruct a new Resource with a different IOR (CORBA object reference) and issue a <methodname>replay_completion</methodname> request containing the new Resource IO. The RecoveryManager for the coordinator will swap this with the original, useless one, and issue <methodname>commit</methodname> to the new (reconstructed) Resource. Until the replay_completion is received, the RecoveryManager will try to send <methodname>commit</methodname> to the Resource reference it had – this will fail with a CORBA System Exception. Exactly which exception depends on the ORB and other details).
+ </para>
+
+ <para>
+ In the second case, the Resource will not be reconstructed. The RecoveryManager at the coordinator will never get through, and will receive System Exceptions forever.
+ </para>
+ <note>
+ <para>
+ It is important to realise that the RecoveryManager cannot distinguish these two cases by any protocol mechanism. There is a perceptible cost in repeatedly attempting to send the commit to an inaccessible Resource. In particular, the timeouts involved will extend the recovery iteration time, and thus potentially leave real resources inaccessible for longer.
+ </para>
+
+ <para>
+ To avoid this, the RecoveryManager will only attempt to send commit to a Resource a limited number of times. After that, it will consider the transaction “assumed complete”. It retains the information about the transaction by changing the object type in the ActionStore. If the Resource eventually does wake up and a replay_completion request is received, the RecoveryManager will activate the transaction and issue the commit request (to the new Resource IOR). The number of times the RecoveryManager will attempt to issue commit at its own initiative, as part of the periodic recovery, is controlled by the property <varname>COMMITTED_TRANSACTION_RETRY_LIMIT</varname> (default is 3 times).
+ </para>
+ </note>
+ </section>
+
+ <section>
+ <title>Expired Entry Removal</title>
+ <para>
+ The operation of the recovery subsystem will cause some entries to be made in the ObjectStore that will not be removed in normal progress. The RecoveryManager has a facility for scanning for and removing items that are very old. Scans and removals are performed by implementations of the <classname>com.arjuna.ats.arjuna.recovery.ExpiryScanner</classname>. Implementations of this interface are loaded by passing the class name as the value of a property whose name begins with “ExpiryScanner”. The RecoveryManager calls the <methodname>scan</methodname> method on each loaded ExpiryScanner implementation at an interval determined by the property <varname>com.arjuna.ats.arjuna.recovery.expiryScanInterval</varname>. This value is given in hours – default is 12. A value of <option>0</option> (zero) will suppress any expiry scanning. If the value supplied is positive, the first scan is performed when RecoveryManager starts; if the value is negative, the first scan is delayed u!
ntil after the first interval, using the absolute value.
+ </para>
+ <para>
+ There are two kinds of item that are scanned for expiry:
+ </para>
+
+ <variablelist>
+ <varlistentry>
+ <term>Contact items</term>
+ <listitem><para>One of these is created by every application process that uses <application>JBoss Transaction Service</application>. They contain the information that allows the RecoveryManager to determine if the process that initiated the transaction is still alive, and the status of the transaction. The expiry time for these is set by the property <varname>com.arjuna.ats.arjuna.recovery.transactionStatusManagerExpiryTime</varname>. The value is a number of hours, and thedefault is 12. 0 (Zero) means never to expire. This is the interval after which a process that cannot be contacted is considered to be dead. The value should be long enough to avoid accidentally removing valid entries due to short lived transient errors such as network disruptions.</para></listitem>
+ </varlistentry>
+ <varlistentry>
+ <term>Assumed complete transactions</term>
+ <listitem><para>See above for detailed explanation. The expiry time is counted from when a transaction assumed to be complete. However, a received replay_completion request will reset the clock. The risk of removing assumed complete transactions is that a prolonged communication outage will mean that a remote Resource cannot connect to the RecoveryManager for the parent transaction. If the assumed complete transaction entry is expired before the communications are recovered, the eventual replay_completion will not find any information, and the Resource will be rolled back, even though the transaction has committed. Consequently, the expiry time for transactions which are assumed to have completed should be set to a value that exceeds the length of any anticipated network outage. The parameter is <varname>ASSUMED_COMPLETE_EXPIRY_TIME</varname>, and is expressed in hours. The default value is 240, and 0 (zero) means never expire.</para></listitem>
+ </varlistentry>
+ </variablelist>
+
+ <para>
+ The relevant ExpiryScanner properties are:
+ </para>
+ <screen>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.
+ expiryScannerTransactionStatusManager"
+ value="com.arjuna.ats.internal.arjuna.recovery.
+ ExpiredTransactionStatusManagerScanner"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.
+ expiryScannerContact"
+ value="com.arjuna.ats.internal.jts.recovery.contact.
+ ExpiredContactScanner"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.
+ expiryScannerTopLevelTran"
+ value="com.arjuna.ats.internal.jts.recovery.transactions.
+ ExpiredTopLevelScanner"/>
+ <property
+ name="com.arjuna.ats.arjuna.recovery.
+ expiryScannerServerTran"
+ value="com.arjuna.ats.internal.jts.recovery.transactions.
+ ExpiredServerScanner"/>
+ </screen>
+
+ <para>
+ There are two ExpiryScannners for the assumed complete transactions because there are different types in the ActionStore.
+ </para>
+ </section>
+ </section>
+
+ <section>
+ <title>Recovery Domains</title>
+ <para>
+ A key part of the recovery subsystem is that the RecoveryManager hosts the OTS <classname>RecoveryCoordinator</classname> objects that handle recovery for transactions initiated in application processes. There are three paths by which information passes between the application process and the RecoveryManager:
+ </para>
+ <itemizedlist>
+ <listitem><para>RecoveryCoordinator object references (IORs) are created in the application process, containing information identifying the transaction in the object key, passed to the Resource objects and then the object key is received by RecoveryManager</para></listitem>
+ <listitem><para>the application process and the RecoveryManager access the same <filename>jbossts-properties.xml</filename> file and thus the same ObjectStore</para></listitem>
+ <listitem><para>direct CORBA invocations from the RecoveryCoordinator to the application process, using information in the contact items. These items are kept in the ObjectStore.</para></listitem>
+ <listitem><para>Multiple recovery domains may be useful in migration scenarios, where separate ObjectStores are useful. However, multiple RecoveryManagers can cause problems with XA datasources if resource-initiated recovery is active on any of them.</para></listitem>
+ </itemizedlist>
+ </section>
+
+ <section>
+ <title>Transaction statuses and replay_completion</title>
+ <para>
+ When a transaction successfully commits, and informs all registered Resources that it has committed, the transaction log is removed from the system. The log is no longer required, because all registered Resources have responded successfully to the prepare/commit invocation sequence. This has the side-effect that if a Resource calls replay_completion on the RecoveryCoordinator after the transaction it represents has committed, the status returned will be StatusRolledback. The transaction system does not keep a record of committed transactions, and as such it assumes that the absence of a transaction log means that the transaction must have rolled back. This corresponds to the presumed abort protocol used by the OTS).
+ </para>
+ </section>
+
+</chapter>
+
Modified: labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/JTS_Programmers_Guide.xml
===================================================================
--- labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/JTS_Programmers_Guide.xml 2010-03-16 19:27:10 UTC (rev 32121)
+++ labs/jbosstm/enterprise/tags/EAP_5_0_1/Transactions_JTS_Programmers_Guide/en-US/JTS_Programmers_Guide.xml 2010-03-17 06:01:06 UTC (rev 32122)
@@ -10,6 +10,7 @@
<xi:include href="Constructing_OTS_Application.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="Interfaces_for_Extending_OTS.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="JBossTS-Example.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
+ <xi:include href="Failure_Recovery.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<xi:include href="Revision_History.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
<index />
</book>
More information about the jboss-svn-commits
mailing list