[jbossts-issues] [JBoss JIRA] (JBTM-1433) Strange XA recovery behaviour - after sucessful commit is recovery called second

Tom Jenkinson (JIRA) jira-events at lists.jboss.org
Mon Feb 4 07:46:51 EST 2013


     [ https://issues.jboss.org/browse/JBTM-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom Jenkinson updated JBTM-1433:
--------------------------------

     Original Estimate: 1 week
    Remaining Estimate: 1 week

    
> Strange XA recovery behaviour - after sucessful commit is recovery called second
> --------------------------------------------------------------------------------
>
>                 Key: JBTM-1433
>                 URL: https://issues.jboss.org/browse/JBTM-1433
>             Project: JBoss Transaction Manager
>          Issue Type: Bug
>      Security Level: Public(Everyone can see) 
>    Affects Versions: 5.0.0.M2
>            Reporter: Ondřej Chaloupka
>            Assignee: Tom Jenkinson
>             Fix For: 4.17.4, 5.0.0.M2
>
>   Original Estimate: 1 week
>  Remaining Estimate: 1 week
>
> Hi,
> I've hit problem for functionality of XA recovery. This was found because of CI job running against master (http://hudson.qa.jboss.com/hudson/job/jbossts-crashrec-tests-postgresql91-master-ci/)
> After change of TM for the 5.0.0.M2 this started to occur.
> This problem is not just for postgresql but on all DBs tested by CI runs (oracle, db2.97, sybase).
> *What is tested and what's happening*
> Test halts server at commit phase (used byteman) before commit is done.
> The test enlists the resource of DB to transaction and then it enlists one XA resource created just for testing purposes for us having 2 participants in the transaction.
> After some debugging I think (but I'm not sure) that when server starts after halt the recovery runs fine. Commit is provided (or at least the DB record is updated sucessfully). But after that narayana tries to run some second round of recovery and in spite of the fact that the transaction was successfully committed (and it seems to be deleted from object store) it tries to do recovery on this transaction once more and the resource manager returns exception because it does not know that id already. 
> I think that the problem is starting with message:
> {code}
> 15:25:28,570 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016008: Local XARecoveryModule.xaRecovery - caught exception: java.lang.NullPointerException
> 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:598) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:413) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:193) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> {code}
> More from log:
> {code}
> 15:25:28,386 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction::doCommit (XAResourceRecord < resource:XAResourceWrapperImpl at 11455a2[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection at 212b9 pad=false overrideRmValue=null productName=PostgreSQL productVersion=9.1.4 jndiName=java:jboss/xa-datasources/VerificationDS], txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:34, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS >, heuristic: TwoPhaseOutcome.FINISH_OK, product: PostgreSQL/9.1.4, jndiName: java:jboss/xa-datasources/CrashRecoveryDS com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord at 13c2797 >)
> 15:25:28,386 TRACE [com.arjuna.ats.jta] (Periodic Recovery) XAResourceRecord.topLevelCommit for XAResourceRecord < resource:XAResourceWrapperImpl at 11455a2[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection at 212b9 pad=false overrideRmValue=null productName=PostgreSQL productVersion=9.1.4 jndiName=java:jboss/xa-datasources/VerificationDS], txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:34, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS >, heuristic: TwoPhaseOutcome.FINISH_OK, product: PostgreSQL/9.1.4, jndiName: java:jboss/xa-datasources/CrashRecoveryDS com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord at 13c2797 >
> 15:25:28,568 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction::updateState() for action-id 0:ffff7f000001:3d6777ab:50f804c8:2b
> 15:25:28,568 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.remove_committed(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction)
> 15:25:28,568 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.remove_state(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, StateType.OS_ORIGINAL)
> 15:25:28,568 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, StateType.OS_SHADOW)
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, 10)
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, StateType.OS_ORIGINAL)
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, 11)
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.currentState(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction) - returning StateStatus.OS_COMMITTED
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, StateType.OS_ORIGINAL)
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, 11)
> 15:25:28,569 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.openAndLock(/home/ochaloup/Transactions/eap-tests-transactions/integration/jbossts/target/jbossas-jbossts/standalone/data/tx-object-store/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/0_ffff7f000001_3d6777ab_50f804c8_2b, FileLock.F_WRLCK, false)
> 15:25:28,570 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.closeAndUnlock(/home/ochaloup/Transactions/eap-tests-transactions/integration/jbossts/target/jbossas-jbossts/standalone/data/tx-object-store/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/0_ffff7f000001_3d6777ab_50f804c8_2b, null, null)
> 15:25:28,570 DEBUG [com.arjuna.ats.arjuna] (Periodic Recovery) RecoverAtomicAction.replayPhase2( 0:ffff7f000001:3d6777ab:50f804c8:2b )  finished
> 15:25:28,570 DEBUG [com.arjuna.ats.arjuna] (Periodic Recovery)  
> 15:25:28,570 DEBUG [com.arjuna.ats.txoj] (Periodic Recovery) TORecoveryModule - second pass
> 15:25:28,570 DEBUG [com.arjuna.ats.arjuna] (Periodic Recovery)  
> 15:25:28,570 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Local XARecoveryModule - second pass
> 15:25:28,570 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Local XARecoveryModule.transactionInitiatedRecovery completed
> 15:25:28,570 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016008: Local XARecoveryModule.xaRecovery - caught exception: java.lang.NullPointerException
> 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:598) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:413) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:193) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [narayana-jts-jacorb-5.0.0.M2-SNAPSHOT.jar:5.0.0.M2-SNAPSHOT (revision: 1fa9d41e685713c7323222a0bb83e41ec1fd3d31)]
> 15:25:28,571 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Have 1 Xids to recover on this pass.
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) StateManager::StateManager( 2, 0 )
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction::BasicAction()
> 15:25:28,572 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Checking whether Xid < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:34, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS > exists in ObjectStore.
> 15:25:28,572 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Looking for 0:ffff7f000001:3d6777ab:50f804c8:2b and /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, StateType.OS_SHADOW)
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, 10)
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, StateType.OS_ORIGINAL)
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) FileSystemStore.genPathName(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction, 11)
> 15:25:28,572 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) ShadowingStore.currentState(0:ffff7f000001:3d6777ab:50f804c8:2b, /StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction) - returning StateStatus.OS_UNKNOWN
> 15:25:28,572 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) No record found for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:34, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS >
> 15:25:28,573 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) XAResourceOrphanFilter com.arjuna.ats.internal.jta.recovery.arjunacore.JTATransactionLogXAResourceOrphanFilter voted ABSTAIN
> 15:25:28,573 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) node name of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:34, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS > is 1
> 15:25:28,573 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) XAResourceOrphanFilter com.arjuna.ats.internal.jta.recovery.arjunacore.JTANodeNameXAResourceOrphanFilter voted ROLLBACK
> 15:25:28,573 INFO  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016013: Rolling back < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:34, subordinatenodename=null, eis_name=java:jboss/xa-datasources/CrashRecoveryDS >
> 15:25:28,746 ERROR [stderr] (Periodic Recovery) org.postgresql.xa.PGXAException: Error rolling back prepared transaction
> 15:25:28,746 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:416)
> 15:25:28,746 ERROR [stderr] (Periodic Recovery) 	at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.rollback(XAManagedConnection.java:342)
> 15:25:28,747 ERROR [stderr] (Periodic Recovery) 	at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:174)
> 15:25:28,747 ERROR [stderr] (Periodic Recovery) 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:734)
> 15:25:28,747 ERROR [stderr] (Periodic Recovery) 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:640)
> 15:25:28,747 ERROR [stderr] (Periodic Recovery) 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:413)
> 15:25:28,747 ERROR [stderr] (Periodic Recovery) 	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:193)
> 15:25:28,748 ERROR [stderr] (Periodic Recovery) 	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789)
> 15:25:28,748 ERROR [stderr] (Periodic Recovery) 	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)
> 15:25:28,748 ERROR [stderr] (Periodic Recovery) Caused by: org.postgresql.util.PSQLException: ERROR: prepared transaction with identifier "131077_AAAAAAAAAAAAAP//fwAAAT1nd6tQ+ATIAAAAKzE=_AAAAAAAAAAAAAP//fwAAAT1nd6tQ+ATIAAAANAAAAAEAAAAA" does not exist
> 15:25:28,748 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2101)
> 15:25:28,749 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1834)
> 15:25:28,749 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
> 15:25:28,749 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:510)
> 15:25:28,749 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:372)
> 15:25:28,749 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:300)
> 15:25:28,750 ERROR [stderr] (Periodic Recovery) 	at org.postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:406)
> 15:25:28,750 ERROR [stderr] (Periodic Recovery) 	... 8 more
> 15:25:28,750 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Have 0 Xids to recover on this pass.
> 15:25:28,750 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Have 1 Xids to recover on this pass.
> 15:25:28,750 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) StateManager::StateManager( 2, 0 )
> 15:25:28,750 TRACE [com.arjuna.ats.arjuna] (Periodic Recovery) BasicAction::BasicAction()
> 15:25:28,750 DEBUG [com.arjuna.ats.jta] (Periodic Recovery) Checking whether Xid < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff7f000001:3d6777ab:50f804c8:2b, node_name=1, branch_uid=0:ffff7f000001:3d6777ab:50f804c8:2e, subordinatenodename=null, eis_name=unknown eis name > exists in ObjectStore.
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira



More information about the jbossts-issues mailing list