XA_RBROLLBACK returned upon a XA forget branch request ------------------------------------------------------ Key: WFLY-3985 URL: https://issues.jboss.org/browse/WFLY-3985 Project: WildFly Issue Type: Bug Components: EJB Reporter: Johnathon Lee The stacktraces [1] show ejb remoting looking up the transaction on a recovery pass and finding that the transaction has rolled back. Recovery tells ejb remoting (it is enlisting an XA resource) to forget about a branch. Ejb remoting on the remote node gets the forget request and tries to import the transaction and gets an exception because the txn has rolled back. Since the forget request is to tell remoting that it no longer needs to keep any state associated with the branch the expectation would be for remoting to catch the exception and handle it gracefully. Per the stacktrace, it looks like remoting should catch the XAException.XA_RBROLLBACK from the getImportedTransaction() call (do whatever they need to do internally) and then return one of the valid codes: XA_OK is probably the correct one as returning XA_RBROLLBACK would be out of spec. [1] WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:b1, node_name=1, branch_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:b2, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@1c3c1d18 > WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:b1, node_name=1, branch_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:b2, subordinatenodename=null, eis_name=unknown eis name > WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:8f, node_name=1, branch_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:90, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@4ff5d522 > WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:8f, node_name=1, branch_uid=0:ffff0a1ef4ed:4e73c967:526fc3c7:90, subordinatenodename=null, eis_name=unknown eis name > ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException INFO [stdout] (Periodic Recovery) ret:false for msg:javax.transaction.xa.XAException ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple.getImportedTransaction(TransactionImporterImple.java:178) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple.getImportedTransaction(TransactionImporterImple.java:178) ERROR [stderr] (Periodic Recovery) at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.getImportedTransaction(EJBRemoteTransactionsRepository.java:150) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.getImportedTransaction(EJBRemoteTransactionsRepository.java:150) ERROR [stderr] (Periodic Recovery) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionRollbackTask.manageTransaction(XidTransactionRollbackTask.java:53) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionRollbackTask.manageTransaction(XidTransactionRollbackTask.java:53) ERROR [stderr] (Periodic Recovery) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionManagementTask.run(XidTransactionManagementTask.java:68) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionManagementTask.run(XidTransactionManagementTask.java:68) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.FutureTask.run(FutureTask.java:262) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.FutureTask.run(FutureTask.java:262) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ERROR [stderr] (Periodic Recovery) at java.lang.Thread.run(Thread.java:744) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.lang.Thread.run(Thread.java:744) ERROR [stderr] (Periodic Recovery) at org.jboss.threads.JBossThread.run(JBossThread.java:122) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.threads.JBossThread.run(JBossThread.java:122) ERROR [stderr] (Periodic Recovery) at ...asynchronous invocation...(Unknown Source) INFO [stdout] (Periodic Recovery) ret:false for msg: at ...asynchronous invocation...(Unknown Source) ERROR [stderr] (Periodic Recovery) at org.jboss.ejb.client.remoting.InvocationExceptionResponseHandler$MethodInvocationExceptionResultProducer.getResult(InvocationExceptionResponseHandler.java:99) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.ejb.client.remoting.InvocationExceptionResponseHandler$MethodInvocationExceptionResultProducer.getResult(InvocationExceptionResponseHandler.java:99) ERROR [stderr] (Periodic Recovery) at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.sendRollback(RemotingConnectionEJBReceiver.java:360) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.sendRollback(RemotingConnectionEJBReceiver.java:360) ERROR [stderr] (Periodic Recovery) at org.jboss.ejb.client.RecoveryOnlyEJBXAResource.rollback(RecoveryOnlyEJBXAResource.java:103) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.ejb.client.RecoveryOnlyEJBXAResource.rollback(RecoveryOnlyEJBXAResource.java:103) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:741) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:741) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:647) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:647) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016006: Local XARecoveryModule.xaRecovery - forget threw exception: javax.transaction.xa.XAException at com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple.getImportedTransaction(TransactionImporterImple.java:178) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.getImportedTransaction(EJBRemoteTransactionsRepository.java:150) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionForgetTask.manageTransaction(XidTransactionForgetTask.java:50) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionManagementTask.run(XidTransactionManagementTask.java:68) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45] at org.jboss.threads.JBossThread.run(JBossThread.java:122) at ...asynchronous invocation...(Unknown Source) at org.jboss.ejb.client.remoting.InvocationExceptionResponseHandler$MethodInvocationExceptionResultProducer.getResult(InvocationExceptionResponseHandler.java:99) at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.sendForget(RemotingConnectionEJBReceiver.java:430) at org.jboss.ejb.client.RecoveryOnlyEJBXAResource.forget(RecoveryOnlyEJBXAResource.java:68) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:675) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] ERROR [stderr] (Periodic Recovery) javax.transaction.xa.XAException INFO [stdout] (Periodic Recovery) ret:false for msg:javax.transaction.xa.XAException ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple.getImportedTransaction(TransactionImporterImple.java:178) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple.getImportedTransaction(TransactionImporterImple.java:178) ERROR [stderr] (Periodic Recovery) at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.getImportedTransaction(EJBRemoteTransactionsRepository.java:150) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.getImportedTransaction(EJBRemoteTransactionsRepository.java:150) ERROR [stderr] (Periodic Recovery) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionRollbackTask.manageTransaction(XidTransactionRollbackTask.java:53) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionRollbackTask.manageTransaction(XidTransactionRollbackTask.java:53) ERROR [stderr] (Periodic Recovery) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionManagementTask.run(XidTransactionManagementTask.java:68) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionManagementTask.run(XidTransactionManagementTask.java:68) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.FutureTask.run(FutureTask.java:262) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.FutureTask.run(FutureTask.java:262) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ERROR [stderr] (Periodic Recovery) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ERROR [stderr] (Periodic Recovery) at java.lang.Thread.run(Thread.java:744) INFO [stdout] (Periodic Recovery) ret:false for msg: at java.lang.Thread.run(Thread.java:744) ERROR [stderr] (Periodic Recovery) at org.jboss.threads.JBossThread.run(JBossThread.java:122) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.threads.JBossThread.run(JBossThread.java:122) ERROR [stderr] (Periodic Recovery) at ...asynchronous invocation...(Unknown Source) INFO [stdout] (Periodic Recovery) ret:false for msg: at ...asynchronous invocation...(Unknown Source) ERROR [stderr] (Periodic Recovery) at org.jboss.ejb.client.remoting.InvocationExceptionResponseHandler$MethodInvocationExceptionResultProducer.getResult(InvocationExceptionResponseHandler.java:99) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.ejb.client.remoting.InvocationExceptionResponseHandler$MethodInvocationExceptionResultProducer.getResult(InvocationExceptionResponseHandler.java:99) ERROR [stderr] (Periodic Recovery) at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.sendRollback(RemotingConnectionEJBReceiver.java:360) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.sendRollback(RemotingConnectionEJBReceiver.java:360) ERROR [stderr] (Periodic Recovery) at org.jboss.ejb.client.RecoveryOnlyEJBXAResource.rollback(RecoveryOnlyEJBXAResource.java:103) INFO [stdout] (Periodic Recovery) ret:false for msg: at org.jboss.ejb.client.RecoveryOnlyEJBXAResource.rollback(RecoveryOnlyEJBXAResource.java:103) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:741) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:741) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:647) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:647) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) INFO [stdout] (Periodic Recovery) ret:false for msg: at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016006: Local XARecoveryModule.xaRecovery - forget threw exception: javax.transaction.xa.XAException at com.arjuna.ats.internal.jta.transaction.arjunacore.jca.TransactionImporterImple.getImportedTransaction(TransactionImporterImple.java:178) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.getImportedTransaction(EJBRemoteTransactionsRepository.java:150) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionForgetTask.manageTransaction(XidTransactionForgetTask.java:50) at org.jboss.as.ejb3.remote.protocol.versionone.XidTransactionManagementTask.run(XidTransactionManagementTask.java:68) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45] at org.jboss.threads.JBossThread.run(JBossThread.java:122) at ...asynchronous invocation...(Unknown Source) at org.jboss.ejb.client.remoting.InvocationExceptionResponseHandler$MethodInvocationExceptionResultProducer.getResult(InvocationExceptionResponseHandler.java:99) at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.sendForget(RemotingConnectionEJBReceiver.java:430) at org.jboss.ejb.client.RecoveryOnlyEJBXAResource.forget(RecoveryOnlyEJBXAResource.java:68) at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:675) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:419) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:194) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2] at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb-4.17.4.Final-redhat-2.jar:4.17.4.Final-redhat-2]