WF transaction client returns XAER_RMERR during abort after server with subordinate transaction fails ----------------------------------------------------------------------------------------------------- Key: WFLY-11937 URL: https://issues.jboss.org/browse/WFLY-11937 Project: WildFly Issue Type: Bug Components: Transactions Affects Versions: 10.1.0.Final, 14.0.0.Final, 15.0.0.Final, 16.0.0.Final Reporter: Ivan Straka Assignee: Ondrej Chaloupka Priority: Major *Scenario* # Server1 enlists dummy XA resource and make JPA call # Server1 calls a EJB on Server2 # Server2 enlists dummy XA resource and make JPA call # Server2 is halted at the beginning of dummy XA resource # Transaction is supposed to be rolled back We can see following lines in the log of Server1 {code:java} 2019-04-02 21:22:41,707 TRACE [com.arjuna.ats.jta] (default task-2) XAResourceRecord.topLevelAbort for XAResourceRecord < resource:Subordinate XAResource at http-remoting://127.0.0.1:8180, txid:< formatId=131077, gtrid_length=32, bqual_length=36, tx_uid=0:ffff7f000001:-70ba203e:5ca3d272:2f, node_name=TWkA, branch_uid=0:ffff7f000001:-70ba203e:5ca3d272:35, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@531d7d94 >, record id=0:ffff7f000001:-70ba203e:5ca3d272:36 2019-04-02 21:22:41,708 WARN [com.arjuna.ats.jta] (default task-2) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=32, bqual_length=36, tx_uid=0:ffff7f000001:-70ba203e:5ca3d272:2f, node_name=TWkA, branch_uid=0:ffff7f000001:-70ba203e:5ca3d272:35, subordinatenodename=null, eis_name=unknown eis name > (Subordinate XAResource at http-remoting://127.0.0.1:8180) failed with exception code XAException.XAER_RMERR: javax.transaction.xa.XAException: WFTXN0026: Failed to send protocol message to remote peer at org.wildfly.transaction.client.provider.remoting.TransactionClientChannel.rollback(TransactionClientChannel.java:98) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.rollback(RemotingRemoteTransactionPeer.java:149) at org.wildfly.transaction.client.SubordinateXAResource.rollback(SubordinateXAResource.java:174) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:362) at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023) at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002) at com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Abort(BasicAction.java:1981) at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1519) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1288) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:75) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:88) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:261) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:362) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:144) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.EjbExceptionTransformingInterceptorFactories$1.processInvocation(EjbExceptionTransformingInterceptorFactories.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:406) at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:565) at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:546) at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:197) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.lang.Thread.run(Thread.java:748) Caused by: org.jboss.remoting3.NotOpenException: Writes closed at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:106) at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:291) at org.jboss.remoting3.util.MessageTracker.openMessage(MessageTracker.java:64) at org.jboss.remoting3.util.InvocationTracker.allocateMessage(InvocationTracker.java:189) at org.jboss.remoting3.util.InvocationTracker.allocateMessage(InvocationTracker.java:205) at org.wildfly.transaction.client.provider.remoting.TransactionClientChannel.rollback(TransactionClientChannel.java:91) ... 58 more 2019-04-02 21:22:41,709 WARN [com.arjuna.ats.arjuna] (default task-2) ARJUNA012089: Top-level abort of action 0:ffff7f000001:-70ba203e:5ca3d272:2f received heuristic decision: TwoPhaseOutcome.HEURISTIC_HAZARD {code} *TLDR;* Server2 is halted and Server1 gets XAER_RMERR. During abort/rollback Server2 is still down and Server1 gets XAER_RMERR. Result of this is that the transaction is in heuristic state and not rolled back. Based of the XA spec (http://pubs.opengroup.org/onlinepubs/009680699/toc.pdf) the XAER_RMFAIL (or XA_RETRY) is the correct XAException error code. _XAER_RMERR_ a resource manager error occurred in the transaction branch _XAER_RMFAIL_ resource manager unavailable _XA_RETRY_ routine returned with no effect and may be reissued