[jboss-jira] [JBoss JIRA] (WFLY-9896) Cluster-cluster EJB transactional invocation rolls back if stateful bean is used and JBOSS-LOCAL-USER auth is not possible

Cluster-cluster EJB transactional invocation rolls back if stateful bean is used and JBOSS-LOCAL-USER auth is not possible -------------------------------------------------------------------------------------------------------------------------- Key: WFLY-9896 URL: https://issues.jboss.org/browse/WFLY-9896 Project: WildFly Issue Type: Bug Components: Clustering, EJB, Transactions Affects Versions: 11.0.0.Final, 12.0.0.Beta1 Reporter: Michal Vinkler Assignee: David Lloyd Priority: Critical Labels: downstream_dependency Scenario: - server-1 (intermediary) -- Running standalone-ha profile (important) -- Has a remote-outbound-connection pointing at server-2 through server-2's public IP address -- Doesn't matter if the intermediary bean is stateless or stateful - server-2 (target) -- On a different machine than server-1 (important, otherwise transaction manager uses JBOSS-LOCAL-USER auth) -- Running standalone-ha profile (important) -- Target bean needs to be stateful (important) - Standalone EJB client invokes an intermediary bean server-1, which as a result invokes target bean on server-2. The first invocation is handled by EJB on target server, but when the transaction should be committed, The following is the log observed on server-1 (and also on the client): {code} 03:58:27,439 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: JBoss EAP 7.1.0.GA (WildFly Core 3.0.10.Final-redhat-1) started in 12893ms - Started 1013 of 1226 services (458 services are lazy, passive or on-demand) 03:59:00,563 INFO [org.wildfly.naming] (default task-7) WildFly Naming version 1.0.7.Final-redhat-1 03:59:00,585 INFO [org.jboss.ejb.client] (default task-7) JBoss EJB Client version 4.0.9.Final-redhat-1 03:59:01,335 INFO [stdout] (default task-7) Intermediate Stateless WhoAmIBean.whoAmI called as anonymous 03:59:01,809 WARN [com.arjuna.ats.arjuna] (default task-7) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0a105a36:-2ab20799:5a2f9a2b:1c, org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization@24aead2a >: org.wildfly.transaction.client.SynchronizationException: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:83) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:209) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:220) at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:265) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380) at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:535) at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:530) at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.beforeCompletion(RemotingRemoteTransactionPeer.java:166) at org.wildfly.transaction.client.SubordinateXAResource.beforeCompletion(SubordinateXAResource.java:135) at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:70) ... 56 more Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:368) at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494) at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208) at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720) at org.xnio.IoUtils$2.execute(IoUtils.java:71) at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693) at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132) at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570) at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494) at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92) ... 59 more 03:59:01,820 WARN [com.arjuna.ats.jta] (default task-7) ARJUNA016129: Could not end XA resource Subordinate XAResource at remote+http://10.16.90.56:8080: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.end(RemotingRemoteTransactionPeer.java:157) at org.wildfly.transaction.client.SubordinateXAResource.end(SubordinateXAResource.java:130) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.doEnd(TransactionImple.java:1089) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.endAssociation(TransactionImple.java:1060) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.endAssociation(XAResourceRecord.java:1287) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:313) at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023) at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002) at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1674) at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1615) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380) at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:535) at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:530) at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:368) at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494) at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208) at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720) at org.xnio.IoUtils$2.execute(IoUtils.java:71) at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693) at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132) at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570) at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494) at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92) ... 61 more 03:59:01,829 WARN [com.arjuna.ats.jta] (default task-7) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a105a36:-2ab20799:5a2f9a2b:f, node_name=1, branch_uid=0:ffff0a105a36:-2ab20799:5a2f9a2b:19, subordinatenodename=null, eis_name=unknown eis name > (Subordinate XAResource at remote+http://10.16.90.56:8080) failed with exception code XAException.XAER_RMFAIL: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.rollback(RemotingRemoteTransactionPeer.java:148) at org.wildfly.transaction.client.SubordinateXAResource.rollback(SubordinateXAResource.java:147) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:330) at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023) at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002) at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1674) at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1615) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:99) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380) at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:535) at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:530) at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:368) at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494) at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208) at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720) at org.xnio.IoUtils$2.execute(IoUtils.java:71) at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693) at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132) at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570) at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494) at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92) ... 58 more 03:59:01,833 WARN [com.arjuna.ats.arjuna] (default task-7) ARJUNA012091: Top-level abort of action 0:ffff0a105a36:-2ab20799:5a2f9a2b:f received TwoPhaseOutcome.FINISH_ERROR from com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord 03:59:01,838 ERROR [org.jboss.as.ejb3.invocation] (default task-7) WFLYEJB0034: EJB Invocation failed on component Intermediate for method public abstract java.lang.String example.ejb.WhoAmIBeanRemote.whoAmI(): javax.ejb.EJBTransactionRolledbackException: Transaction rolled back at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:138) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:118) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380) at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:535) at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:530) at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:193) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction. at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) ... 43 more Caused by: org.wildfly.transaction.client.SynchronizationException: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:83) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:209) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:220) at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:265) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289) ... 48 more Caused by: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.beforeCompletion(RemotingRemoteTransactionPeer.java:166) at org.wildfly.transaction.client.SubordinateXAResource.beforeCompletion(SubordinateXAResource.java:135) at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:70) ... 56 more Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:368) at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494) at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208) at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720) at org.xnio.IoUtils$2.execute(IoUtils.java:71) at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693) at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132) at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570) at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494) at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92) ... 59 more {code}