EJB server to server fails during transaction commit (Elytron, Remoting) ------------------------------------------------------------------------ Key: WFLY-9028 URL: https://issues.jboss.org/browse/WFLY-9028 Project: WildFly Issue Type: Bug Components: EJB, Remoting, Security Reporter: Josef Cacek Assignee: David Lloyd Priority: Blocker Server to server EJB call between protected EJBs fails on transaction commit with following exception int the server1 log: {noformat} 13:38:08,423 ERROR [org.jboss.as.ejb3.invocation] (default task-7) WFLYEJB0034: EJB Invocation failed on component EntryBean for method public abstract java.lang.String[] org.wildfly.test.seccontext.shared.Entry.doubleWhoAmI(java.lang.String,java.lang.String,org.wildfly.test.seccontext.shared.ReAuthnType) throws java.lang.Exception: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:138) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:118) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:332) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:240) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:327) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.security.IdentityOutflowInterceptor.processInvocation(IdentityOutflowInterceptor.java:73) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.security.RolesAllowedInterceptor.processInvocation(RolesAllowedInterceptor.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.security.SecurityDomainInterceptor.processInvocation(SecurityDomainInterceptor.java:44) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:256) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:240) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:380) at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:479) at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:474) at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:182) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction. at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1301) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:92) ... 47 more Caused by: org.wildfly.transaction.client.SynchronizationException: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:83) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:206) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:217) at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.beforeCompletion(AbstractTransaction.java:262) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:368) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:91) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289) ... 52 more Caused by: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer$1.beforeCompletion(RemotingRemoteTransactionPeer.java:166) at org.wildfly.transaction.client.SubordinateXAResource.beforeCompletion(SubordinateXAResource.java:135) at org.wildfly.transaction.client.XAOutflowedResources$1.beforeCompletion(XAOutflowedResources.java:70) ... 60 more Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: (none) at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:375) at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:179) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:482) at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:471) at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208) at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720) at org.xnio.IoUtils$2.execute(IoUtils.java:71) at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693) at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132) at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570) at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:471) at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:410) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentity(RemotingRemoteTransactionPeer.java:87) at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:92) ... 63 more {noformat} *Scenario details:* Elytron is configured as security provider on servers and Elytron client API is used for authentication. Protected stateless beans are used. {noformat} EJB Client -> Entry bean on Server1 -> WhoAmI bean on Server2 {noformat} * Client uses {{AuthenticationConfiguration}} with valid credentials for Server1 and calls the Entry bean there (PLAIN SASL mechanism used) * Entry bean on Server1 uses valid credentials for Server2 and calls the WhoAmI bean on Server2 there (PLAIN SASL mechanism used) * WhoAmI bean on Server2 just returns the caller principal name This issue is related ot JBEAP-11822 - it's part of it.