[jboss-jira] [JBoss JIRA] (WFLY-6498) EJBClient UserTransactions with multiple method invocations generate lock conflicts

Richard Achmatowicz (JIRA) issues at jboss.org
Tue Jul 12 12:34:00 EDT 2016 

    [ https://issues.jboss.org/browse/WFLY-6498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264371#comment-13264371 ] 

Richard Achmatowicz commented on WFLY-6498:
-------------------------------------------

The issue was fixed by following the strategy in the last paragraph of the previous comment. 

There is a lot of detail in the comments above, so i'll try to summarise the changes made in this issue:

When making an invocation on a SFSB, interceptors are used to get the SFSB instance from the ISPN cache before the invocation starts, and to release the instance (and at the same time commit the changes made) to the cache when the invocation completes. In the case of nested beans, this would mean having two transactions created and committed: one for the outer bean and one for the inner bean. A batching mechanism (based on Batcher and Batch) was created to allow multiple ISPN cache operations on the same thread to be coalesced (interposed) into the same ISPN transaction.  This means that all the work performed on that thread, even if multiple beans are accessed, will be represented in one transaction, committed when the original invocation commits. The mechanism is implemented by intercepting calls to get(), release() and discard() of the class DistributableCache, which is the interface the EJB code uses to access the ISPN cache, as well as associating a Batch with the current thread and the SFSB instance itself. This works fine when making individual remote invocations on CMT and BMT beans.

If we now want to wrap several invocations on SFSBs in an EJBCLient UserTransaction, we want all work on those beans to be interposed into a single ISPN transaction, which is committed when the UserTransaction commits. The existing Batching mechanism can't handle that as the various invocations happen on separate threads. So the Batching mechanism was extended to check for data stored in the TransactionSynchronizationRepository, which would be available across invocations. The data stored there was the single Batch associated with the UserTransactioin. get(), release() and discard() were then modified to retrieve the batch from there if present, and use that batch for interposing the various calls to get(), release() and discard(). Finally, the timing of when the Batch associated with the SFSB instance for the first invocation in the UserTransaction had to be adjusted due to the registered Synchronization of the UserTransaction delaying the final call to release().


> EJBClient UserTransactions with multiple method invocations generate lock conflicts 
> ------------------------------------------------------------------------------------
>
>                 Key: WFLY-6498
>                 URL: https://issues.jboss.org/browse/WFLY-6498
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 10.0.0.Final
>            Reporter: Richard Achmatowicz
>            Assignee: Paul Ferraro
>             Fix For: 11.0.0.Alpha1
>
>
> Using the EJBClient library, we should be able to do the following from a standalone EJBClient application:
> // create a UserTransaction associated with a clustered server node X
> // make an invocation on an EJB on X
> // make an invocation on an EJB on X
> // commit the UserTransaction
> However, doing so results in this exception which occurs during processing of the second invocation:
> {noformat}
> 11:16:22,580 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-57) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166] and requestor GlobalTransaction:<node-0>:120:local. Lock is held by GlobalTransaction:<node-0>:118:local
> 	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
> 	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:190)
> 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
> 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:113)
> 	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
> 	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77)
> 	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> 	at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> 	at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> 	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> 	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> 	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> 	at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:176)
> 	at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:153)
> 	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
> 	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> 	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> 	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> 	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> 	at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
> 	at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
> 	at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> 	at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:87)
> 	at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:49)
> 	at org.wildfly.clustering.ejb.infinispan.InfinispanBeanManager.findBean(InfinispanBeanManager.java:244)
> 	at org.jboss.as.ejb3.cache.distributable.DistributableCache.get(DistributableCache.java:124)
> 	at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:59)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)
> 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:333)
> 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> 	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> 	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> 	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> 	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> 	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:327)
> 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:200)
> 	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:262)
> 	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> 	at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> 	at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> {noformat}
> The exception does not arise if we perform only one invocation within the UserTransaction.
> This exception is repeatable.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)

More information about the jboss-jira mailing list