[infinispan-issues] [JBoss JIRA] (ISPN-5142) Cross site state transfer - retry mechanism not working correctly

[RH Bugzilla Integration (JIRA)]

     [ https://issues.jboss.org/browse/ISPN-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

RH Bugzilla Integration updated ISPN-5142:
------------------------------------------
        Bugzilla Update: Perform
    Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1181129


> Cross site state transfer - retry mechanism not working correctly
> -----------------------------------------------------------------
>
>                 Key: ISPN-5142
>                 URL: https://issues.jboss.org/browse/ISPN-5142
>             Project: Infinispan
>          Issue Type: Bug
>          Components: State Transfer
>    Affects Versions: 7.1.0.Beta1
>            Reporter: Matej Čimbora
>            Assignee: Pedro Ruivo
>
> 2 sites - main (2 nodes), backup (3 nodes)
> Scenario:
> 1. take consumer site offline
> 2. write data into producer site
> 3. invoke push(backup) operation
> 4. write data into consumer site (such that key set overlaps with the one used in step#2)
> Configuration:
> - sync backup, 2pc
> - Pessimistic TX, multiple operations in a single TX (e.g. 20)  
> Consumer site logs indicate occuring lock acquisition problems, leading to chunk not being applied. In that case, the chunk should be re-sent, however, according to logs, it does not seem to happen.
> pruivo's note: retry mechanism can end up in an infinite loop
> 11:25:53,549 WARN  [org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl] (remote-thread-3) ISPN000291: Unable to apply X-Site state chunk.
> org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [3 seconds] on key [key_0000000000002652] for requestor [GlobalTransaction:<localhost-17627>:25:local]! Lock held by [GlobalTransaction:<localhost-11416>:43:remote]
> 	at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)
> 	at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171)
> 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169)
> 	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.lockAndRegisterBackupLock(PessimisticLockingInterceptor.java:304)
> 	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:101)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> 	at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:344)
> 	at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:241)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> 	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:183)
> 	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:119)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> 	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> 	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
> 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> 	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
> 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> 	at org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl.applyStateInTransaction(XSiteStateConsumerImpl.java:111)
> 	at org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl.applyState(XSiteStateConsumerImpl.java:92)
> 	at org.infinispan.xsite.statetransfer.XSiteStatePushCommand.perform(XSiteStatePushCommand.java:50)
> 	at org.infinispan.remoting.LocalInvocation.call(LocalInvocation.java:43)
> 	at org.infinispan.xsite.BackupReceiverImpl.handleStateTransferState(BackupReceiverImpl.java:140)
> 	at org.infinispan.xsite.statetransfer.XSiteStatePushCommand.performInLocalSite(XSiteStatePushCommand.java:32)
> 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$3.run(CommandAwareRpcDispatcher.java:250)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)