[infinispan-issues] [JBoss JIRA] (ISPN-5142) Cross site state transfer - retry mechanism not working correctly

[Matej Čimbora (JIRA)]

Matej Čimbora created ISPN-5142:
-----------------------------------

             Summary: Cross site state transfer - retry mechanism not working correctly
                 Key: ISPN-5142
                 URL: https://issues.jboss.org/browse/ISPN-5142
             Project: Infinispan
          Issue Type: Bug
          Components: State Transfer
    Affects Versions: 7.1.0.Beta1
            Reporter: Matej Čimbora
            Assignee: Pedro Ruivo


2 sites - main (2 nodes), backup (3 nodes)

Scenario:
1. take consumer site offline
2. write data into producer site
3. invoke push(backup) operation
4. write data into consumer site (such that key set overlaps with the one used in step#2)

Configuration:
- sync backup, 2pc
- Pessimistic TX, multiple operations in a single TX (e.g. 20)  

Consumer site logs indicate occuring lock acquisition problems, leading to chunk not being applied. In that case, the chunk should be re-sent, however, according to logs, it does not seem to happen.

pruivo's note: retry mechanism can end up in an infinite loop

11:25:53,549 WARN  [org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl] (remote-thread-3) ISPN000291: Unable to apply X-Site state chunk.
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [3 seconds] on key [key_0000000000002652] for requestor [GlobalTransaction:<localhost-17627>:25:local]! Lock held by [GlobalTransaction:<localhost-11416>:43:remote]
	at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)
	at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169)
	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.lockAndRegisterBackupLock(PessimisticLockingInterceptor.java:304)
	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPutKeyValueCommand(PessimisticLockingInterceptor.java:101)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:344)
	at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:241)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:183)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:119)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:34)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
	at org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl.applyStateInTransaction(XSiteStateConsumerImpl.java:111)
	at org.infinispan.xsite.statetransfer.XSiteStateConsumerImpl.applyState(XSiteStateConsumerImpl.java:92)
	at org.infinispan.xsite.statetransfer.XSiteStatePushCommand.perform(XSiteStatePushCommand.java:50)
	at org.infinispan.remoting.LocalInvocation.call(LocalInvocation.java:43)
	at org.infinispan.xsite.BackupReceiverImpl.handleStateTransferState(BackupReceiverImpl.java:140)
	at org.infinispan.xsite.statetransfer.XSiteStatePushCommand.performInLocalSite(XSiteStatePushCommand.java:32)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$3.run(CommandAwareRpcDispatcher.java:250)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.11#6341)