[JBoss JIRA] (ISPN-3281) Deadlock in non-transactional caches during rebalance

Friday, 28 June 2013



    [
https://issues.jboss.org/browse/ISPN-3281?page=com.atlassian.jira.plugin....
] 

Dan Berindei commented on ISPN-3281:
------------------------------------

This issue caused a test failure in CI for
ReplicatedTwoNodesMapReduceTest.testinvokeMapReduceOnSubsetOfKeysWithCollatorAsync: 

{noformat}
java.util.concurrent.ExecutionException: org.infinispan.CacheException:
java.util.concurrent.ExecutionException: org.infinispan.CacheException:
org.infinispan.CacheException: Could not move intermediate keys/values for M/R task
7960fb02-f76d-44e9-8ac2-e576f24d4126
	at
org.infinispan.distexec.mapreduce.MapReduceTask$MapReduceTaskFuture.get(MapReduceTask.java:784)
	at
org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testinvokeMapReduceOnSubsetOfKeysWithCollatorAsync(BaseWordCountMapReduceTest.java:308)
        ...
Caused by: org.infinispan.CacheException: java.util.concurrent.ExecutionException:
org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate
keys/values for M/R task 7960fb02-f76d-44e9-8ac2-e576f24d4126
	at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:369)
	at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:656)
	at org.infinispan.distexec.mapreduce.MapReduceTask$3.call(MapReduceTask.java:674)
	at
org.infinispan.distexec.mapreduce.MapReduceTask$MapReduceTaskFuture.get(MapReduceTask.java:782)	...
22 more
Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException:
org.infinispan.CacheException: Could not move intermediate keys/values for M/R task
7960fb02-f76d-44e9-8ac2-e576f24d4126
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:867)
	at
org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:461)
	at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:363)	... 25
more
Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move
intermediate keys/values for M/R task 7960fb02-f76d-44e9-8ac2-e576f24d4126
	at
org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
	at
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocally(MapReduceTask.java:989)
	at
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$200(MapReduceTask.java:916)
	at
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:938)
	at
org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:934)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)	... 5 more
Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R
task 7960fb02-f76d-44e9-8ac2-e576f24d4126
	at
org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:296)
	at
org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)	...
12 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from
NodeB-44791, see cause for remote stack trace
	at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
	at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
	at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
	at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
	at
org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
	at
org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
	at
org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
	at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
	at
org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
	at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
	at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
	at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
	at org.infinispan.CacheImpl.put(CacheImpl.java:890)
	at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
	at org.infinispan.CacheImpl.put(CacheImpl.java:229)
	at
org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:291)	...
13 more
Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from
NodeA-19576, see cause for remote stack trace
	at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
	at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
	at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
	at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
	at
org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:132)
	at
org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
	at
org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
	at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
	at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
	at
org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at
org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
	at
org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
	at
org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
	at
org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
	at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
	at
org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)
	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)
	at
org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
	at
org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
	at
org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)	...
3 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after
[10 seconds] on key [in] for requestor [Thread[remote-thread-1,NodeA,5,main]]! Lock held
by [Thread[transport-thread-4,NodeA,5,main]]
	at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:214)
	at
org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:197)
	at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149)
	at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:145)
	at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:69)	...
31 more
{noformat}
                
...
 Deadlock in non-transactional caches during rebalance
 -----------------------------------------------------

                 Key: ISPN-3281
                 URL: https://issues.jboss.org/browse/ISPN-3281
             Project: Infinispan
          Issue Type: Bug
          Components: Locking and Concurrency, State transfer
    Affects Versions: 5.3.0.Final
            Reporter: Dan Berindei
            Assignee: Mircea Markus
             Fix For: 6.0.0.Final


 Say we have a cache with node A and node B joins. The cache topology id is 1,
primary_owner(k) = A in the current CH and primary_owner(k) = B in the pending CH.
 1. Node A starts a put(k, v) command during the rebalance. It thinks it's the primary
owner, so it acquires the lock locally and it forwards the command to B.
 2. B installs topology 2, primary_owner(k) = B in the current CH, and there is no pending
CH.
 3. B receives the put(k, v) command from A. It thinks it's the primary owner, so it
acquires the lock locally and it forwards the command to A.
 4. A receives the put(k, v) command from B. Again it thinks it's the primary owner
and tries to acquire the lock locally, but it times out because the lock is held by
another thread (from step 1).
 I think it may be enough to update the topology id in the put(k, v) command on node B,
before forwarding it back to A. That way, the command will block on node A until topology
2 is installed, and it won't try to lock the key again. 
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009