ReplTotalOrderStateTransferFunctional1PcTest.testSTWithThirdWritingTxCache random failures ------------------------------------------------------------------------------------------ Key: ISPN-3299 URL: https://issues.jboss.org/browse/ISPN-3299 Project: Infinispan Issue Type: Bug Components: RPC, State transfer Affects Versions: 5.3.0.Final Reporter: Dan Berindei Assignee: Pedro Ruivo Fix For: 6.0.0.Final Attachments: rtostf1pt.log.gz ReplTotalOrderStateTransferFunctional1PcTest.testSTWithThirdWritingTxCache sometimes times out. It seems to deadlock because a VersionedPrepareCommand and the REBALANCE_START command are delivered in different order on two different nodes: {noformat} 01:27:31,740 TRACE (Incoming-2,ISPN,NodeL-3115:) [CommandAwareRpcDispatcher] Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=repl-to-1pc-nbst, type=REBALANCE_START, sender=NodeK-31445, joinInfo=null, topologyId=3, currentCH=ReplicatedConsistentHash{members=[NodeK-31445, NodeL-3115], numSegments=1, primaryOwners=[0]}, pendingCH=ReplicatedConsistentHash{members=[NodeK-31445, NodeL-3115, NodeM-36388], numSegments=1, primaryOwners=[0]}, throwable=null, viewId=2} [sender=NodeK-31445] 01:27:31,740 TRACE (Incoming-1,ISPN,NodeK-31445:) [CommandAwareRpcDispatcher] Attempting to execute command: VersionedPrepareCommand {modifications=[PutKeyValueCommand{key=test12, value=12, flags=null, putIfAbsent=false, metadata=EmbeddedMetadata{version=null}, successful=true}], onePhaseCommit=false, versionsSeen={test12=null}, gtx=GlobalTransaction:<NodeL-3115>:22544:local, cacheName='repl-to-1pc-nbst'} [sender=NodeL-3115] 01:27:31,740 TRACE (Incoming-3,ISPN,NodeL-3115:) [CommandAwareRpcDispatcher] Attempting to execute command: VersionedPrepareCommand {modifications=[PutKeyValueCommand{key=test12, value=12, flags=null, putIfAbsent=false, metadata=EmbeddedMetadata{version=null}, successful=true}], onePhaseCommit=false, versionsSeen={test12=null}, gtx=GlobalTransaction:<NodeL-3115>:22544:local, cacheName='repl-to-1pc-nbst'} [sender=NodeL-3115] 01:27:31,741 TRACE (Incoming-3,ISPN,NodeL-3115:) [TotalOrderManager] Transaction [NodeL-3115:22544] will wait for [TotalOrderLatchImpl{latch=java.util.concurrent.CountDownLatch@5868f0ca[Count = 1], name='StateTransfer-3'}] and locked [test12] 01:27:31,741 TRACE (Incoming-2,ISPN,NodeK-31445:) [CommandAwareRpcDispatcher] Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=repl-to-1pc-nbst, type=REBALANCE_START, sender=NodeK-31445, joinInfo=null, topologyId=3, currentCH=ReplicatedConsistentHash{members=[NodeK-31445, NodeL-3115], numSegments=1, primaryOwners=[0]}, pendingCH=ReplicatedConsistentHash{members=[NodeK-31445, NodeL-3115, NodeM-36388], numSegments=1, primaryOwners=[0]}, throwable=null, viewId=2} [sender=NodeK-31445] 01:27:31,741 TRACE (Incoming-2,ISPN,NodeK-31445:repl-to-1pc-nbst) [TotalOrderManager] State Transfer start. It will wait for [TotalOrderLatchImpl{latch=java.util.concurrent.CountDownLatch@51988b1e[Count = 1], name='NodeL-3115:22544'}] {noformat}