OptimisticTxPartitionAndMergeDuringPrepareTest.testPrimaryOwnerIsolatedPartition fails randomly ----------------------------------------------------------------------------------------------- Key: ISPN-8227 URL: https://issues.jboss.org/browse/ISPN-8227 Project: Infinispan Issue Type: Bug Components: Core, Test Suite - Core Affects Versions: 9.1.0.Final Reporter: Radim Vansa Assignee: Radim Vansa Attachments: log.txt The issue seems to be that during several topology changes the originator sends 2 VersionedPrepareCommands, 1 VersionedCommitCommand and 1 TxCompletionNotificationCommand. These commands are resent after cluster merge and then executed in arbitrary order, and after the completion notification command removes tx information from {{TransactionTable}} another prepare command adds it again. Logs attached, don't get confused by the fact that {code} 14:54:29,056 TRACE [org.infinispan.transaction.impl.TransactionTable] (remote-thread-test-NodeC-p146-t4) Created and registered remote transaction RemoteTransaction{modifications=[PutKeyValueCommand{key=MagicKey#k1{8/537A1109/107@test-NodeB-18592}, value=final-value, flags=[], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}, PutKeyValueCommand{key=MagicKey#k2{9/0AD74A7B/101@test-NodeC-10992}, value=final-value, flags=[], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}], lookedUpEntries={}, lockedKeys=[], backupKeyLocks=[], lookedUpEntriesTopology=2147483647, isMarkedForRollback=false, tx=GlobalTx:test-NodeA-55522:15, state=null} {code} appears in the log before {code} 14:54:29,056 TRACE [org.infinispan.transaction.impl.TransactionTable] (remote-thread-test-NodeC-p146-t6) Removed remote transaction GlobalTx:test-NodeA-55522:15 ? RemoteTransaction{modifications=[PutKeyValueComm and{key=MagicKey#k1{8/537A1109/107@test-NodeB-18592}, value=final-value, flags=[], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{ lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}, PutKeyValueCommand{key=MagicKey#k2{9/0AD74A7B/101@test-NodeC-10992}, value=final-value, flags=[], commandInvocationId=CommandInvocation:lo cal:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}], lookedUpEntries={MagicKey#k2{9/0AD74A7B/101@test- NodeC-10992}=VersionedRepeatableReadEntry(51ecb641){key=MagicKey#k2{9/0AD74A7B/101@test-NodeC-10992}, value=final-value, isCreated=false, isChanged=true, isRemoved=false, isExpired=false, skipLookup=true, metada ta=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=13, version=1}}}, MagicKey#k1{8/537A1109/107@test-NodeB-18592}=VersionedRepeatableReadEntry(7a4c1b3f){key=MagicKey# k1{8/537A1109/107@test-NodeB-18592}, value=final-value, isCreated=false, isChanged=true, isRemoved=false, isExpired=false, skipLookup=true, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=Sim pleClusteredVersion{topologyId=13, version=1}}}}, lockedKeys=[MagicKey#k2{9/0AD74A7B/101@test-NodeC-10992}], backupKeyLocks=[MagicKey#k1{8/537A1109/107@test-NodeB-18592}], lookedUpEntriesTopology=14, isMarkedFor Rollback=false, tx=GlobalTx:test-NodeA-55522:15, state=null} {code} While we check {{TransactionTable.isTransactionCompleted()}} within {{remoteTransactions.compute}}, there is a race between calling {{TransactionTable.removeRemoteTransaction}} and {{TransactionTable.markTransactionCompleted}}. My suggestion is to mark transaction as complete from within the {{removeRemoteTransaction}} method, using {{remoteTransactions.compute}}.