Transaction boundary commands can be completed before state transfer applies an active transaction -------------------------------------------------------------------------------------------------- Key: ISPN-3527 URL: https://issues.jboss.org/browse/ISPN-3527 Project: Infinispan Issue Type: Bug Components: State transfer Affects Versions: 5.2.7.Final Reporter: Erik Salter Assignee: Dan Berindei Fix For: 6.0.0.Beta2, 6.0.0.Final There is a condition where a transaction boundary command, like a RollbackCommand, can be processed by the new owner of a tx-encapsulated key before it applies the active transaction. On a view change: The forwarding node waits until its transactions from the new topology ID is set before issuing the command. 2013-09-17 14:28:33,725 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) Invoked with command RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=-1} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@1f225e51] 2013-09-17 14:28:33,727 TRACE [org.infinispan.statetransfer.StateTransferInterceptor] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) handleTopologyAffectedCommand for command RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=-1} 2013-09-17 14:28:33,727 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) Waiting for transaction data for topology 26, current topology is 25 ... 2013-09-17 14:28:34,101 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) east-dht5-60816(CMC-Denver-CO) broadcasting call RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=26} to recipient list [west-dht4-48045(CH2-Chicago-IL), east-dht5-60816(CMC-Denver-CO), east-dht2-60243(CMC-Denver-CO)] However, the receiving node has not finished applying the transactions from the forwarding node yet for this topology ID. So the tx cannot be found, resulting in a stale tx and an unusable key. 2013-09-17 14:28:34,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Calling perform() on RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote, cacheName='eigAllocation', topologyId=26} 2013-09-17 14:28:34,116 TRACE [org.infinispan.commands.tx.AbstractTransactionBoundaryCommand] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Did not find a RemoteTransaction for GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote 2013-09-17 14:28:34,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) About to send back response SuccessfulResponse{responseValue=null} for command RollbackCommand {gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote, cacheName='eigAllocation', topologyId=26} ... 2013-09-17 14:28:34,161 TRACE [org.infinispan.transaction.TransactionTable] (OOB-408,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Created and registered remote transaction RemoteTransaction{modifications=[PutKeyValueCommand{key=EdgeResourceCacheKey[edgeDeviceId=4109,resourceId=16825], value=EdgeInputResource [edgeInputId=16825, currentBandwidth=0, maxBandwidth=1048576000], flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true, ignorePreviousValue=false}], lookedUpEntries={}, lockedKeys=null, backupKeyLocks=null, missingLookedUpEntries=false, isMarkedForRollback=false, tx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote} 2013-09-17 14:28:34,733 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-408,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Signalling transaction data received for topology 26