Dan Berindei created ISPN-11093: ----------------------------------- Summary: WriteSkewException possible with concurrent passivation Key: ISPN-11093 URL: https://issues.redhat.com/browse/ISPN-11093 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 10.1.0.CR1 Reporter: Dan Berindei Assignee: Will Burns Fix For: 10.1.0.Final {{VersionedRepeatableReadEntry.skewed()}} replaces the context entry's version with the version returned by {{CacheLoaderInterceptor.loadAndStoreInDataContainer()}}, apparently for the benefit of total order transactions, but running for all transactions, not just total order. Somehow {{loadAndStoreInDataContainer()}} can miss the entries in {{PassivationPersistenceManager}}'s temporary passivation map, causing random failures in {{WriteSkewGetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation[NON_OWNER, DIST_SYNC, TO=true, isolation=REPEATABLE_READ]}}. {noformat} 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [PassivationPersistenceManager] Storing entry temporarily during passivation for key GroupKey{group='test-group', key=5} 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [PersistenceManagerImpl] Writing to all stores for id 11566 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [VersionedRepeatableReadEntry] No entry for key GroupKey{group='test-group', key=5} found in data container 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [VersionedRepeatableReadEntry] Updating version in metadata SimpleClusteredVersion{topologyId=9, version=1} -> SimpleClusteredVersion{topologyId=0, version=0} 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [VersionedRepeatableReadEntry] Comparing versions SimpleClusteredVersion{topologyId=0, version=0} and SimpleClusteredVersion{topologyId=9, version=1} for key GroupKey{group='test-group', key=5}: BEFORE 12:48:34,193 TRACE (persistence-thread-Test-NodeA-p4387-t3:[]) [DummyInMemoryStore] Store MarshallableEntryImpl{keyBytes=ByteBufferImpl{length=67, offset=0, bytes=8201\/\o\r\g\.\i\n\f\i\n\i\s\p\a\n\.\t\e\s\t\.\c\o\r\e\.\G\r\o\u\p\T\e\s\t\s\S\C\I\.\G\r\o\u\p\K\e\y8A010E0A0A\t\e\s\t\-\g\r\o\u\p1005}, valueBytes=ByteBufferImpl{length=4, offset=0, bytes=\J02\v\5}, metadataBytes=ByteBufferImpl{length=12, offset=0, bytes=9801048A0106120408091001}, key=GroupKey{group='test-group', key=5}, value=v5, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=9, version=1}}, created=-1, lastUsed=-1, marshaller=org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl@64ddf323} in dummy map store@21df61dc 12:48:34,194 TRACE (async-thread-Test-NodeA-p4378-t6:[]) [CompletionStages] Continuing execution of id 11566 12:48:34,194 TRACE (async-thread-Test-NodeA-p4378-t6:[]) [PassivationPersistenceManager] Removed temporary entry during passivation for key GroupKey{group='test-group', key=5} 12:48:34,207 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [TotalOrderRemoteTransactionState] [Test-NodeB-45589:2742] Current status is [PREPARING], setting status to: PREPARED 12:48:34,207 DEBUG (async-thread-Test-NodeA-p4378-t4:[]) [TotalOrderInterceptor] Exception while preparing for transaction Test-NodeB-45589:2742. Local=false org.infinispan.transaction.WriteSkewException: Write skew detected on key GroupKey{group='test-group', key=5} for transaction RemoteTransaction{modifications=[..., RemoveCommand{key=GroupKey{group='test-group', key=5}, value=null, metadata=null, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, valueMatcher=MATCH_ALWAYS, topologyId=9}, ...], lookedUpEntries={..., GroupKey{group='test-group', key=5}=VersionedRepeatableReadEntry(66deadfb){key=GroupKey{group='test-group', key=5}, value=null, isCreated=false, isChanged=true, isRemoved=true, isExpired=false, skipLookup=true, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=0, version=0}}}, ...}, lockedKeys=[], backupKeyLocks=[], lookedUpEntriesTopology=9, isMarkedForRollback=false, tx=GlobalTx:Test-NodeB-45589:2742, state=TotalOrderRemoteTransactionState{transactionState=[PREPARING], globalTransaction='Test-NodeB-45589:2742'}} at org.infinispan.transaction.impl.WriteSkewHelper.lambda$performTotalOrderWriteSkewCheckAndReturnNewVersions$1(WriteSkewHelper.java:113) ~[classes/:?] 12:48:34,248 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.distribution.groups.WriteSkewGetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation[NON_OWNER, DIST_SYNC, TO=true, isolation=REPEATABLE_READ] org.infinispan.commons.CacheException: Could not commit implicit transaction at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1940) ~[classes/:?] at org.infinispan.cache.impl.CacheImpl.transactionalRemoveGroup(CacheImpl.java:655) ~[classes/:?] at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:638) ~[classes/:?] at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:631) ~[classes/:?] at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeGroup(AbstractDelegatingAdvancedCache.java:303) ~[classes/:?] at org.infinispan.distribution.groups.GetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation(GetGroupKeysTest.java:166) ~[test-classes/:?] {noformat} FWIW the trace log of the test doesn't include any line with the text "Retrieved entry for key %s from temporary passivation map", for any key. -- This message was sent by Atlassian Jira (v7.13.8#713008)