[infinispan-issues] [JBoss JIRA] (ISPN-11093) WriteSkewException possible with concurrent passivation

[Dan Berindei (Jira)]

     [ https://issues.redhat.com/browse/ISPN-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dan Berindei updated ISPN-11093:
--------------------------------
    Attachment: ISPN-10310_non_blocking_stateconsumer_20191216-1244_WriteSkewGetGroupKeysTest-infinispan-core.log.gz


> WriteSkewException possible with concurrent passivation
> -------------------------------------------------------
>
>                 Key: ISPN-11093
>                 URL: https://issues.redhat.com/browse/ISPN-11093
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 10.1.0.CR1
>            Reporter: Dan Berindei
>            Assignee: Will Burns
>            Priority: Major
>              Labels: testsuite_stability
>             Fix For: 10.1.0.Final
>
>         Attachments: ISPN-10310_non_blocking_stateconsumer_20191216-1244_WriteSkewGetGroupKeysTest-infinispan-core.log.gz
>
>
> {{VersionedRepeatableReadEntry.skewed()}} replaces the context entry's version with the version returned by {{CacheLoaderInterceptor.loadAndStoreInDataContainer()}}, apparently for the benefit of total order transactions, but running for all transactions, not just total order. Somehow {{loadAndStoreInDataContainer()}} can miss the entries in {{PassivationPersistenceManager}}'s temporary passivation map, causing random failures in {{WriteSkewGetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation[NON_OWNER, DIST_SYNC, TO=true, isolation=REPEATABLE_READ]}}.
> {noformat}
> 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [PassivationPersistenceManager] Storing entry temporarily during passivation for key GroupKey{group='test-group', key=5}
> 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [PersistenceManagerImpl] Writing to all stores for id 11566
> 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [VersionedRepeatableReadEntry] No entry for key GroupKey{group='test-group', key=5} found in data container
> 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [VersionedRepeatableReadEntry] Updating version in metadata SimpleClusteredVersion{topologyId=9, version=1} -> SimpleClusteredVersion{topologyId=0, version=0}
> 12:48:34,192 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [VersionedRepeatableReadEntry] Comparing versions SimpleClusteredVersion{topologyId=0, version=0} and SimpleClusteredVersion{topologyId=9, version=1} for key GroupKey{group='test-group', key=5}: BEFORE
> 12:48:34,193 TRACE (persistence-thread-Test-NodeA-p4387-t3:[]) [DummyInMemoryStore] Store MarshallableEntryImpl{keyBytes=ByteBufferImpl{length=67, offset=0, bytes=8201\/\o\r\g\.\i\n\f\i\n\i\s\p\a\n\.\t\e\s\t\.\c\o\r\e\.\G\r\o\u\p\T\e\s\t\s\S\C\I\.\G\r\o\u\p\K\e\y8A010E0A0A\t\e\s\t\-\g\r\o\u\p1005}, valueBytes=ByteBufferImpl{length=4, offset=0, bytes=\J02\v\5}, metadataBytes=ByteBufferImpl{length=12, offset=0, bytes=9801048A0106120408091001}, key=GroupKey{group='test-group', key=5}, value=v5, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=9, version=1}}, created=-1, lastUsed=-1, marshaller=org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl at 64ddf323} in dummy map store at 21df61dc
> 12:48:34,194 TRACE (async-thread-Test-NodeA-p4378-t6:[]) [CompletionStages] Continuing execution of id 11566
> 12:48:34,194 TRACE (async-thread-Test-NodeA-p4378-t6:[]) [PassivationPersistenceManager] Removed temporary entry during passivation for key GroupKey{group='test-group', key=5}
> 12:48:34,207 TRACE (async-thread-Test-NodeA-p4378-t4:[]) [TotalOrderRemoteTransactionState] [Test-NodeB-45589:2742] Current status is [PREPARING], setting status to: PREPARED
> 12:48:34,207 DEBUG (async-thread-Test-NodeA-p4378-t4:[]) [TotalOrderInterceptor] Exception while preparing for transaction Test-NodeB-45589:2742. Local=false
> org.infinispan.transaction.WriteSkewException: Write skew detected on key GroupKey{group='test-group', key=5} for transaction RemoteTransaction{modifications=[..., RemoveCommand{key=GroupKey{group='test-group', key=5},
> value=null, metadata=null, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, valueMatcher=MATCH_ALWAYS, topologyId=9}, ...], lookedUpEntries={..., GroupKey{group='test-group', key=5}=VersionedRepeatableReadEntry(66deadfb){key=GroupKey{group='test-group', key=5}, value=null, isCreated=false, isChanged=true, isRemoved=true, isExpired=false, skipLookup=true, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=0, version=0}}}, ...}, lockedKeys=[], backupKeyLocks=[], lookedUpEntriesTopology=9, isMarkedForRollback=false, tx=GlobalTx:Test-NodeB-45589:2742, state=TotalOrderRemoteTransactionState{transactionState=[PREPARING], globalTransaction='Test-NodeB-45589:2742'}}
> 	at org.infinispan.transaction.impl.WriteSkewHelper.lambda$performTotalOrderWriteSkewCheckAndReturnNewVersions$1(WriteSkewHelper.java:113) ~[classes/:?]
> 12:48:34,248 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.distribution.groups.WriteSkewGetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation[NON_OWNER, DIST_SYNC, TO=true, isolation=REPEATABLE_READ]
> org.infinispan.commons.CacheException: Could not commit implicit transaction
> 	at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1940) ~[classes/:?]
> 	at org.infinispan.cache.impl.CacheImpl.transactionalRemoveGroup(CacheImpl.java:655) ~[classes/:?]
> 	at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:638) ~[classes/:?]
> 	at org.infinispan.cache.impl.CacheImpl.removeGroup(CacheImpl.java:631) ~[classes/:?]
> 	at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.removeGroup(AbstractDelegatingAdvancedCache.java:303) ~[classes/:?]
> 	at org.infinispan.distribution.groups.GetGroupKeysTest.testRemoveGroupKeysWithPersistenceAndPassivation(GetGroupKeysTest.java:166) ~[test-classes/:?]
> {noformat}
> FWIW the trace log of the test doesn't include any line with the text "Retrieved entry for key %s from temporary passivation map", for any key.



--
This message was sent by Atlassian Jira
(v7.13.8#713008)