[infinispan-issues] [JBoss JIRA] (ISPN-6997) PessimisticTxPartitionAndMergeDuringRuntimeTest.testOriginatorIsolatedPartition random failures

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289842#comment-13289842 ] 

Dan Berindei commented on ISPN-6997:
------------------------------------

There's yet another way for the test to fail: a remote tx command can create a new remote transaction even when the tx originator is in another partition. If the partitions merge immediately afterwards, neither the originator nor the remote node's TransactionTable will clean up the transaction.

{noformat}
11:55:08,098 TRACE (ForkThread-1,PessimisticTxPartitionAndMergeDuringRuntimeTest:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=MagicKey#k1{4/63E634F0/190 at NodeJ-23917}, value=final-value, flags=[], putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true} and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext at 4da814c3]
11:55:08,105 DEBUG (testng-PessimisticTxPartitionAndMergeDuringRuntimeTest:[]) [GMS] NodeJ-23917: installing view [NodeJ-23917|17] (3) [NodeJ-23917, NodeK-36209, NodeL-60381]
11:55:08,106 TRACE (timeout-thread-NodeJ-p75-t1:[]) [TransactionTable] Checking for transactions originated on leavers. Current cache members are [NodeJ-23917, NodeK-36209, NodeL-60381], remote transactions: 0
11:55:08,106 TRACE (timeout-thread-NodeJ-p75-t1:[]) [TransactionTable] No remote transactions pertain to originator(s) who have left the cluster.
11:55:08,114 TRACE (remote-thread-NodeK-p82-t5:[]) [InvocationContextInterceptor] Invoked with command LockControlCommand{cache=pes-cache, keys=[MagicKey#k1{4/63E634F0/190 at NodeJ-23917}], flags=[], unlock=false, gtx=GlobalTx:NodeI-50271:9} and InvocationContext [org.infinispan.context.impl.RemoteTxInvocationContext at 203d110e]
11:55:08,115 TRACE (OOB-3,NodeJ-23917:[]) [TransactionTable] Created and registered remote transaction RemoteTransaction{modifications=[], lookedUpEntries={}, lockedKeys=[], backupKeyLocks=[], lookedUpEntriesTopology=2147483647, isMarkedForRollback=false, tx=GlobalTx:NodeI-50271:9, state=null}
11:55:08,118 TRACE (remote-thread-NodeJ-p74-t6:[]) [InvocationContextInterceptor] Invoked with command LockControlCommand{cache=pes-cache, keys=[MagicKey#k1{4/63E634F0/190 at NodeJ-23917}], flags=[], unlock=false, gtx=GlobalTx:NodeI-50271:9} and InvocationContext [org.infinispan.context.impl.RemoteTxInvocationContext at 203d110e]
11:55:08,119 DEBUG (testng-PessimisticTxPartitionAndMergeDuringRuntimeTest:[]) [GMS] NodeJ-23917: installing view MergeView::[NodeI-50271|20] (4) [NodeI-50271, NodeJ-23917, NodeK-36209, NodeL-60381], 2 subgroups: [NodeI-50271|18] (1) [NodeI-50271], [NodeJ-23917|19] (3) [NodeJ-23917, NodeK-36209, NodeL-60381]
11:55:08,120 TRACE (remote-thread-NodeJ-p74-t6:[]) [TxInterceptor] Verifying transaction: originatorMissing=false, alreadyCompleted=false
11:55:08,126 TRACE (transport-thread-NodeJ-p76-t6:[]) [TransactionTable] Checking for transactions originated on leavers. Current cache members are [NodeI-50271, NodeJ-23917, NodeK-36209, NodeL-60381], remote transactions: 1
11:55:08,126 TRACE (transport-thread-NodeJ-p76-t6:[]) [TransactionTable] Checking transaction GlobalTx:NodeI-50271:9
11:55:08,126 TRACE (transport-thread-NodeJ-p76-t6:[]) [TransactionTable] No remote transactions pertain to originator(s) who have left the cluster.
12:13:13,030 ERROR (testng-PessimisticTxPartitionAndMergeDuringRuntimeTest:[]) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.PessimisticTxPartitionAndMergeDuringRuntimeTest.testOriginatorIsolatedPartition
java.lang.AssertionError: There are pending transactions!
	at org.infinispan.test.MultipleCacheManagersTest.assertNoTransactions(MultipleCacheManagersTest.java:794) ~[test-classes/:?]
	at org.infinispan.partitionhandling.BaseTxPartitionAndMergeTest.finalAsserts(BaseTxPartitionAndMergeTest.java:96) ~[test-classes/:?]
	at org.infinispan.partitionhandling.BasePessimisticTxPartitionAndMergeTest.doTest(BasePessimisticTxPartitionAndMergeTest.java:82) ~[test-classes/:?]
	at org.infinispan.partitionhandling.PessimisticTxPartitionAndMergeDuringRuntimeTest.testOriginatorIsolatedPartition(PessimisticTxPartitionAndMergeDuringRuntimeTest.java:33) ~[test-classes/:?]
{noformat}


> PessimisticTxPartitionAndMergeDuringRuntimeTest.testOriginatorIsolatedPartition random failures
> -----------------------------------------------------------------------------------------------
>
>                 Key: ISPN-6997
>                 URL: https://issues.jboss.org/browse/ISPN-6997
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core, Test Suite - Core
>    Affects Versions: 9.0.0.Alpha4
>            Reporter: Dan Berindei
>            Assignee: Dan Berindei
>              Labels: testsuite_stability
>             Fix For: 9.0.0.Beta1
>
>
> The test starts with a cluster of 4 nodes, and splits it in 2 partitions while a transaction is trying to lock a key. After the transaction fails, it checks that the transaction has been cleaned up properly.
> On one of the owners, {{transactionTable.cleanupLeaverTransactions}} is being called only before the split and after the merge, never with the list of members during the split. That means it never sees the transaction as an orphan, and doesn't remove it.
> {noformat}
> 15:16:18,893 TRACE (testng-PTPAMDRT:[]) [PTPAMDRT] Local tx=[], remote tx=[GlobalTx:PTPAMDRT-NodeI-3337:28616], for cache PTPAMDRT-NodeJ-27814 
> 15:16:18,893 ERROR (testng-PTPAMDRT:[]) [TestSuiteProgress] Test failed: org.infinispan.partitionhandling.PTPAMDRT.testOriginatorIsolatedPartition
> java.lang.AssertionError: There are pending transactions!
> 	at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.8.8.jar:?]
> 	at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) ~[testng-6.8.8.jar:?]
> 	at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:223) ~[test-classes/:?]
> 	at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:519) ~[test-classes/:?]
> 	at org.infinispan.test.MultipleCacheManagersTest.assertNoTransactions(MultipleCacheManagersTest.java:794) ~[test-classes/:?]
> 	at org.infinispan.partitionhandling.BaseTxPartitionAndMergeTest.finalAsserts(BaseTxPartitionAndMergeTest.java:96) ~[test-classes/:?]
> 	at org.infinispan.partitionhandling.BasePessimisticTxPartitionAndMergeTest.doTest(BasePessimisticTxPartitionAndMergeTest.java:82) ~[test-classes/:?]
> 	at org.infinispan.partitionhandling.tionAndMergeDuringRuntimeTest.testOriginatorIsolatedPartition(PessimisticTxPartitionAndMergeDuringRuntimeTest.java:33) ~[test-classes/:?]
> {noformat}
> {{OptimisticTxPartitionAndMergeDuringCommitTest.testPrimaryOwnerIsolatedPartition}} has similar random failures.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)