RehashWithL1Test.testPutWithRehashAndCacheClear random failures --------------------------------------------------------------- Key: ISPN-7801 URL: https://issues.jboss.org/browse/ISPN-7801 Project: Infinispan Issue Type: Bug Components: Test Suite - Core Affects Versions: 9.0.0.Final Reporter: Dan Berindei Assignee: Dan Berindei Labels: testsuite_stability Fix For: 9.1.0.Final The test kills the only owner of a key and checks that when a node starts owning an L1 entry, it doesn't send it to other nodes during state transfer. Then it adds a new node (owning the key) and checks that the key isn't transferred to the new node, and it's deleted from L1 on the old nodes. The problem is that it doesn't wait, it assumes all the nodes have already removed it by the time {{getCache()}} returns on the joiner. {noformat} 03:24:27,606 TRACE (jgroups-5,Test-NodeB-54331:[]) [L1WriteSynchronizer] Caching remotely retrieved entry for key k0 in L1 03:24:27,607 TRACE (jgroups-5,Test-NodeB-54331:[]) [DefaultDataContainer] Store MortalCacheEntry{key=k0, value=some data} in container 03:24:26,754 DEBUG (testng-Test:[]) [Test] Populating L1 on Test-NodeA-2588 03:24:27,514 DEBUG (testng-Test:[]) [Test] Populating L1 on Test-NodeB-54331 03:24:27,777 DEBUG (testng-Test:[]) [Test] Populating L1 on Test-NodeC-65326 03:24:27,777 DEBUG (testng-Test:[]) [Test] Killing node Test-NodeC-65326 03:24:27,781 TRACE (transport-thread-Test-NodeA-p51-t2:[Topology-___defaultcache]) [DefaultDataContainer] Removed MortalCacheEntry{key=k0, value=some data} from container *** The entry is not removed from NodeB at this point 03:24:27,936 DEBUG (testng-Test:[]) [Test] Checking values on Test-NodeA-2588 03:24:27,998 TRACE (jgroups-5,Test-NodeB-54331:[]) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=MortalCacheValue{value=some data, lifespan=600000, created=1493943867607}} for command ClusteredGetCommand{key=k0, flags=[]} 03:24:28,034 TRACE (jgroups-7,Test-NodeA-2588:[]) [L1WriteSynchronizer] Caching remotely retrieved entry for key k0 in L1 03:24:28,044 TRACE (jgroups-7,Test-NodeA-2588:[]) [DefaultDataContainer] Store MortalCacheEntry{key=k0, value=some data} in container 03:24:28,519 DEBUG (testng-Test:[]) [Test] Checking values on Test-NodeB-54331 03:24:28,595 DEBUG (testng-Test:[]) [Test] Starting a new joiner 03:24:30,261 TRACE (transport-thread-Test-NodeA-p51-t6:[Topology-___defaultcache]) [InvocationContextInterceptor] Invoked with command InvalidateCommand{keys=[k0, k1, k2, k3, k4, k5, k6, k7, k8, k9]} and InvocationContext [org.infinispan.context.impl.NonTxInvocationContext@54c5cc1d] 03:24:30,292 DEBUG (testng-Test:[]) [Test] Checking values on Test-NodeA-2588 03:24:30,355 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.distribution.rehash.RehashWithL1Test.testPutWithRehashAndCacheClear java.lang.AssertionError: wrong value for k0 at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.8.8.jar:?] at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) ~[testng-6.8.8.jar:?] at org.testng.AssertJUnit.assertNull(AssertJUnit.java:282) ~[testng-6.8.8.jar:?] at org.infinispan.distribution.rehash.RehashWithL1Test.testPutWithRehashAndCacheClear(RehashWithL1Test.java:78) ~[test-classes/:?] *** Too late 03:24:30,360 TRACE (transport-thread-Test-NodeA-p51-t6:[Topology-___defaultcache]) [DefaultDataContainer] Removed MortalCacheEntry{key=k0, value=some data} from container {noformat}