Hot Rod server non-hash aware topology updates can include non-members ---------------------------------------------------------------------- Key: ISPN-9759 URL: https://issues.jboss.org/browse/ISPN-9759 Project: Infinispan Issue Type: Bug Components: Test Suite - Server Affects Versions: 9.4.2.Final Reporter: Dan Berindei Priority: Critical Labels: testsuite_stability Fix For: 10.0.0.Beta1 Attachments: HotRod12ReplicationTest.log.gz When sending a non-hash aware topology update, the server includes all the servers that are present in the topology cache. This includes both servers that don't have the cache running yet (if the cache was started dynamically) and servers that are shutting down or already shutdown (because a node doesn't remove itself from the address cache before shutting down). When a node shut down, the remaining nodes eventually see the view change and remove the stopped server from the address cache, but likely after sending a topology update with the new topology id to the clients. In cases where a rebalance is not necessary (e.g. replicated caches, or a single node is alive), a corrected topology update is never sent to the client. This is causing random failures in {{HotRod10ReplicationTest.testReplicatedPutWithTopologyChanges}},{{HotRodReplicationTest.testReplicatedPutWithTopologyChanges}} and {{HotRod12ReplicationTest.testReplicatedPutWithTopologyChanges}} I saw it on a pull request build first, but I see it (and its subclasses) has been randomly failing in master as well. I am able to reliably reproduce the failure on my machine if I add a small delay in {{CrashedMemberDetectorListener.detectCrashedMember()}}. {noformat} 16:24:18,288 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.server.hotrod.HotRod12ReplicationTest.testReplicatedPutWithTopologyChanges java.lang.AssertionError: expected:<3> but was:<2> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.9.9.jar:?] at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) ~[testng-6.9.9.jar:?] at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) ~[testng-6.9.9.jar:?] at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245) ~[testng-6.9.9.jar:?] at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252) ~[testng-6.9.9.jar:?] at org.infinispan.server.hotrod.HotRodReplicationTest.testReplicatedPutWithTopologyChanges(HotRodReplicationTest.java:145) ~[test-classes/:?] {noformat} https://ci.infinispan.org/job/Infinispan/job/master/878/