Members can re-appear by itself in the consistent hash after leaving -------------------------------------------------------------------- Key: ISPN-3035 URL: https://issues.jboss.org/browse/ISPN-3035 Project: Infinispan Issue Type: Bug Components: State transfer Affects Versions: 5.2.5.Final, 5.3.0.Alpha1 Reporter: Dan Berindei Assignee: Dan Berindei Fix For: 5.3.0.Beta1 Attachments: dret.log Seen as an intermittent failure in DataRehashedEventTest: {noformat} 2013-04-23 14:07:45,459 DEBUG (testng-DataRehashedEventTest) [org.infinispan.manager.DefaultCacheManager] Stopping cache manager ISPN on NodeC-58711 2013-04-23 14:07:45,468 INFO (testng-DataRehashedEventTest) [org.infinispan.remoting.transport.jgroups.JGroupsTransport] ISPN000080: Disconnecting and closing JGroups Channel 2013-04-23 14:07:46,469 DEBUG (testng-DataRehashedEventTest) [org.jgroups.protocols.pbcast.GMS] NodeC-58711: sending LEAVE request to NodeA-28008 2013-04-23 14:07:46,489 DEBUG (Incoming-2,ISPN,NodeA-28008) [org.jgroups.protocols.pbcast.GMS] NodeA-28008: installing [NodeA-28008|4] [NodeA-28008, NodeB-46156, NodeC-58711] 2013-04-23 14:07:46,491 DEBUG (asyncTransportThread-0,NodeA) [org.infinispan.topology.ClusterTopologyManagerImpl] Starting cluster-wide rebalance for cache ___defaultcache, topology = CacheTopology{id=8, currentCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeA-28008, NodeB-46156]}, pendingCH=DefaultConsistentHash{numSegments=60, numOwners=2, members=[NodeA-28008, NodeB-46156, NodeC-58711]}} 2013-04-23 14:07:49,493 ERROR (testng-DataRehashedEventTest) [org.infinispan.test.fwk.UnitTestTestNGListener] Test testJoinAndLeave(org.infinispan.statetransfer.DataRehashedEventTest) failed. java.lang.AssertionError: expected [2] but found [6] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertEquals(Assert.java:123) at org.testng.Assert.assertEquals(Assert.java:370) at org.testng.Assert.assertEquals(Assert.java:380) at org.infinispan.statetransfer.DataRehashedEventTest.testJoinAndLeave(DataRehashedEventTest.java:114) {noformat} The initial cluster has 3 nodes: A, B, C. C is killed, but somehow remains in the ClusterCacheStatus on the coordinator. Then C re-appears in the JGroups view (possibly a JGroups issue). The problem in Infinispan is that the coordinator now sees C as a joiner, and it rebalances the cache to include C in the consistent hash again.