[ https://issues.jboss.org/browse/ISPN-5123?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-5123: ------------------------------------ The logs suggest that we're seeing a concurrency issue in JGroups. After {{NodeA}} is killed, the new coordinator ({{NodeB}}) broadcasts a {{CacheTopologyControlCommand(GET_STATUS)}} command to the other nodes ({{NodeC}} and {{NodeD}}). Both nodes send back a {{SuccessfulResponse}}, but {{NodeB}} only sees the response from {{NodeD}}, and proceeds to eliminate {{NodeC}} from all the cache topologies: {noformat} 23:22:21,328 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t6) [JGroupsTransport] dests=null, command=CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=MultiNodeDistributedTest-NodeB-28202, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4}, mode=SYNCHRONOUS, timeout=240000 23:22:21,329 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t6) [MessageDispatcher] real_dests=[MultiNodeDistributedTest-NodeC-19310, MultiNodeDistributedTest-NodeD-7096] 23:22:21,351 TRACE (remote-thread-MultiNodeDistributedTest-NodeD-p26-t3) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=...} for command CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4} 23:22:21,361 TRACE (remote-thread-MultiNodeDistributedTest-NodeC-p18-t6) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=...} for command CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4} 23:22:21,418 TRACE (remote-thread-MultiNodeDistributedTest-NodeD-p26-t3) [UNICAST3] MultiNodeDistributedTest-NodeD-7096 --> DATA(MultiNodeDistributedTest-NodeB-28202: #24, conn_id=1) 23:22:21,418 TRACE (remote-thread-MultiNodeDistributedTest-NodeC-p18-t6) [UNICAST3] MultiNodeDistributedTest-NodeC-19310 --> DATA(MultiNodeDistributedTest-NodeB-28202: #68, conn_id=1) 23:22:21,427 TRACE (OOB-2,MultiNodeDistributedTest-NodeB-28202) [UNICAST3] MultiNodeDistributedTest-NodeB-28202: delivering MultiNodeDistributedTest-NodeD-7096#24 23:22:21,431 TRACE (OOB-1,MultiNodeDistributedTest-NodeB-28202) [UNICAST3] MultiNodeDistributedTest-NodeB-28202: delivering MultiNodeDistributedTest-NodeC-19310#68 23:22:21,455 TRACE (OOB-2,MultiNodeDistributedTest-NodeB-28202) [JGroupsTransport] Responses: [sender=MultiNodeDistributedTest-NodeC-19310, received=false, suspected=false] [sender=MultiNodeDistributedTest-NodeD-7096, retval=SuccessfulResponse{responseValue=...} , received=true, suspected=false] 23:22:21,455 DEBUG (transport-thread-MultiNodeDistributedTest-NodeB-p8-t6) [ClusterTopologyManagerImpl] Got 2 status responses. members are [MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeC-19310, MultiNodeDistributedTest-NodeD-7096] 23:22:21,484 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterCacheStatus] Added joiner MultiNodeDistributedTest-NodeB-28202 to cache ___defaultcache: members = [MultiNodeDistributedTest-NodeB-28202], joiners = [MultiNodeDistributedTest-NodeB-28202] 23:22:21,484 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterCacheStatus] Added joiner MultiNodeDistributedTest-NodeD-7096 to cache ___defaultcache: members = [MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeD-7096], joiners = [MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeD-7096] 23:22:21,484 DEBUG (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=10, rebalanceId=5, currentCH=DefaultConsistentHash{ns=256, owners = (3)[MultiNodeDistributedTest-NodeB-28202: 80+81, MultiNodeDistributedTest-NodeC-19310: 90+81, MultiNodeDistributedTest-NodeD-7096: 86+94]}, pendingCH=null, unionCH=null, actualMembers=[MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeC-19310, MultiNodeDistributedTest-NodeD-7096]}, availability mode = null 23:22:21,486 DEBUG (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=11, rebalanceId=5, currentCH=DefaultConsistentHash{ns=256, owners = (2)[MultiNodeDistributedTest-NodeB-28202: 121+40, MultiNodeDistributedTest-NodeD-7096: 135+45]}, pendingCH=null, unionCH=null, actualMembers=[MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeD-7096]}, availability mode = null {noformat} -- This message was sent by Atlassian JIRA (v6.4.11#64026)