[infinispan-issues] [JBoss JIRA] (ISPN-5123) MultiNodeDistributedTest deadlock

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13177007#comment-13177007 ] 

Dan Berindei commented on ISPN-5123:
------------------------------------

The logs suggest that we're seeing a concurrency issue in JGroups. After {{NodeA}} is killed, the new coordinator ({{NodeB}}) broadcasts a {{CacheTopologyControlCommand(GET_STATUS)}} command to the other nodes ({{NodeC}} and {{NodeD}}). Both nodes send back a {{SuccessfulResponse}}, but {{NodeB}} only sees the response from {{NodeD}}, and proceeds to eliminate {{NodeC}} from all the cache topologies:

{noformat}
23:22:21,328 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t6) [JGroupsTransport] dests=null, command=CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=MultiNodeDistributedTest-NodeB-28202, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4}, mode=SYNCHRONOUS, timeout=240000
23:22:21,329 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t6) [MessageDispatcher] real_dests=[MultiNodeDistributedTest-NodeC-19310, MultiNodeDistributedTest-NodeD-7096]
23:22:21,351 TRACE (remote-thread-MultiNodeDistributedTest-NodeD-p26-t3) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=...}  for command CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4}
23:22:21,361 TRACE (remote-thread-MultiNodeDistributedTest-NodeC-p18-t6) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=...}  for command CacheTopologyControlCommand{cache=null, type=GET_STATUS, sender=null, joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=4}
23:22:21,418 TRACE (remote-thread-MultiNodeDistributedTest-NodeD-p26-t3) [UNICAST3] MultiNodeDistributedTest-NodeD-7096 --> DATA(MultiNodeDistributedTest-NodeB-28202: #24, conn_id=1)
23:22:21,418 TRACE (remote-thread-MultiNodeDistributedTest-NodeC-p18-t6) [UNICAST3] MultiNodeDistributedTest-NodeC-19310 --> DATA(MultiNodeDistributedTest-NodeB-28202: #68, conn_id=1)
23:22:21,427 TRACE (OOB-2,MultiNodeDistributedTest-NodeB-28202) [UNICAST3] MultiNodeDistributedTest-NodeB-28202: delivering MultiNodeDistributedTest-NodeD-7096#24
23:22:21,431 TRACE (OOB-1,MultiNodeDistributedTest-NodeB-28202) [UNICAST3] MultiNodeDistributedTest-NodeB-28202: delivering MultiNodeDistributedTest-NodeC-19310#68
23:22:21,455 TRACE (OOB-2,MultiNodeDistributedTest-NodeB-28202) [JGroupsTransport] Responses: [sender=MultiNodeDistributedTest-NodeC-19310, received=false, suspected=false]
[sender=MultiNodeDistributedTest-NodeD-7096, retval=SuccessfulResponse{responseValue=...} , received=true, suspected=false]

23:22:21,455 DEBUG (transport-thread-MultiNodeDistributedTest-NodeB-p8-t6) [ClusterTopologyManagerImpl] Got 2 status responses. members are [MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeC-19310, MultiNodeDistributedTest-NodeD-7096]
23:22:21,484 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterCacheStatus] Added joiner MultiNodeDistributedTest-NodeB-28202 to cache ___defaultcache: members = [MultiNodeDistributedTest-NodeB-28202], joiners = [MultiNodeDistributedTest-NodeB-28202]
23:22:21,484 TRACE (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterCacheStatus] Added joiner MultiNodeDistributedTest-NodeD-7096 to cache ___defaultcache: members = [MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeD-7096], joiners = [MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeD-7096]
23:22:21,484 DEBUG (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=10, rebalanceId=5, currentCH=DefaultConsistentHash{ns=256, owners = (3)[MultiNodeDistributedTest-NodeB-28202: 80+81, MultiNodeDistributedTest-NodeC-19310: 90+81, MultiNodeDistributedTest-NodeD-7096: 86+94]}, pendingCH=null, unionCH=null, actualMembers=[MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeC-19310, MultiNodeDistributedTest-NodeD-7096]}, availability mode = null
23:22:21,486 DEBUG (transport-thread-MultiNodeDistributedTest-NodeB-p8-t1) [ClusterTopologyManagerImpl] Updating cluster-wide current topology for cache ___defaultcache, topology = CacheTopology{id=11, rebalanceId=5, currentCH=DefaultConsistentHash{ns=256, owners = (2)[MultiNodeDistributedTest-NodeB-28202: 121+40, MultiNodeDistributedTest-NodeD-7096: 135+45]}, pendingCH=null, unionCH=null, actualMembers=[MultiNodeDistributedTest-NodeB-28202, MultiNodeDistributedTest-NodeD-7096]}, availability mode = null
{noformat}

> MultiNodeDistributedTest deadlock
> ---------------------------------
>
>                 Key: ISPN-5123
>                 URL: https://issues.jboss.org/browse/ISPN-5123
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Test Suite - Query
>    Affects Versions: 7.1.0.Alpha1
>            Reporter: Gustavo Fernandes
>            Assignee: Gustavo Fernandes
>         Attachments: infinispan-infinispan-query.log, stack.zip, trace.tar.gz
>
>
> I've been seeing this intermittent problem in my environment. Sometimes the query suite hangs for 30min (and then proceeds). See attached stack trace.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)