[infinispan-issues] [JBoss JIRA] (ISPN-2738) Joining node ignored by hotrod clients in REPL clustering mode

[Galder Zamarreño (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749845#comment-12749845 ] 

Galder Zamarreño commented on ISPN-2738:
----------------------------------------

The problem does indeed look related to ISPN-2632 and I think it's linked to removal of coordination between the address cache and the topology id update. The problem seems to be that the Hot Rod server sends a new topology id before the cache has been updated, so when a new added, it says: here's the new topology ID but the cache has not yet been updated. The client now has a new id but the members are the same. When the cache is eventually updated with the new node, the topology ID is not increased, so clients will never talk to it. Here's a snippet from node01.log that proofs what I say:

{code}12:43:03,137 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-119) Decoded header HotRodHeader{op=GetRequest, version=12, 
messageId=1974, cacheName=testCache, flag=0, clientIntelligence=3, topologyId=8}
...
12:43:03,229 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-107) Decoded header HotRodHeader{op=GetRequest, version=12, 
messageId=2626, cacheName=testCache, flag=0, clientIntelligence=3, topologyId=9}
...
12:43:03,753 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (OOB-197,null) Updating entry (key=node02/default removed=false valid=true 
changed=true created=true loaded=false value=172.18.1.3:11222]
...
node01.log:86873:12:43:03,780 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-119) Decoded header HotRodHeader{op=PutRequest, 
version=12, messageId=1992, cacheName=testCache, flag=6, clientIntelligence=3, topologyId=9}{code}

@Dan, this is precisely the reason why the interceptor in HotRodServer was created. To coordinate and make sure that the new topology ID is not sent before the cache has been updated. This is crucial is part of the code I added to deal with resilience testing in previous testing round.
                
> Joining node ignored by hotrod clients in REPL clustering mode
> --------------------------------------------------------------
>
>                 Key: ISPN-2738
>                 URL: https://issues.jboss.org/browse/ISPN-2738
>             Project: Infinispan
>          Issue Type: Bug
>    Affects Versions: 5.2.0.CR2
>            Reporter: Michal Linhard
>            Assignee: Galder Zamarreño
>             Fix For: 5.2.0.Final
>
>
> resilience 4-3-4 REPL mode for JDG 6.1.0.ER9 (infinispan 5.2.0.CR2):
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EDG6/view/EDG-REPORTS-RESILIENCE/job/edg-60-resilience-repl-4-3/31/artifact/report/stats-throughput.png
> after rejoin of killed node the load is not redistributed to all three nodes again

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira