[infinispan-issues] [JBoss JIRA] (ISPN-1408) Reduce Hot Rod topology information memory consumption

[Dan Berindei (Commented) (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12634397#comment-12634397 ] 

Dan Berindei commented on ISPN-1408:
------------------------------------

Once ISPN-658 is in, you won't be able to use the JGroups view id as a node can stop a cache but remain a JGroups cluster member.

Since the HotRod clients can retry on failure I think it would be safe to use CacheViewsManager.getCommittedView() to get the latest installed view. Once we get non-blocking state transfer in we'll probably have to switch to CacheViewsManager.getPendingView() (and make that method return the latest committed view when if there is no pending view).

                
> Reduce Hot Rod topology information memory consumption
> ------------------------------------------------------
>
>                 Key: ISPN-1408
>                 URL: https://issues.jboss.org/browse/ISPN-1408
>             Project: Infinispan
>          Issue Type: Enhancement
>          Components: Cache Server
>            Reporter: Galder Zamarreño
>            Assignee: Galder Zamarreño
>              Labels: hotrod
>             Fix For: 5.2.0.FINAL
>
>
> Hot Rod server storage of topology views should be trimmed. Currently it stores:
> ||key||value||
> |view|view_id,[sever_endpoint,cluster_addr [cache_name:hash_id]]
|
> This results in storing far too much information, and particularly when virtual nodes are enabled, this explodes in size. This affects both Hot Rod memory consumption and network activity since these ids and server endpoint information needs to be send back to client. For example, with 1000 virtual nodes, 8 nodes and 10 caches and taking in account each hash id is a 4 byte integer, that results in ~300kb of data.
> On top of that, this is stored in a single key which means that concurrent startups can be problematic.  
> There's a couple of things that can be done to reduce the size of this:
> # Get clients to calculate hash ids. This will require configured number of virtual nodes to be sent back to clients and also ISPN-1404 so that both the client and the server can hash node positions on the same thing.
> # Piggy back on JGroups' view id to avoid Hot Rod having to maintain its own view id.
> The end result is:
> ||key||value||
> |cluster|sever_endpoint|
> Which reduces the memory consumption and avoids concurrent startup issues.
> This would require a new version of the Hot Rod protocol, potentially a 1.1 version rather than 2.0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira