[infinispan-issues] [JBoss JIRA] Commented: (ISPN-78) Large object support

[Gabriel Ciuloaica (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597859#comment-12597859 ] 

Gabriel Ciuloaica commented on ISPN-78:
---------------------------------------

Hi Olaf,

Sure, the all idea is that when you use a RS algorithm you generates the chunks such that each chunk has some more data added to it. In order to reconstruct the original object you need just a subset of those chunks. 
There is an algorithm to determine how to split the object in chunks based on availability requirement.
You'll never need all of them. Let me try to find a good example. I remember seen something some time ago...



> Large object support
> --------------------
>
>                 Key: ISPN-78
>                 URL: https://issues.jboss.org/browse/ISPN-78
>             Project: Infinispan
>          Issue Type: Feature Request
>          Components: Core API
>            Reporter: Manik Surtani
>            Assignee: Olaf Bergner
>             Fix For: 5.1.0.BETA1, 5.1.0.Final
>
>
> if each VM is allocated a 2GB heap and you have a 100 nodes in a grid with 1 redundant copy for each key, you have a theoretical addressable heap of 100GB.  But you are limited by (half) the heap of a single VM per entry, since entries are stored whole.
> E.g., cache.put(k, my2GBObject) will fail since you need at least 2GB for the object + another 2GB for its serialized form.
> This gets worse when you try cache.put(k, my10GBObject).  This *should* be possible if we have a theoretical 100GB heap.
> Potential solutions here are to fragment large objects, and store each fragment under separate keys.  Another approach would be to directly stream objects to disk. etc.  Needs thought and design, possibly a separate API to prevent 'pollution" of the more simplistic API.  (JumboCache?)
> Re: fragmenting, issues to overcome:
> How many chunks to fragment into?  Max size of each key could be configured, but how do we determine the size of an Object?  VM instrumentation?  Or perhaps the JumboCache only stores byte[]'s?  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira