[infinispan-issues] [JBoss JIRA] Commented: (ISPN-78) Large object support
[
https://issues.jboss.org/browse/ISPN-78?page=com.atlassian.jira.plugin.sy...
]
Thomas Fuller commented on ISPN-78:
-----------------------------------
Regarding "[H]ow do we determine the size of an Object? VM instrumentation?"
This is a good question because I've been thinking about exactly how to do this, but
for another purpose -- basically determining what the memory mapping of the grid looks
like as a whole, so I'd be interested in getting involved in this discussion.
Regarding "Potential solutions here are to fragment large objects, and store each
fragment under separate keys."
IMO this sounds messy and I can see this resulting in some strange bugs.
"Another approach would be to directly stream objects to disk. etc. Needs thought and
design, possibly a separate API to prevent 'pollution" of the more simplistic
API. (JumboCache?)"
IMO this seems like a much cleaner approach.
Tom
Large object support
--------------------
Key: ISPN-78
URL: https://issues.jboss.org/browse/ISPN-78
Project: Infinispan
Issue Type: Feature Request
Components: Core API
Reporter: Manik Surtani
Assignee: Manik Surtani
Fix For: 5.1.0.BETA1, 5.1.0.Final
if each VM is allocated a 2GB heap and you have a 100 nodes in a grid with 1 redundant
copy for each key, you have a theoretical addressable heap of 100GB. But you are limited
by (half) the heap of a single VM per entry, since entries are stored whole.
E.g., cache.put(k, my2GBObject) will fail since you need at least 2GB for the object +
another 2GB for its serialized form.
This gets worse when you try cache.put(k, my10GBObject). This *should* be possible if we
have a theoretical 100GB heap.
Potential solutions here are to fragment large objects, and store each fragment under
separate keys. Another approach would be to directly stream objects to disk. etc. Needs
thought and design, possibly a separate API to prevent 'pollution" of the more
simplistic API. (JumboCache?)
Re: fragmenting, issues to overcome:
How many chunks to fragment into? Max size of each key could be configured, but how do
we determine the size of an Object? VM instrumentation? Or perhaps the JumboCache only
stores byte[]'s?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira