[infinispan-issues] [JBoss JIRA] Commented: (ISPN-78) Large object support

[Olaf Bergner (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-78?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597763#comment-12597763 ] 

Olaf Bergner commented on ISPN-78:
----------------------------------

Hi Trustin,

1) Well, chunking is the solution MongoDB found for their GridFS - http://www.mongodb.org/display/DOCS/GridFS+Specification - and so far it seems to serve them well. At least I know that people are successfully using GridFS.
That being said it might indeed be desirable to additionally support an alternative where large objects are written through the cache to disk. I personally would opt to first implement the current solution based on chunking and see how it goes. Later, if the need arises, we should be able to transparently support write-through caches, too.

2) Right now the interface is
{code}
public interface StreamingHandler<K> {

   void writeToKey(K key, InputStream largeObject);

   OutputStream writeToKey(K key);

   InputStream readFromKey(K key);

   boolean removeKey(K key);

   StreamingHandler<K> withFlags(Flag... flags);
}
{code}
i.e. I support {{transferFrom(InputStream)}} via {{writeToKey(K, InputStream}}} and {{write(byte)}} via {{OutputStream writeToKey(K)}}. Using the latter a client will have to call {{close()}} on the {{OutputStream}} returned to complete the write. It seems to me that this very similar to your {{LargeObjectWriter}}.

3) Yes, method naming could definitely be improved and {{newLargeObject(K key)}} seems to make sense. I'm not so sure, though, that it would still be appropriate when *overwriting* an existing key. Maybe simply {{largeObject(K key)}}?

4) I don't think a user should know about the chunks a large object is partitioned into. After all, it's just an implementation detail and the exact structure of the partitioning scheme chosen by INFINISPAN is likely immaterial to the domain at hand. One exception might be when the user chooses the configurable chunk size according to not technical but functional reasons, i.e. he knows that a single record within a large file will *always* be 256k in size. But then he may use the {{InputStream}} returned from {{readKey(K key)}} to read a large object in chunks of arbitrary size.

As a general approach I would like to first get the interface and - by extension - implementation details directly affecting the interface right. How to best implement that interface is a question we may answer later.

The more I think about it, the less likely it seems to me that a user will *ever* want to mix "regular" and large objects within a single cache. At least most applications I know of wouldn't want to do so. Plus I imagine that it might prove difficult to tune a cache containing regular as well as large objects. Maybe we should restrict storage of large objects to dedicated caches? This would have the added bonus of being able to define appropriate default settings for large object caches.

Any thoughts?


> Large object support
> --------------------
>
>                 Key: ISPN-78
>                 URL: https://issues.jboss.org/browse/ISPN-78
>             Project: Infinispan
>          Issue Type: Feature Request
>          Components: Core API
>            Reporter: Manik Surtani
>            Assignee: Olaf Bergner
>             Fix For: 5.1.0.BETA1, 5.1.0.Final
>
>
> if each VM is allocated a 2GB heap and you have a 100 nodes in a grid with 1 redundant copy for each key, you have a theoretical addressable heap of 100GB.  But you are limited by (half) the heap of a single VM per entry, since entries are stored whole.
> E.g., cache.put(k, my2GBObject) will fail since you need at least 2GB for the object + another 2GB for its serialized form.
> This gets worse when you try cache.put(k, my10GBObject).  This *should* be possible if we have a theoretical 100GB heap.
> Potential solutions here are to fragment large objects, and store each fragment under separate keys.  Another approach would be to directly stream objects to disk. etc.  Needs thought and design, possibly a separate API to prevent 'pollution" of the more simplistic API.  (JumboCache?)
> Re: fragmenting, issues to overcome:
> How many chunks to fragment into?  Max size of each key could be configured, but how do we determine the size of an Object?  VM instrumentation?  Or perhaps the JumboCache only stores byte[]'s?  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira