[infinispan-dev] Extend GridFS

[Bela Ban]

On 7/13/11 4:24 AM, Yuri de Wit wrote:
> Ok, I made good progress today and I have close to 100% of my existing
> app running on top of GridFS loading chunked data directly from our
> existing data files. I have made bug fixes and enhancements to the
> org.infinispan.io package and the rest of the implementation went into
> the 3 new classes I mentioned before: FileCacheStore,
> FileDataCacheStore and FileMetadataCacheStore.


Excellent !

I suggest open a pull request on Infinispan, so the changes make it into 
the code base.


>  I basically got the
> functionality right by running single node and now I want to start
> testing in a cluster.


OK


> One of the todo's is to implement chunked writes and I will see if I
> can tackle that later this week. Another todo that seems worth
> exploring is GZipping the chunks as soon as it is read from the file
> system or only when marshalling and sending it over the wire.

Yes. One option is to add COMPRESS to the JGroups protocol stack used. 
Enabled tracing for org.jgroups.protocols.COMPRESS allows you to see the 
compression ratio, to confirm that it makes sense to use compression. Of 
course this depends on the type of data you put into the virtual file 
system.


> Are there ways to plug that in? I saw something like StreamingMarshaller
> that could decouple this from the actual GridFilesystem.

Yes, you could always do this at the Infinispan level (or even at the 
level of GridFS, but that would mean duplicating functionality).

-- 
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat