Do we plan to provide some sort of IO interface to Infinispan ? The idea is to let users write large files to the grid and read them as well. We could think of implementing all of the classes of java.io.* (NIO as well ?) and layering them on top of Infinispan. I'm thinking an implementation e.g. of a file stream could chunk the stream up into blocks of 1000 bytes. Each block has an ID and the IDs for all blocks of a given file are stored in the inode. The inode is referenced by the filename and contains the list of block IDs. Each block could potentially be stored on a different cluster node and - depending on repl-count - be stored multiple times in the cluster. Example: reading a file: - The impl of the stream locates the inode based on a consistent hash over the filename - The file ptr is at 0 - When reading 100 bytes, the first ID is fetched from the inode and its block is read from a cluster node, again based on the consistent hash(block-ID) - When all the data has been read from the block, we have to fetch the next block and so on For random access files, we'd compute the block ID based on the fileptr % block-size, this would be simple to implement. If we did this, we could truly have an in-memory cluster filesystem and handle files larger than the physical memory on any given box ! As a second line of defense, e.g. when the entire cluster crashes, we could still stream the modified blocks back to disk, but this could be done asynchronously. Note that the Gridblocks project once did this, but they stored all blocks on disk, and their project was not about in-memory file systems. Google File System does something similar, but their data is also stored on disk. I don't know what Amazon's BigTable does, have to read up on it. WDYT ? -- Bela Ban Lead JGroups / Clustering Team JBoss - a division of Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Yes, a FUSE impl would be nice. Also, maybe an S3 impl of the SOAP interface ? I don't know how complex that protocol is, but I don't think this would be too complex. Of course, this depends on Amazon documenting their S3 protocol...

Manik Surtani wrote: > > On 27 Apr 2009, at 14:18, Bela Ban wrote: > > Yes, a FUSE impl would be nice. Also, maybe an S3 impl of the SOAP >> interface ? I don't know how complex that protocol is, but I don't think >> this would be too complex. Of course, this depends on Amazon documenting >> their S3 protocol... >> > > Exposing S3 as a File and as a FUSE plugin is probably trivial with > Adrian's JClouds [1] > > [1] http://code.google.com/p/javaclouds/ > Hmm, am I looking in the wrong place, this project seems to be completely devoid of content ? Hi Adrian, I strongly suggest you rename javaclouds to jclouds or something that doesn't have 'java' in it (trademarked) -- Bela Ban Lead JGroups / Clustering Team JBoss - a division of Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

k, thx, I'll take a look Adrian Cole wrote: > I agree.. > > http://code.google.com/p/jclouds/ is the real url. > > I submitted a cancel request for the other one. I just published the > source > today, documentation is coming! > > I'll have a look at fuse. It does sound like a good match. > > Cheers, > -Adrian > > On Mon, Apr 27, 2009 at 3:38 PM, Bela Ban <bban(a)redhat.com&gt; wrote: > > > >> Manik Surtani wrote: >> >> >> >>> On 27 Apr 2009, at 14:18, Bela Ban wrote: >>> >>> Yes, a FUSE impl would be nice. Also, maybe an S3 impl of the SOAP >>> >>> >>>> interface ? I don't know how complex that protocol is, but I don't >>>> think >>>> this would be too complex. Of course, this depends on Amazon >>>> documenting >>>> their S3 protocol... >>>> >>>> >>>> >>> Exposing S3 as a File and as a FUSE plugin is probably trivial with >>> Adrian's JClouds [1] >>> >>> [1] http://code.google.com/p/javaclouds/ >>> >>> >>> >> Hmm, am I looking in the wrong place, this project seems to be completely >> devoid of content ? >> >> Hi Adrian, I strongly suggest you rename javaclouds to jclouds or >> something >> that doesn't have 'java' in it (trademarked) >> >> -- >> Bela Ban >> Lead JGroups / Clustering Team >> JBoss - a division of Red Hat >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> > > > -- Bela Ban Lead JGroups / Clustering Team JBoss - a division of Red Hat