[infinispan-dev] Memory is the new Disk, Disk is the new Tape

Do we plan to provide some sort of IO interface to Infinispan ? The idea is to let users write large files to the grid and read them as well. We could think of implementing all of the classes of java.io.* (NIO as well ?) and layering them on top of Infinispan. I'm thinking an implementation e.g. of a file stream could chunk the stream up into blocks of 1000 bytes. Each block has an ID and the IDs for all blocks of a given file are stored in the inode. The inode is referenced by the filename and contains the list of block IDs. Each block could potentially be stored on a different cluster node and - depending on repl-count - be stored multiple times in the cluster. Example: reading a file: - The impl of the stream locates the inode based on a consistent hash over the filename - The file ptr is at 0 - When reading 100 bytes, the first ID is fetched from the inode and its block is read from a cluster node, again based on the consistent hash(block-ID) - When all the data has been read from the block, we have to fetch the next block and so on For random access files, we'd compute the block ID based on the fileptr % block-size, this would be simple to implement. If we did this, we could truly have an in-memory cluster filesystem and handle files larger than the physical memory on any given box ! As a second line of defense, e.g. when the entire cluster crashes, we could still stream the modified blocks back to disk, but this could be done asynchronously. Note that the Gridblocks project once did this, but they stored all blocks on disk, and their project was not about in-memory file systems. Google File System does something similar, but their data is also stored on disk. I don't know what Amazon's BigTable does, have to read up on it. WDYT ? -- Bela Ban Lead JGroups / Clustering Team JBoss - a division of Red Hat