[infinispan-dev] SimpleFileCacheStore

[Radim Vansa]

On 07/31/2013 10:31 AM, Mircea Markus wrote:
> On 30 Jul 2013, at 20:03, Shane Johnson <shjohnso at redhat.com> wrote:
>
>> One option might be to use a fix key set size and simply increment the value for each key by X every time it is written. Sort of like an object with a collection and every time a nested object is added to the collection, the parent object is written to the cache.
> In this example the aggregated objects should hold a foreign key to the aggregator, otherwise the object would grow indefinitely causing OOMs.
> But good point nevertheless: if the size of every object varies from 1k,2k,3k..Nk circularly, then the total disk capacity allocated for storing an entry is ((1+N)*N)/2.
> So for storing 100MB of data you'd end up with a file with size 5GB. On top of that the memory consumption grows proportionally, as we keep in memory information about all the allocated segments on disk.
>
>
I am afraid that with single-file storage you cannot evade this without 
coalescing and splitting of empty blocks, which also results in 
fragmentation. This would mean either reads for record on each side to 
check if it's free (slowdown) or all blocks sorted by offset in memory 
(another memory requirement). There may be other options such as 
buddy-block system but I think these have degenerative allocation 
patterns as well.

It's a pity there is no generic space-allocation structure in Java, as 
Doug Lea (author of long-time used C malloc implementation) wrote part 
of the concurrent stuff in Java runtime libraries...

Radim