[infinispan-dev] SimpleFileCacheStore
Radim Vansa
rvansa at redhat.com
Wed Jul 31 06:57:36 EDT 2013
On 07/31/2013 10:31 AM, Mircea Markus wrote:
> On 30 Jul 2013, at 20:03, Shane Johnson <shjohnso at redhat.com> wrote:
>
>> One option might be to use a fix key set size and simply increment the value for each key by X every time it is written. Sort of like an object with a collection and every time a nested object is added to the collection, the parent object is written to the cache.
> In this example the aggregated objects should hold a foreign key to the aggregator, otherwise the object would grow indefinitely causing OOMs.
> But good point nevertheless: if the size of every object varies from 1k,2k,3k..Nk circularly, then the total disk capacity allocated for storing an entry is ((1+N)*N)/2.
> So for storing 100MB of data you'd end up with a file with size 5GB. On top of that the memory consumption grows proportionally, as we keep in memory information about all the allocated segments on disk.
>
>
I am afraid that with single-file storage you cannot evade this without
coalescing and splitting of empty blocks, which also results in
fragmentation. This would mean either reads for record on each side to
check if it's free (slowdown) or all blocks sorted by offset in memory
(another memory requirement). There may be other options such as
buddy-block system but I think these have degenerative allocation
patterns as well.
It's a pity there is no generic space-allocation structure in Java, as
Doug Lea (author of long-time used C malloc implementation) wrote part
of the concurrent stuff in Java runtime libraries...
Radim
More information about the infinispan-dev
mailing list