[infinispan-dev] grouping and GridFS

[Dennis Reed]

It doesn't make sense.  :)

The reason grid file systems exist is to distribute the file around the 
cluster.
(both for performance so the network interface of a single server isn't 
a bottleneck,
and for disk space so the available space on a single server isn't a 
bottlenect)

If you don't want to distribute the file, a grid filesystem probably 
isn't the right choice.

-Dennis

On 03/05/2014 09:04 AM, Ales Justin wrote:
> But yeah, the moment I start chunking, I would still like to have the grouped -- same node.
> Or that doesn't make sense?
> (hence having this discussion ;-)
>
> -Ales
>
> On 05 Mar 2014, at 16:01, Sanne Grinovero <sanne at infinispan.org> wrote:
>
>> On 5 March 2014 14:54, Ales Justin <ales.justin at gmail.com> wrote:
>>> Why do you chunk at all if you want them stored together?
>>>
>>> I only use chunking if I can't avoid it, to spread large files.
>>>
>>> That's what's GridFS all about -- store very large files.
>>> Hence chunking.
>>>
>>> So you're saying we should know the limit of what we can store on 1 node,
>>> if bigger, spread, therefore no grouping.
>> Yes, but a very conservative approximation would be good enough: you
>> don't need hardware specifications to figure out a reasonable
>> threshold.
>> If I had to make up a number out of thin air, I'd pick something
>> around 10MB: any file below that threshold would not use chunking and
>> be nicely stored together to be retrieved efficiently; beyond that
>> start distributing.
>> (this figure could probably use some testing if you're looking into performance)
>>
>> Sanne
>>
>>> -Ales
>>>
>>> On 5 Mar 2014 11:22, "Ales Justin" <ales.justin at gmail.com> wrote:
>>>> Just having a discussion with Bela about this.
>>>>
>>>> I guess having "grouping" on GridFS' content would make sense.
>>>> e.g. put all chunks on the same node
>>>>
>>>> Is this doable?
>>>> Afaiu, we would need to have some sort of "similarity" function for
>>>> content's metadata?
>>>>
>>>> -Ales
>>>>