[infinispan-dev] MapReduce limitations and suggestions.

Mon Feb 17 07:25:35 EST 2014

On 17 February 2014 08:42, Dan Berindei <dan.berindei at gmail.com> wrote:
> Hi Etienne
>
> I was going to suggest using a combiner - the combiner would process the
> mapper results from just one node, so you should need at most double the
> memory on that node. I guess we could reduce the memory requirements even
> more if the combiner could run concurrently with the mapper... Vladimir,
> does it sound like a reasonable feature request?

Yes that's something I've discussed with Vladimir before.
The problem is - as the LEADS experts of M/R like Evangelios explained
to us in London - is that in many practical use cases you can't apply
a combiner, and the Reducer needs to be run on the full set.

I think Evangelios also mentioned that the actual the set processed by
the Reducer is also expected to be sorted, apparently it's
"interesting" that we don't do such things. This can be taken as a
negative point as not all problems are solvable, but is also making it
interesting for being able to resolve some other problems with a
higher level of efficiency so it's not necessarily something that we
might want to throw away.

Might be interesting to keep our design with the current limitations,
and to also pursue a second mode of operation in which we make a good
Hadoop integration, to not reinvent the wheel in the area of the more
complex tasks, also providing the benefit of API compatibility to
allow other systems such as Apache Nutch and Mahout to run on
Infinispan without significant changes.

>
> I'm afraid in your situation using a cache store wouldn't help, as the
> intermediate values for the same key are stored as a list in a single entry.
> So if all cars are red, there would be just one intermediate key in the
> intermediate cache, and there would be nothing to evict to the cache store.
> Vladimir, do you think we could somehow "chunk" the intermediary values into
> multiple entries grouped by the intermediary key, to support this scenario?
>
> For reference, though, a limited version of what you're asking for is
> already available. You can change the configuration of the intermediary
> cache by defining a "__tmpMapReduce" cache in your configuration. That
> configuration will be used for all M/R tasks, whether they use the shared
> intermediate cache or they create their own.

I really hope we can get rid of temporary caches, but if need be
please make sure each task has an isolated execution context: names of
temporary caches - or their keys - need to avoid collisions with other
jobs.
Also, if we start spawning additional caches automagically I have no
idea how people will be able to define boundaries of heap size we're
allowed to use: such matters can not be left to the user's
responsibility to figure out.

Sanne

>
> Cheers
> Dan
>
>
>
> On Mon, Feb 17, 2014 at 10:18 AM, Etienne Riviere <etienne.riviere at unine.ch>
> wrote:
>>
>> Hi Radim,
>>
>> I might misunderstand your suggestion but many M/R jobs actually require
>> to run the two phases one after the other, and henceforth to store the
>> intermediate results somewhere. While some may slightly reduce intermediate
>> memory usage by using a combiner function (e.g., the word-count example), I
>> don’t see how we can avoid intermediate storage altogether.
>>
>> Thanks,
>> Etienne (leads project — as Evangelos who initiated the thread)
>>
>> On 17 Feb 2014, at 08:48, Radim Vansa <rvansa at redhat.com> wrote:
>>
>> > I think that the intermediate cache is not required at all. The M/R
>> > algorithm itself can (and should!) run with memory occupied by the
>> > result of reduction. The current implementation with Map first and
>> > Reduce after that will always have these problems, using a cache for
>> > temporary caching the result is only a workaround.
>> >
>> > The only situation when temporary cache could be useful is when the
>> > result grows linearly (or close to that or even more) with the amount of
>> > reduced entries. This would be the case for groupBy producing Map<Color,
>> > List<Entry>> from all entries in cache. Then the task does not scale and
>> > should be redesigned anyway, but flushing the results into cache backed
>> > by cache store could help.
>> >
>> > Radim
>> >
>> > On 02/14/2014 04:54 PM, Vladimir Blagojevic wrote:
>> >> Tristan,
>> >>
>> >> Actually they are not addressed in this pull request but the feature
>> >> where custom output cache is used instead of results being returned is
>> >> next in the implementation pipeline.
>> >>
>> >> Evangelos, indeed, depending on a reducer function all intermediate
>> >> KOut/VOut pairs might be moved to a single node. How would custom cache
>> >> help in this case?
>> >>
>> >> Regards,
>> >> Vladimir
>> >>
>> >>
>> >> On 2/14/2014, 10:16 AM, Tristan Tarrant wrote:
>> >>> Hi Evangelos,
>> >>>
>> >>> you might be interested in looking into a current pull request which
>> >>> addresses some (all?) of these issues
>> >>>
>> >>> https://github.com/infinispan/infinispan/pull/2300
>> >>>
>> >>> Tristan
>> >>>
>> >>> On 14/02/2014 16:10, Evangelos Vazaios wrote:
>> >>>> Hello everyone,
>> >>>>
>> >>>> I started using the MapReduce implementation of Infinispan and I came
>> >>>> across some possible limitations. Thus,  I want to make some
>> >>>> suggestions
>> >>>> about the MapReduce (MR) implementation of Infinispan.
>> >>>> Depending on the algorithm,  there might be some memory problems,
>> >>>> especially for intermediate results.
>> >>>> An example of such a case is  group by. Suppose that we have a
>> >>>> cluster
>> >>>> of 2 nodes with 2 GB  available. Let a distributed cache, where
>> >>>> simple
>> >>>> car objects (id,brand,colour) are stored and the total size of data
>> >>>> is
>> >>>> 3.5GB. If all objects have the same colour , then all 3.5 GB would go
>> >>>> to
>> >>>> only one reducer, as a result an OutOfMemoryException will be thrown.
>> >>>>
>> >>>> To overcome these limitations, I propose to add as parameter the name
>> >>>> of
>> >>>> the intermediate cache to be used. This will enable the creation of a
>> >>>> custom configured cache that deals with the memory limitations.
>> >>>>
>> >>>> Another feature that I would like to have is to set the name of the
>> >>>> output cache. The reasoning behind this is similar to the one
>> >>>> mentioned
>> >>>> above.
>> >>>>
>> >>>> I wait for your thoughts on these two suggestions.
>> >>>>
>> >>>> Regards,
>> >>>> Evangelos
>> >>>> _______________________________________________
>> >>>> infinispan-dev mailing list
>> >>>> infinispan-dev at lists.jboss.org
>> >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> infinispan-dev mailing list
>> >>> infinispan-dev at lists.jboss.org
>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >> _______________________________________________
>> >> infinispan-dev mailing list
>> >> infinispan-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> >
>> >
>> > --
>> > Radim Vansa <rvansa at redhat.com>
>> > JBoss DataGrid QA
>> >
>> > _______________________________________________
>> > infinispan-dev mailing list
>> > infinispan-dev at lists.jboss.org
>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev