[infinispan-dev] MapReduce limitations and suggestions.

[Marcelo Pasin]

On 18/Feb/2014, at 10:59 , Dan Berindei <dan.berindei at gmail.com> wrote:
> I think Hadoop only loads a block of intermediate values in memory at once, and can even sort the intermediate values (with a user-supplied comparison function) so that the reduce function can work on a sorted list without loading the values in memory itself.

Actually, Hadoop sorts in the map node, the last two steps being sort and combine. Reduce nodes fetch partitions from the map nodes and just merges them. Such partitions are fetched incrementally, and whenever a given key ends in all partially fetched partitions, reduce() is called.

Cheers, MP
--
Marcelo Pasin
Université de Neuchâtel · Institut d'informatique
rue Emile-Argand 11 · Case postale 158 · 2000 Neuchâtel · Switzerland