Re: [infinispan-dev] MapReduce limitations and suggestions.
On 2/18/2014, 4:59 AM, Dan Berindei wrote:
The limitation we have now is that in the reduce phase, the entire
list of values for one intermediate key must be in memory at once. I
think Hadoop only loads a block of intermediate values in memory at
once, and can even sort the intermediate values (with a user-supplied
comparison function) so that the reduce function can work on a sorted
list without loading the values in memory itself.
Dan and others,
This is where Sanne's idea comes into play. Why collect entire list of
intermediate values for each intermediate key and then invoke reduce on
those values when we can invoke reduce each time new intermediate value
gets inserted?
https://issues.jboss.org/browse/ISPN-3999
Cheers,
Vladimir