<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Feb 18, 2014 at 5:46 PM, Evangelos Vazaios <span dir="ltr"><<a href="mailto:vagvaz@gmail.com" target="_blank">vagvaz@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On 02/18/2014 05:36 PM, Vladimir Blagojevic wrote:<br>
> On 2/18/2014, 4:59 AM, Dan Berindei wrote:<br>
>><br>
>> The limitation we have now is that in the reduce phase, the entire<br>
>> list of values for one intermediate key must be in memory at once. I<br>
>> think Hadoop only loads a block of intermediate values in memory at<br>
>> once, and can even sort the intermediate values (with a user-supplied<br>
>> comparison function) so that the reduce function can work on a sorted<br>
>> list without loading the values in memory itself.<br>
>><br>
>><br>
> Dan and others,<br>
><br>
> This is where Sanne's idea comes into play. Why collect entire list of<br>
> intermediate values for each intermediate key and then invoke reduce on<br>
> those values when we can invoke reduce each time new intermediate value<br>
> gets inserted?<br>
><br>
</div>Because you cant. What you are saying is more like combining than<br>
reducing. If there is a combiner in the MapReduceTask you can execute<br>
the combiner on a subset (in your case 2) values with the same key and<br>
output one. But, this is not possible always.<br></blockquote><div><br></div>In theory we could stream each intermediate value independently to the combiner and then to the node of the reducer, and the reducer could start up immediately on the reducer node instead of waiting for the mapping phase to finish on all the mapping nodes (blocking when it doesn't have any more values to process). But I imagine that would be kind of tricky to implement.<br>
<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">> <a href="https://issues.jboss.org/browse/ISPN-3999" target="_blank">https://issues.jboss.org/browse/ISPN-3999</a><br>
><br>
> Cheers,<br>
> Vladimir<br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
><br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>