[infinispan-dev] TopologySafe Map / Reduce

[Vladimir Blagojevic]

On 2014-10-10, 3:03 AM, Dan Berindei wrote:
>
> The problem is that the intermediate keys aren't in the same segment: 
> we want the reduce phase to access only keys local to the reducing 
> node, and keys in different input segments can yield values for the 
> same intermediate key. So like you say, we'd have to retry on every 
> topology change in the intermediary cache, not just the ones affecting 
> segment _i_.
>
If we have to retry for all segments on every topology change than I am 
not sure why would it make sense to work on this optimization and 
topology handling mechanism at all. We have to handle the cases where 
one node might have completed map phase and inserted deltas, while the 
other only started inserting deltas, and the third one is still doing 
map phase and has not inserted any deltas at all. The same thing with 
reduce portion. It seems to me that in the end any algorithm we come up 
with will not be not much better than: detect topology change, retry 
map/reduce job.

Vladimir