Re: [infinispan-dev] TopologySafe Map / Reduce
On 2014-10-10, 3:03 AM, Dan Berindei wrote:
The problem is that the intermediate keys aren't in the same segment:
we want the reduce phase to access only keys local to the reducing
node, and keys in different input segments can yield values for the
same intermediate key. So like you say, we'd have to retry on every
topology change in the intermediary cache, not just the ones affecting
segment _i_.
If we have to retry for all segments on every topology change than I am
not sure why would it make sense to work on this optimization and
topology handling mechanism at all. We have to handle the cases where
one node might have completed map phase and inserted deltas, while the
other only started inserting deltas, and the third one is still doing
map phase and has not inserted any deltas at all. The same thing with
reduce portion. It seems to me that in the end any algorithm we come up
with will not be not much better than: detect topology change, retry
map/reduce job.
Vladimir