Re: [infinispan-dev] Map Reduce 2.0
On 12-02-20 3:43 PM, Manik Surtani wrote:
I was under the impression reduce is distributed too? Don't we
do the mapping on each node, then a first-pass reduce on each node too, before streaming
results back to the caller node?
What we do in first-pass reduce is essentially
combine and we should not
do that blindly because this eager reduction/combine only works when
reduce function is both /commutative/ and /associative/! This can lead
to problems when it is not:
http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4...
So yes first-pass reduce is distributed but second-phase reduce should
be distributed as well! Currently it is not!
This all makes sense as well, however one problem with the consistent
hash approach is that it is prone to change when there is a topology change. How would
you deal with that? Would you maintain a history of consistent hashes?
I don't think I understand! Even if there is a topology change
intermediate results Map<KOut, List<VOut>> will be migrated, all we need
is KOut's, we can hash it and find out where List<VOut> are, no?
Vladimir
Attachments:
- attachment.html (text/html — 1.8 KB)