2) Mapper and Reducer should work taking advantage of multiple cores
even on the same node .. so not just divide& Â conquer across multiple
nodes but also locally. Was this done already?
Mapping is done across the entire cluster. Reduction is done only on one
node. We want to change that soon
https://community.jboss.org/wiki/Infinispan60-MapReduceEnhancements
I meant I'd need a way to process all cache entries with a single Map
instance but taking advantage of all CPU cores of a system: The
Mappers should be cloned and passed to different Runnables, each one
being fed a partition of the data (and ideally the first one to finish
should steal some work from the others).
I need to find my Master's thesis work (look at [1] for an
overview). I implemented a compiler for a Higher-Order parallel
programming language which went a bit further than map + reduce. You
basically build programs out of implicitly parallel functions, tell
it the number of nodes and the compiler maps the functions on the
nodes, implementing "sub-reduces" in the appropriate places. The
theory is probably more interesting than the implementation itself
(C++ w/MPI), but it's probably still worth a read and might inspire
some new directions where we can push Infinispan's distexec module.