[infinispan-dev] Parallel M/R

[Vladimir Blagojevic]

On 12/6/2013, 11:40 AM, Mircea Markus wrote:
> Hmm I think you could leverage the parallel iteration from the EquivalentConcurrentHashMapV8 there instead of writing it yourself ;)
>
Hi, for those interested in parallel M/R I have uploaded my first 
proposal that will hopefully, with your input, soon turn into a PR. A 
couple of things to note: we do parallel iteration using JDK8 
ConcurrentHashMap port (see DefaultDataContainer). Doing parallel 
iteration makes sense when we have to iterate over all keys in data 
container and check whether each individual key should be traversed 
(using Mircea's filter). Sometimes it does not make sense to iterate 
over all keys as input keys are provided to M/R task and input keys 
represent only a small subset of all keys in the data container. Even in 
those cases we do parallel execution but in a slightly different 
approach (again see DefaultDataContainer).

Please provide feedback directly on 
https://github.com/vblagoje/infinispan/tree/t_2284_mm or if you have 
some general comments here on dev list.

Thanks for you feedback,
Vladimir