[infinispan-issues] [JBoss JIRA] (ISPN-2284) Execute Mapper and Reducer tasks in parallel where possible

[Mircea Markus (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927618#comment-12927618 ] 

Mircea Markus commented on ISPN-2284:
-------------------------------------

Starting with Infinispan 6.0 we already allow parallel iteration of the entries stored in the persistent storage: see AdvancedAsyncCacheLoader.process(KeyFilter, CacheLoaderTask, Executor executor..) method.
One possible way of doing this would be to have a new method:
{code:java}
DataContainer.process(Executor executor) 
{code}
Then have a thread pool that's used for configuring the number of thread taking care of the parallel iteration.
Also the DefaultDataContainer already can implement this by leveraging the fact that DataContainer.entries is an EquivalentConcurrentHashMapV8 implementation which already support parallel iteration of entries.
                
> Execute Mapper and Reducer tasks in parallel where possible
> -----------------------------------------------------------
>
>                 Key: ISPN-2284
>                 URL: https://issues.jboss.org/browse/ISPN-2284
>             Project: Infinispan
>          Issue Type: Feature Request
>          Components: Distributed Execution and Map/Reduce
>    Affects Versions: 5.2.0.Alpha3
>            Reporter: Vladimir Blagojevic
>            Assignee: Vladimir Blagojevic
>             Fix For: 7.0.0.Final
>
>         Attachments: Infinispan-MapReduce Job Times.png
>
>
> In our current implementation of Map/Reduce, Mapper and Reducer tasks executed on remote JVMs load and process key/values serially on a single thread. Where and if possible we should use executors to process keys/values in parallel.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira