[infinispan-issues] [JBoss JIRA] (ISPN-2284) Execute Mapper and Reducer tasks in parallel where possible
[
https://issues.jboss.org/browse/ISPN-2284?page=com.atlassian.jira.plugin....
]
Mircea Markus commented on ISPN-2284:
-------------------------------------
Starting with Infinispan 6.0 we already allow parallel iteration of the entries stored in
the persistent storage: see AdvancedAsyncCacheLoader.process(KeyFilter, CacheLoaderTask,
Executor executor..) method.
One possible way of doing this would be to have a new method:
{code:java}
DataContainer.process(Executor executor)
{code}
Then have a thread pool that's used for configuring the number of thread taking care
of the parallel iteration.
Also the DefaultDataContainer already can implement this by leveraging the fact that
DataContainer.entries is an EquivalentConcurrentHashMapV8 implementation which already
support parallel iteration of entries.
Execute Mapper and Reducer tasks in parallel where possible
-----------------------------------------------------------
Key: ISPN-2284
URL: https://issues.jboss.org/browse/ISPN-2284
Project: Infinispan
Issue Type: Feature Request
Components: Distributed Execution and Map/Reduce
Affects Versions: 5.2.0.Alpha3
Reporter: Vladimir Blagojevic
Assignee: Vladimir Blagojevic
Fix For: 7.0.0.Final
Attachments: Infinispan-MapReduce Job Times.png
In our current implementation of Map/Reduce, Mapper and Reducer tasks executed on remote
JVMs load and process key/values serially on a single thread. Where and if possible we
should use executors to process keys/values in parallel.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira