[infinispan-issues] [JBoss JIRA] (ISPN-4022) M/R: Run the combiner concurrently with the mapper

[Dan Berindei (JIRA)]

Dan Berindei created ISPN-4022:
----------------------------------

             Summary: M/R: Run the combiner concurrently with the mapper
                 Key: ISPN-4022
                 URL: https://issues.jboss.org/browse/ISPN-4022
             Project: Infinispan
          Issue Type: Feature Request
          Components: Core, Distributed Execution and Map/Reduce
    Affects Versions: 6.0.1.Final
            Reporter: Dan Berindei
            Assignee: Vladimir Blagojevic
             Fix For: 7.0.0.Final


Because we only run the combiner after we finished the mapping phase, we need to keep all the results of the mapping phase in memory at once. We should split the output of the mapper into chunks and allow the combiner to process chunks while the mapper is still running, relieving some of the memory pressure. Maybe even block the mapper if there are too many chunks in-flight.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira