[infinispan-dev] Map/Reduce enhancements with ISPN-2037
Vladimir Blagojevic
vblagoje at redhat.com
Fri Jul 13 13:13:09 EDT 2012
Guys,
I believe I finished ISPN-2037 that Sanne needed for his queries.
However, adding ISPN-2037 commit to current master involves injection of
CacheLoaderManager into MapReduceCommand. That solution is pretty ugly
and we are switching soon to a new map/reduce implementation anyway.
Going in that direction is a waste IMHO.
I already talked to Mircea and Manik about using a Manager-like service
much like RecoveryManager or EvictionManager to manage incoming
map/reduce invocations on Infinispan node. MapReduceManager is per cache
component. It turned out to be an excellent fit! I converted our current
solution where map is done in distributed fashion while reduction is
done on master task node to this new approach. I made it a default just
as it is now. Then I added all the previous work on new map/reduce
reviewed by Dan and Galder and adjusted it to this new Manager paradigm.
Users can choose if they want reduce part done in distributed approach
simply by using a different MapReduceTask constructor. And finally I
added ISPN-2037 as a separate commit on top of this rework.
This proposal is at
https://github.com/vblagoje/infinispan/commits/t_mr_final
I would like to hear your thoughts and comments before proceeding forward.
Regards,
Vladimir
More information about the infinispan-dev
mailing list