[infinispan-dev] Distributed execution framework - API proposal(s)

[Manik Surtani]

Hi there.

Lets take the 2 API sets (M/R and distributed exec) separately:

M/R
* Why does MapReduceTask extend Set?  It probably shouldn't even extend DistributedRemoteTask, since some methods such as execute(Callable) doesn't make sense for MapReduceTask.  Perhaps a common super-interface?  That alone should simplify the generics... 
* Why does Mapper.map() return a Map? [1] Shouldn't it return a single transformation for a given K and V?
* Reducer.reduce()'s return type should not necessarily be the same type as the mapped transformation.  Also, does the reducer really need the key? Surely just the transformed version of each K/V pair.
* Similarly, Collator.add() should just need the address and the reduced result from each node (each node would only produce 1 result!)

DistExec
* Why do you have execute() and executeAsync() with no params?  What do these methods do?
* Not sure what DistributedTaskContext is all about.  What does read() and write() do?  Is this meant to be a "layer" in front of the cache in question?  Why not just pass in a ref to the cache?
* What does Factory do?

Almost there!  :-)

Cheers
Manik

[1] How many times can I say "map" in one sentence?  :)


On 10 Jan 2011, at 19:28, Vladimir Blagojevic wrote:

> On 11-01-05 3:39 PM, Manik Surtani wrote:
>> 
>> 
>>> 
>>> 
>>> Awesome! I love this fluent API proposal!
>>> 
>> 
>> Cool.  Then if this is the case lets stick with the fluent API - I think it would make life much easier for developers.
>> 
>> --
>> Manik Surtani
>> 
> 
> Proposal Manik and I talked about is here https://github.com/vblagoje/infinispan/commit/59b0c3cb78bce70fdcd49d2d36f68de110944fc6
> 
> I have slightly changed it to better match map/reduce paradigm described here http://labs.google.com/papers/mapreduce.html
> Therefore, we now have both map/reduce and a more simple "execute this task unit on remote nodes and return results" proposal. 
> Users might not like number of type parameters in MapReduceTask. I wanted to be as precise as possible to begin with and we can discuss how and if to reduce them.
> Example for map/reduce is WordCountExample.java. Have a look.
> 
> I put a proposal in distexec package. I wanted to have parent package to branch from. We need to accommodate simple execution model, map/reduce, various execution policies and so on. They could not fit all in mapreduce package and its children. Hence distexec package. If you think you have better name - speak up.
> 
> 
> Regards,
> Vladimir
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20110111/dc82ad59/attachment.html