[infinispan-dev] Distributed execution framework - API proposal(s)

[Vladimir Blagojevic]

On 11-01-11 12:27 PM, Manik Surtani wrote:
> You can do that but, even as the paper suggests, you usually end up with just 1 result.  Essentially it is a case of how simple we want to make the API.  I think for a large number of the use cases we've come across, mapping a K and V to a single T works fine (this is a simple value mapping you see in many functional languages).  The alternative is to map an entire tuple (e.g., k1, v1 becomes k2, v2) but as Java doesn't support multiple return values, so then you end up with Mapper.map() returning a tuple (Map.Entry<K2, V2>  in Java) which I find clunky.
>
> Perhaps, if there is adequate demand, we could extend Mapper with TupleMapper, which would support this.
>

Ok noted, but I still need a clarification about your proposal:

Reducer<T, R>  {
	// incrementally reduces a transformed entry.  Called once for each T produced by the mapper.
	// previously reduced value passed in each time.
	R reduce(T, R);
}

Collator<R>  {
	// Adds reduced results from remote nodes.  Called once for each R returned by a RemoteReducer.
	add(Address origin, R remote);

	// collates all results added so far.
	R collate();
}

So far I understand that we invoke reduce (T) for each returned T map(K,V) invocation. Now, where is this previously reduced R parameter coming from? Are you invoking reduce phase serially across a list of all returned results of T map(K,V) passing R from last invocation of reduce in a new invocation of reduce?



>>>>> * Similarly, Collator.add() should just need the address and the
>>>>> reduced result from each node (each node would only produce 1 result!)
>>>>>
>>>>> DistExec
>>>>> * Why do you have execute() and executeAsync() with no params?  What
>>>>> do these methods do?
>>>> In case user provided Factory for DistributedCallable. We have to handle
>>>> cases where simple no parameter constructor for callables is not
>>>> sufficient - hence factory.
>>> Why are we constructing DistributedCallables?  Surely the user passes in an instance?
>>>
>> Ok, so we need to serialize/migrate across JVMs this instance of
>> DistributedCallable and invoke it on a target Infinispan node, right? If
>> so, then we need to add K...input as a parameter to call function since
>> each migrated instance (on execution node) needs to have different input
>> for invocation.
> Yeah good point.  My bad.  :)  We'd need to document this appropriately in the DistributedCallable javadocs.
>

:-) What is more elegant and accommodating for interface implementers - 
passing parameter K to call of DistributedCallable or using factories 
for DistributedCallable?

Cheers,
Vladimir