<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 11 Jan 2011, at 16:16, Vladimir Blagojevic wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On 11-01-11 12:27 PM, Manik Surtani wrote:<br><blockquote type="cite">You can do that but, even as the paper suggests, you usually end up with just 1 result. Essentially it is a case of how simple we want to make the API. I think for a large number of the use cases we've come across, mapping a K and V to a single T works fine (this is a simple value mapping you see in many functional languages). The alternative is to map an entire tuple (e.g., k1, v1 becomes k2, v2) but as Java doesn't support multiple return values, so then you end up with Mapper.map() returning a tuple (Map.Entry<K2, V2> in Java) which I find clunky.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Perhaps, if there is adequate demand, we could extend Mapper with TupleMapper, which would support this.<br></blockquote><blockquote type="cite"><br></blockquote><br>Ok noted, but I still need a clarification about your proposal:<br><br>Reducer<T, R> {<br><span class="Apple-tab-span" style="white-space:pre"> </span>// incrementally reduces a transformed entry. Called once for each T produced by the mapper.<br><span class="Apple-tab-span" style="white-space:pre"> </span>// previously reduced value passed in each time.<br><span class="Apple-tab-span" style="white-space:pre"> </span>R reduce(T, R);<br>}<br><br>Collator<R> {<br><span class="Apple-tab-span" style="white-space:pre"> </span>// Adds reduced results from remote nodes. Called once for each R returned by a RemoteReducer.<br><span class="Apple-tab-span" style="white-space:pre"> </span>add(Address origin, R remote);<br><br><span class="Apple-tab-span" style="white-space:pre"> </span>// collates all results added so far.<br><span class="Apple-tab-span" style="white-space:pre"> </span>R collate();<br>}<br><br>So far I understand that we invoke reduce (T) for each returned T map(K,V) invocation. Now, where is this previously reduced R parameter coming from? Are you invoking reduce phase serially across a list of all returned results of T map(K,V) passing R from last invocation of reduce in a new invocation of reduce?<br></div></blockquote><div><br></div><div>How about this pseudocode:</div><div><br></div><div>On each node:</div><div> mapped = list()</div><div> for entry in cache.entries:</div><div> t = mapper.map(entry.key, entry.value)</div><div> mapped.add(t)</div><div><br></div><div> r = null</div><div> for t in mapped:</div><div> r = reducer.reduce(t, r)</div><div><br></div><div> return r to calling node</div><div><br></div><div>On calling node:</div><div> reduced_results = invoke map reduce task on all nodes, retrieve map{address: result}</div><div> for r in reduced_results.entries:</div><div> remote_address = r.key</div><div> remote_reduced_result = r.value</div><div> collator.add(remote_address, remote_reduced_result)</div><div><br></div><div> return collator.collate()</div><div> </div><div><br></div><blockquote type="cite"><div><span class="Apple-style-span" style="color: rgb(84, 0, 0); ">* Similarly, Collator.add() should just need the address and the</span><br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">reduced result from each node (each node would only produce 1 result!)<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">DistExec<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">* Why do you have execute() and executeAsync() with no params? What<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">do these methods do?<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">In case user provided Factory for DistributedCallable. We have to handle<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">cases where simple no parameter constructor for callables is not<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">sufficient - hence factory.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Why are we constructing DistributedCallables? Surely the user passes in an instance?<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Ok, so we need to serialize/migrate across JVMs this instance of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">DistributedCallable and invoke it on a target Infinispan node, right? If<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">so, then we need to add K...input as a parameter to call function since<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">each migrated instance (on execution node) needs to have different input<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">for invocation.<br></blockquote></blockquote><blockquote type="cite">Yeah good point. My bad. :) We'd need to document this appropriately in the DistributedCallable javadocs.<br></blockquote><blockquote type="cite"><br></blockquote><br>:-) What is more elegant and accommodating for interface implementers - <br>passing parameter K to call of DistributedCallable or using factories <br>for DistributedCallable?<br></div></blockquote><div><br></div><div>Hmm. I think both approaches are best, to allow for the most flexibility. So perhaps overload execute() with the following:</div><div><br></div><div>execute(DistributedCallable c); // Javadoc says DistributedCallable must have a no arg ctor</div><div>execute(DistributedCallableFactory cf); // Javadoc says DistributedCallable must have a no arg ctor</div><div>execute(Class<? extends DistributedCallable> cc); </div><div>execute(Class<? extends DistributedCallableFactory> cfc); </div><div><div><br></div><div>Also, wrt. names and methods on the API on the Distributed Exec part of things, wdyt about aligning it with JDK executors (see separate email from me)?</div><div><br></div><div>And - not trying to confuse you, promise! - here's some scope for additional work later on as well: network classloaders to deal with non-homogenous deployments with remote tasks like these. ;)</div><div><br></div><div>Cheers</div><div>Manik</div></div><br><blockquote type="cite"><div><br>Cheers,<br>Vladimir<br>_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>https://lists.jboss.org/mailman/listinfo/infinispan-dev<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org">manik@jboss.org</a></div><div><a href="http://twitter.com/maniksurtani">twitter.com/maniksurtani</a></div><div><br></div><div>Lead, Infinispan</div><div><a href="http://www.infinispan.org">http://www.infinispan.org</a></div><div><br></div></div></span><br class="Apple-interchange-newline">
</div>
<br></body></html>