<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 30 Nov 2010, at 12:56, 이희승 (Trustin Lee) wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>On 11/30/2010 07:11 PM, Manik Surtani wrote:<br></div></blockquote><div><br></div><div><SNIP /></div><br><blockquote type="cite"><div><blockquote type="cite">* Do we want to support this for replicated mode? Typically, replicated clusters will be small and won't really benefit from task execution. Maybe we add in support for replication later, if there is adequate demand? It shouldn't change the API, etc.<br></blockquote><br>For example, a user might want to run a cluster of 100 nodes with 2<br>replicas just like Google File System does. Why is such cluster<br>supposed to be small? Could you elaborate?<br></div></blockquote><div><br></div><div>Ah, in Infinispan Replication refers to REPL mode, which means every node maintains a replica. E.g., if you have 100 nodes, you have 100 copies of everything.</div><div><br></div><div>Distribution refers to using a Consistent Hash algorithm to maintain a fixed number of copies (e.g., 2) regardless of how big a cluster is (e.g., 100 nodes). I see the F/J framework being useful here.</div><div><br></div><div><a href="http://docs.jboss.org/infinispan/4.2/apidocs/org/infinispan/config/Configuration.html#setCacheMode(java.lang.String)">http://docs.jboss.org/infinispan/4.2/apidocs/org/infinispan/config/Configuration.html#setCacheMode(java.lang.String)</a></div><br><blockquote type="cite"><div><blockquote type="cite">* Could you define where each of these tasks are executed? E.g., in a grid of {A, B, C, D} and you invoke your example on A, the local tasks happen on any node which contains the data it needs, while the global task happens on A? Surely in many cases it makes sense to "reduce" before pushing results (final or intermediate) on the network?<br></blockquote><br>Each node runs the first two sub-tasks and then one node is chosen to<br>run the last one. Timeline will look like this:<br><br> A: Task1 .. Task2 ... Task3<br> B: Task1 .. Task2 /<br> C: Task1 .. Task2 /<br> D: Task1 .. Task2 /<br></div></blockquote><div><br></div><div>Right, but this is specific to your example. How would one specify what sort of characteristics they need from each task?</div><br><blockquote type="cite"><div><br>Because Task1 and 2 are executed in the same node, they could be done<br>via a simple invocation chaining. The transition from Task2 to Task3<br>will involve network communication. In the example, the output is only<br>one entry, so it will be kept in memory until the node A's Task3 picks<br>them up. If Task2 generates a lot of entries, each node will hold them<br>in memory and block the task execution on ctx.write() until the Task3<br>picks them up.<br><br>One potential problem is when Node A fails while running Task3. Because<br>the intermediary entries are loaded into memory and lost as soon as they<br>are consumed, there's no way to re-retrieve the intermediary output and<br>the entire task needs to be re-run. A user can simply split the task to<br>two parts to store the intermediary output in the cache.<br><br><blockquote type="cite">* Non-Java language support - this should be out of scope for now, as it adds a layer of complexity we don't need. We will certainly add support for map/reduce over Hot Rod in future, perhaps using JSON/JavaScript, but not for now. :)<br></blockquote><blockquote type="cite">* Client/Server development: again, out of scope for now. See above. Once we have map/reduce over Hot Rod, client/server would be trivial.<br></blockquote><br>Agreed. I didn't mean to implement in 5.0 unless there is good deal of<br>demand. :-)<br><br>More feed back appreciated in advanced, folks! :-)<br><br>-- <br>Trustin Lee, <a href="http://gleamynode.net/">http://gleamynode.net/</a><br><br>_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>https://lists.jboss.org/mailman/listinfo/infinispan-dev</div></blockquote></div><br><div>
<span class="Apple-style-span" style="font-size: 12px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org">manik@jboss.org</a></div><div>Lead, Infinispan</div><div>Lead, JBoss Cache</div><div><a href="http://www.infinispan.org">http://www.infinispan.org</a></div><div><a href="http://www.jbosscache.org">http://www.jbosscache.org</a></div><div><br></div></div></span><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></body></html>