<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Tue, Feb 25, 2014 at 9:31 PM, Vladimir Blagojevic <span dir="ltr"><<a href="mailto:vblagoje@redhat.com" target="_blank">vblagoje@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey,<br>
<br>
I am starting to like this thread more and more :-) In conclusion, for<br>
distributed executors we are not adding any new APIs because Callable<br>
implementers can already write to cache using existing API. We don't<br>
have to add any new elaborate callback/listener API either as users have<br>
not requested but should investigate Hadoop Reporter like interface to<br>
allow users some sense of task current execution phase.<br>
<br>
For map/reduce we will add a new method:<br>
<div class=""><br>
public void execute(Cache<KOut, VOut> resultsCache);<br>
<br>
</div>Using fluent MapReduceTask API users would be able to specify an<br>
intermediate cache:<br>
<br>
public MapReduceTask<KIn, VIn, KOut, VOut> usingIntermediateCache(String<br>
cacheName);<br>
<br>
We are not adding MapReduceTaskExecutionListener but more like JMX stats<br>
<div class="">for the MapReduce tasks in general: like average execution time, count<br>
etc. Also the ability to cancel a running task through JMX/JON would be<br>
nice.<br></div></blockquote></div><br></div><div class="gmail_extra">For statistics, I was thinking of adding a getStatistics() method to MapReduceTask that would return an object with the duration of each phase and the number of keys processed on each node, after the M/R task is done. This could probably be extended such that it gives the user in-progress information as well.<br>
<br></div><div class="gmail_extra">The in-progress information would also tie in nicely with a progress listener, but I feel the events you proposed are too coarse. If the user wanted to display a progress bar in his application, and the cluster only had 2 nodes, the progress bar would hover for half of the time around 0% and for the other half of the time around 50%. So we'd need to keep reporting something while a phase is in progress (e.g. by splitting a node's keys to more than one mapping task, and reporting the end of each subtask), otherwise the listener wouldn't be of much use.<br>
<br></div><div class="gmail_extra">Anyway, this would be something nice to have, but I don't think it's very important, so supplying some global statistics via JMX should be enough for now.<br></div><div class="gmail_extra">
<br></div><div class="gmail_extra">Cheers<br></div><div class="gmail_extra">Dan<br></div></div>