<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Brent,<br>
<br>
I see your point but this change would require quite a big retooling
of MapReduce API and we can not do that in 5.2 minor release. Lets
leave that for 6.0. <br>
<br>
For 5.2 I want to add method MapReduceTask
combinedWith(Reducer<KOut,VOut> combiner). That way would
properly deal with <i>commutative</i> and <i>associative </i>reducer
and eliminate potential problems we currently do not address: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/">http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/</a>
<br>
<br>
Regards,<br>
Vladimir<br>
<br>
On 12-01-31 2:46 PM, Vladimir Blagojevic wrote:
<blockquote cite="mid:4F2828DB.5020106@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Response from Brent Douglas:<br>
<br>
Hi Vladimir,
<div><br>
</div>
<div>I'm not sure this is the same thing being discussed however
if it is not I had intended to request this anyhow. When I
looked into using infinispans's map reduce facility this is the
the task I came up with:</div>
<div><br>
</div>
<div><a moz-do-not-send="true" href="http://pastebin.com/7GGjVnVt">http://pastebin.com/7GGjVnVt</a></div>
<div><br>
</div>
<div>I would prefer to specify it as:</div>
<div><br>
</div>
<div><a moz-do-not-send="true" href="http://pastebin.com/HTSq3g66">http://pastebin.com/HTSq3g66</a></div>
<div><br>
</div>
<div>I'm pretty sure this is the not intended use case but it
distributes the creation of my reports which is what I want.
It's not really a big deal for me as I can get around this
limitation by creating a wrapper class such as in the first
example but it would be nice if I did not have to. Is this a
reasonable request?</div>
<div><br>
</div>
<div>Also, and this is probably just wrong, when I use this I
bundle up the task and execute it via JMS. Would it be
reasonable to make Collator extend Serializable?</div>
<div><br>
</div>
<div>Brent</div>
<br>
<br>
On 12-01-30 11:47 AM, Vladimir Blagojevic wrote:
<blockquote cite="mid:4F26AD6E.3030206@redhat.com" type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
Guys,<br>
<br>
I was looking at this again recently and I still do not
understand how combiner could have different interface than
Reducer! Hadoop forces a user to implement combiner as a Reducer
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://developer.yahoo.com/hadoop/tutorial/module4.html#functionality">http://developer.yahoo.com/hadoop/tutorial/module4.html#functionality</a>
and <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setCombinerClass%28java.lang.Class%29">http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setCombinerClass%28java.lang.Class%29</a>
In addition, the original paper does not mention any change of
types.<br>
<br>
What we have admittedly done wrong is to apply Reducer on
individual Mapper
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
without checking if a reduce function is both <i>commutative</i>
and <i>associative</i>! This can lead to problems: <a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/">http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/</a><br>
<br>
So yes, I am all for adding Combiner (it should do the optional
reducer per mapper we do automatically now) but I do not see why
we have to change the interface!<br>
<br>
<br>
Regards,<br>
Vladimir<br>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
infinispan-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
infinispan-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
</body>
</html>