[infinispan-dev] Adding a Combiner

Vladimir Blagojevic vblagoje at redhat.com
Tue Jan 31 12:46:03 EST 2012


Response from Brent Douglas:

Hi Vladimir,

I'm not sure this is the same thing being discussed however if it is not 
I had intended to request this anyhow.  When I looked into using 
infinispans's map reduce facility this is the the task I came up with:

http://pastebin.com/7GGjVnVt

I would prefer to specify it as:

http://pastebin.com/HTSq3g66

I'm pretty sure this is the not intended use case but it distributes the 
creation of my reports which is what I want. It's not really a big deal 
for me as I can get around this limitation by creating a wrapper class 
such as in the first example but it would be nice if I did not have 
to. Is this a reasonable request?

Also, and this is probably just wrong, when I use this I bundle up the 
task and execute it via JMS. Would it be reasonable to make Collator 
extend Serializable?

Brent


On 12-01-30 11:47 AM, Vladimir Blagojevic wrote:
> Guys,
>
> I was looking at this again recently and I still do not understand how 
> combiner could have different interface than Reducer! Hadoop forces a 
> user to implement combiner as a Reducer 
> http://developer.yahoo.com/hadoop/tutorial/module4.html#functionality 
> and 
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setCombinerClass%28java.lang.Class%29 
> In addition, the original paper does not mention any change of types.
>
> What we have admittedly done wrong is to apply Reducer on individual 
> Mapper without checking if a reduce function is both /commutative/ and 
> /associative/! This can lead to problems: 
> http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/
>
> So yes, I am all for adding Combiner (it should do the optional 
> reducer per mapper we do automatically now) but I do not see why we 
> have to change the interface!
>
>
> Regards,
> Vladimir
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20120131/e81987fa/attachment.html 


More information about the infinispan-dev mailing list