[infinispan-dev] Adding a Combiner

[Galder Zamarreño]

LOL! This is what I got when accessing those pastebins…


On Jan 31, 2012, at 6:46 PM, Vladimir Blagojevic wrote:

> Response from Brent Douglas:
> 
> Hi Vladimir,
> 
> I'm not sure this is the same thing being discussed however if it is not I had intended to request this anyhow.  When I looked into using infinispans's map reduce facility this is the the task I came up with:
> 
> http://pastebin.com/7GGjVnVt
> 
> I would prefer to specify it as:
> 
> http://pastebin.com/HTSq3g66
> 
> I'm pretty sure this is the not intended use case but it distributes the creation of my reports which is what I want. It's not really a big deal for me as I can get around this limitation by creating a wrapper class such as in the first example but it would be nice if I did not have to. Is this a reasonable request?
> 
> Also, and this is probably just wrong, when I use this I bundle up the task and execute it via JMS. Would it be reasonable to make Collator extend Serializable?
> 
> Brent
> 
> 
> On 12-01-30 11:47 AM, Vladimir Blagojevic wrote:
>> Guys,
>> 
>> I was looking at this again recently and I still do not understand how combiner could have different interface than Reducer! Hadoop forces a user to implement combiner as a Reducer http://developer.yahoo.com/hadoop/tutorial/module4.html#functionality and http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setCombinerClass%28java.lang.Class%29 In addition, the original paper does not mention any change of types.
>> 
>> What we have admittedly done wrong is to apply Reducer on individual Mapper without checking if a reduce function is both commutative and associative! This can lead to problems: http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/
>> 
>> So yes, I am all for adding Combiner (it should do the optional reducer per mapper we do automatically now) but I do not see why we have to change the interface!
>> 
>> 
>> Regards,
>> Vladimir
>> 
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> 
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20120201/6a77bd38/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2012-02-01 at 10.28.58 AM.png
Type: image/png
Size: 453712 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/infinispan-dev/attachments/20120201/6a77bd38/attachment-0001.png