[infinispan-dev] Adding a Combiner

[Vladimir Blagojevic]

Brent,

I see your point but this change would require quite a big retooling of 
MapReduce API and we can not do that in 5.2 minor release. Lets leave 
that for 6.0.

For 5.2 I want to add method MapReduceTask 
combinedWith(Reducer<KOut,VOut> combiner). That way would properly deal 
with /commutative/ and /associative /reducer and eliminate potential 
problems we currently do not address: 
http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/ 


Regards,
Vladimir

On 12-01-31 2:46 PM, Vladimir Blagojevic wrote:
> Response from Brent Douglas:
>
> Hi Vladimir,
>
> I'm not sure this is the same thing being discussed however if it is 
> not I had intended to request this anyhow.  When I looked into using 
> infinispans's map reduce facility this is the the task I came up with:
>
> http://pastebin.com/7GGjVnVt
>
> I would prefer to specify it as:
>
> http://pastebin.com/HTSq3g66
>
> I'm pretty sure this is the not intended use case but it distributes 
> the creation of my reports which is what I want. It's not really a big 
> deal for me as I can get around this limitation by creating a wrapper 
> class such as in the first example but it would be nice if I did not 
> have to. Is this a reasonable request?
>
> Also, and this is probably just wrong, when I use this I bundle up the 
> task and execute it via JMS. Would it be reasonable to make Collator 
> extend Serializable?
>
> Brent
>
>
> On 12-01-30 11:47 AM, Vladimir Blagojevic wrote:
>> Guys,
>>
>> I was looking at this again recently and I still do not understand 
>> how combiner could have different interface than Reducer! Hadoop 
>> forces a user to implement combiner as a Reducer 
>> http://developer.yahoo.com/hadoop/tutorial/module4.html#functionality 
>> and 
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setCombinerClass%28java.lang.Class%29 
>> In addition, the original paper does not mention any change of types.
>>
>> What we have admittedly done wrong is to apply Reducer on individual 
>> Mapper without checking if a reduce function is both /commutative/ 
>> and /associative/! This can lead to problems: 
>> http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/
>>
>> So yes, I am all for adding Combiner (it should do the optional 
>> reducer per mapper we do automatically now) but I do not see why we 
>> have to change the interface!
>>
>>
>> Regards,
>> Vladimir
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20120131/88eb0674/attachment.html