[infinispan-dev] Adding a Combiner
Vladimir Blagojevic
vblagoje at redhat.com
Tue Jan 31 14:45:06 EST 2012
Brent,
I see your point but this change would require quite a big retooling of
MapReduce API and we can not do that in 5.2 minor release. Lets leave
that for 6.0.
For 5.2 I want to add method MapReduceTask
combinedWith(Reducer<KOut,VOut> combiner). That way would properly deal
with /commutative/ and /associative /reducer and eliminate potential
problems we currently do not address:
http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/
Regards,
Vladimir
On 12-01-31 2:46 PM, Vladimir Blagojevic wrote:
> Response from Brent Douglas:
>
> Hi Vladimir,
>
> I'm not sure this is the same thing being discussed however if it is
> not I had intended to request this anyhow. When I looked into using
> infinispans's map reduce facility this is the the task I came up with:
>
> http://pastebin.com/7GGjVnVt
>
> I would prefer to specify it as:
>
> http://pastebin.com/HTSq3g66
>
> I'm pretty sure this is the not intended use case but it distributes
> the creation of my reports which is what I want. It's not really a big
> deal for me as I can get around this limitation by creating a wrapper
> class such as in the first example but it would be nice if I did not
> have to. Is this a reasonable request?
>
> Also, and this is probably just wrong, when I use this I bundle up the
> task and execute it via JMS. Would it be reasonable to make Collator
> extend Serializable?
>
> Brent
>
>
> On 12-01-30 11:47 AM, Vladimir Blagojevic wrote:
>> Guys,
>>
>> I was looking at this again recently and I still do not understand
>> how combiner could have different interface than Reducer! Hadoop
>> forces a user to implement combiner as a Reducer
>> http://developer.yahoo.com/hadoop/tutorial/module4.html#functionality
>> and
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Job.html#setCombinerClass%28java.lang.Class%29
>> In addition, the original paper does not mention any change of types.
>>
>> What we have admittedly done wrong is to apply Reducer on individual
>> Mapper without checking if a reduce function is both /commutative/
>> and /associative/! This can lead to problems:
>> http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/
>>
>> So yes, I am all for adding Combiner (it should do the optional
>> reducer per mapper we do automatically now) but I do not see why we
>> have to change the interface!
>>
>>
>> Regards,
>> Vladimir
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20120131/88eb0674/attachment.html
More information about the infinispan-dev
mailing list