[infinispan-dev] Map/Reduce or other batch processing on CacheLoader stored entries

[Manik Surtani]

Yes, as a one-off, but there should be a mechanism to set up internal structures and clean up/send finalisation messages to Hibernate Search or completion RPCs, etc.

On 25 May 2012, at 11:31, Sanne Grinovero wrote:

> On 25 May 2012 10:57, Manik Surtani <manik at jboss.org> wrote:
>> #processEntriesWith(Processor p)
>> 
>> Processor extends Lifecycle { // Lifecycle for start() and stop() methods…
>>   void process(CacheEntry e);
>>   void process(Collection<CacheEntry> e);
>>   boolean processMoreEntries();
>> }
> 
> why the LifeCycle start/stop ?
> I expect to use it as a one-off, not as something which is
> "permanently hooked": looks like you' re thinking about a different
> problem?
> 
> The use case I'm thinkin of is when we need to iterate on all entries
> in the cachestore, such as :
> - Map/Reduce
>  - evaluating the average value of some attribute
>  - word counting
> - MassIndexer
> 
> In all the use cases I'm having in mind, you want to process all
> entries, and only once.
> So #processMoreEntries would be redundant, and I think we should
> choose just one between CacheEntry or Collection<CacheEntry>.. let's
> go with the simple CacheEntry ?
> Should be able to avoid creation of short lived collections, and when
> passing collections one would likely need to iterate on each element
> anyway to route the invocation so some internal process(CacheEntry e);
> 
> -- Sanne
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org