[infinispan-dev] Map/Reduce or other batch processing on CacheLoader stored entries

[Sanne Grinovero]

Tristan,

On 10 May 2012 06:43, Tristan Tarrant <ttarrant at redhat.com> wrote:
> On 05/09/2012 12:10 PM, Sanne Grinovero wrote:
>> 3) I need the Map/Reduce to execute also on all entries stored in the
>> CacheLoader entries. I don't believe that's the case today.. and even
>> if I wanted to use just the Executor, I believe the CacheLoader API
>> needs to provide some option to load all keys in a stream form
>> appropriate for batch processing: using loadAll will likely get me
>> into an OOM.
>>
> Even a simple loadAll(Callable<V>) would do the trick

I'm not understanding this. Could you elaborate a bit?

I was more thinking of a

loadAllInto(Consumer engine);

where Consumer has

#processEntry( entry )
or
#processEntry( key, value )
or maybe
#processEntry( List<entry> ) if it turns out we should better load
things in batches.

This might suggest
loadAllInto(Consumer engine, int suggestedBatchSize);

I would also expect Consumer to be threadsafe, so that multiple
batches or entries can be pushed to it concurrently, and be optionally
blocking so that the Consumer implementation can defend itself by
providing some back pressure.

thanks,
Sanne

>
> Tristan
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev