[infinispan-dev] New draft for distributed execution framework

[Manik Surtani]

Hi Trustin

Thanks for this.  A few comments:

* 1st section, by node affinity, I presume you mean data locality?
* 2nd section: data processing tasks would be sent out to specific nodes where required data is available.  Not to every node.
* Do we want to support this for replicated mode?  Typically, replicated clusters will be small and won't really benefit from task execution.  Maybe we add in support for replication later, if there is adequate demand?  It shouldn't change the API, etc.
* +1 re: multi-cores on each node.  I presume this would rely on an Executor set up on each node for this purpose?  And a user would have the opportunity to configure this executor (e.g., based on the type of hardware in use)?
* Could you define where each of these tasks are executed?  E.g., in a grid of {A, B, C, D} and you invoke your example on A, the local tasks happen on any node which contains the data it needs, while the global task happens on A?  Surely in many cases it makes sense to "reduce" before pushing results (final or intermediate) on the network?
* Non-Java language support - this should be out of scope for now, as it adds a layer of complexity we don't need.  We will certainly add support for map/reduce over Hot Rod in future, perhaps using JSON/JavaScript, but not for now.  :)
* Client/Server development: again, out of scope for now.  See above.  Once we have map/reduce over Hot Rod, client/server would be trivial.

Cheers
Manik


On 30 Nov 2010, at 09:12, Trustin Lee wrote:

> Ouch, here's the URL:
> 
> http://community.jboss.org/wiki/DistributedDataStreamProcessingFrameworkInInfinispan
> 
> On 11/30/2010 03:42 PM, Trustin Lee wrote:
>> Hi folks,
>> 
>> I posted the new draft for the distributed execution framework.  It's
>> written based on the discussion between Vladimir and me.  I initially
>> thought about publishing the API as a git branch, but I guess it's a
>> good idea to have a couple rounds before doing that.  I tried to be as
>> specific as possible, but please feel free to let me know what could be
>> improved, what is missing, and so on.
>> 
>> Cheers,
>> Trustin
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
manik at jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org