Re: [infinispan-dev] distributed fork-join executor prototype

Hi Matt, there are subtle differences between map-reduce scenarios and task distribution: map-reduce works mostly on cached data (lots of reads with few writes), whereas task distribution produces mostly new data (lots of writes). If I misunderstood you, and you're actually doing map-reduce, then using Infinispan is fine. Comments inline ... On 3/4/13 5:09 AM, matt hoffman wrote: So if we use DIST, we have a write (insert) to N nodes, a local read, one or more writes (updates) and a final write (done), followed by a (potentially remote) read by the coordinator. I'm wondering whether the intermediate writes (updates) are needed, as they trigger messages to other nodes: if the intermediate steps are relatively small, then there's no need to back them up to other nodes, as others could simply restart the work from scratch if a node crashes. Most of the time, nodes don't crash, so the update-writes might not be amortized by crashes, unless it's a lot of work to recreate those intermediate steps. Anyway, yes, this is write-heavy, and there are a few disadvantages using DIST: - Unneeded writes for intermediate work - DIST reshuffles state on a view change. This means that even if a given worker doesn't crash, it might have to surrender its work to a different worker if the view changes in the middle of a task. Actually, your system can probably deal with this, but even if a worker continues processing a task during a view change, chances are its writes are now mostly remote, which is inefficient - While work stealing is a good thing in general, in Infinispan it leads to contention on the same locks, ie. the original worker and the worker which stole the task. While I'm a big proponent of Infinispan, there are cases where it's the wrong tool for the job, e.g. for messaging systems, notification buses and master-slave work distribution. Got it, using DIST. You could use a similar mechanism to send tasks to a subset, using a consistent hash over the task ID, for example. In a large system, this would certainly be better as the tasks are now not stored by everyone, but only by a small subset. My system used multicasts because it was a prototype and mcasts were simpler to implement. OK, makes senses. But this gets back to using the right tool. If I has given the task of implementing such a system, I'd intuitively had used JMS queues, or a peer-to-peer system like JGroups. Yes, in my design, a task is only rescheduled to a different node if the original worker node *died*, ie. was removed from the view, and not if the consistent hash over the task ID pointed to a different node. Agreed. But as I said above, this could be done by computing a hash over the task and sending the work to a subset of the cluster (with N unicasts) rather than a multicast. Or you could multicast the task and only the nodes which match the hash(task_id) would store the task... the former is probably more efficient, the latter simpler to implement. Although either case doesn't sound like it's overly complicated to implement. I wrote JBossCache (Infinispan's predecessor) and my design was geared towards read-heavy workloads. TBH, I just *assumed* but didn't really know whether this is still the case with Infinispan, perhaps someone from the Infinispan team can chime in ? Yes, see above NP, cheers, -- Bela Ban, JGroups lead (http://www.jgroups.org)