[infinispan-dev] asynchronous change manager for high load Lucene

[Manik Surtani]

On 30 Mar 2010, at 14:09, Bela Ban wrote:

> 
> 
> Sanne Grinovero wrote:
>>>> You could use the cluster view, which is the same across all nodes, 
>>>> and pick the first element in the list. Or you could run an 
>>>> agreement protocol, which deterministically elects a master.
>>> Looks simple, deterministic and elegant.
>> Bela Ban wrote:
>> 
>> Agreed, that looks like a clean solution, but would scale better if we 
>> could decide a master for each different index, to spread the load 
>> instead of choosing always the same node for all the work, ideas on that?
>> I could hash the indexname (the identifier) and % cluster view list 
>> size, WDYT ?
> 
> Absolutely, this makes a lot of sense. However, this will require moving 
> indices around when the cluster view changes.
> 
> With a simple modulo based algorithm you'll have a lot of rebalancing 
> going on, so maybe something like a consistent hash could determine the 
> server or servers on which a given index is stored. You could even store 
> an index on more than 1 nodes, similar to what Infinispan does with DIST 
> and numOwners=2 (for example).
> 
> Even better: *if* an index can be recreated from the real data, you 
> could use a consistent hash and only store an index *once*, and you 
> could use a Cache instance for it !

Yes, this would be the plan.  The indexes themselves are *read* by every node in the cluster and hence is stored in the cache.  It is just the updates that will be processed by a selected processor singleton.

--
Manik Surtani
manik at jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org