I think this is an important feature to have soon; My understanding of it: We default with the feature off, and newly discovered nodes are added/removed as usual. With a JMX operatable switch, one can disable this: If a remote node is joining the JGroups view, but rehash is off: it will be added to a to-be-installed view, but this won't be installed until rehash is enabled again. This gives time to add more changes before starting the rehash, and would help a lot to start larger clusters. If the [self] node is booting and joining a cluster with manual rehash off, the start process and any getCache() invocation should block and wait for it to be enabled. This would need of course to override the usually low timeouts. When a node is suspected it's a bit a different story as we need to make sure no data is lost. The principle is the same, but maybe we should have two flags: one which is a "soft request" to avoid rehashes of less than N members (and refuse N>=numOwners ?), one which is just disable it and don't care: data might be in a cachestore, data might not be important. Which reminds me, we should consider as well a JMX command to flush the container to the CacheLoader. --Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 31 January 2012 16:06, Bela Ban<bban(a)redhat.com> wrote: > This is essentially what I suggested at the Lisbon meeting, right ? Yes! > I think Dan had a design wiki on this somewhere... Just rising it here as it was moved to 6.0, while I think it deserves a dedicated thread to better think about it. If it's not hard, I think it should be done sooner. But while I started the thread to wake up the brilliant minds, I can't volunteer for this to make it happen. Sanne > > > On 1/31/12 4:53 PM, Sanne Grinovero wrote: >> I think this is an important feature to have soon; >> >> My understanding of it: >> >> We default with the feature off, and newly discovered nodes are >> added/removed as usual. With a JMX operatable switch, one can disable >> this: >> >> If a remote node is joining the JGroups view, but rehash is off: it >> will be added to a to-be-installed view, but this won't be installed >> until rehash is enabled again. This gives time to add more changes >> before starting the rehash, and would help a lot to start larger >> clusters. >> >> If the [self] node is booting and joining a cluster with manual rehash >> off, the start process and any getCache() invocation should block and >> wait for it to be enabled. This would need of course to override the >> usually low timeouts. >> >> When a node is suspected it's a bit a different story as we need to >> make sure no data is lost. The principle is the same, but maybe we >> should have two flags: one which is a "soft request" to avoid rehashes >> of less than N members (and refuse N>=numOwners ?), one which is just >> disable it and don't care: data might be in a cachestore, data might >> not be important. Which reminds me, we should consider as well a JMX >> command to flush the container to the CacheLoader. >> >> --Sanne >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Bela Ban > Lead JGroups (http://www.jgroups.org) > JBoss / Red Hat > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

...such as bringing up a backup data center. -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban Sent: Tuesday, January 31, 2012 11:18 AM To: infinispan-dev(a)lists.jboss.org Subject: Re: [infinispan-dev] Proposal: ISPN-1394 Manual rehashing in 5.2 I cannot volunteer either, but I find it important to be done in 5.2 ! Unless rehashing works flawlessly with a large number of nodes joining at the same time, I think manual rehashing is crucial... On 1/31/12 5:13 PM, Sanne Grinovero wrote: > On 31 January 2012 16:06, Bela Ban<bban(a)redhat.com>  wrote: >> This is essentially what I suggested at the Lisbon meeting, right ? > > Yes! > >> I think Dan had a design wiki on this somewhere... > > Just rising it here as it was moved to 6.0, while I think it deserves > a dedicated thread to better think about it. If it's not hard, I think > it should be done sooner. > But while I started the thread to wake up the brilliant minds, I can't > volunteer for this to make it happen. > > Sanne > >> >> >> On 1/31/12 4:53 PM, Sanne Grinovero wrote: >>> I think this is an important feature to have soon; >>> >>> My understanding of it: >>> >>> We default with the feature off, and newly discovered nodes are >>> added/removed as usual. With a JMX operatable switch, one can disable >>> this: >>> >>> If a remote node is joining the JGroups view, but rehash is off: it >>> will be added to a to-be-installed view, but this won't be installed >>> until rehash is enabled again. This gives time to add more changes >>> before starting the rehash, and would help a lot to start larger >>> clusters. >>> >>> If the [self] node is booting and joining a cluster with manual rehash >>> off, the start process and any getCache() invocation should block and >>> wait for it to be enabled. This would need of course to override the >>> usually low timeouts. >>> >>> When a node is suspected it's a bit a different story as we need to >>> make sure no data is lost. The principle is the same, but maybe we >>> should have two flags: one which is a "soft request" to avoid rehashes >>> of less than N members (and refuse N>=numOwners ?), one which is just >>> disable it and don't care: data might be in a cachestore, data might >>> not be important. Which reminds me, we should consider as well a JMX >>> command to flush the container to the CacheLoader. >>> >>> --Sanne >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Bela Ban >> Lead JGroups (http://www.jgroups.org) >> JBoss / Red Hat >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Bela Ban Lead JGroups (http://www.jgroups.org) JBoss / Red Hat _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 1 February 2012 11:23, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Bela, you're right, this is essentially what we talked about in Lisbon: > https://community.jboss.org/wiki/AsymmetricCachesAndManualRehashingDesign > > For joins I actually started working on a policy of coalescing joins > that happen one after the other in a short time interval. The current > implementation is very primitive, as I shifted focus to stability, but > it does coalesce joins 1 second after another join started (or while > that join is still running). > > I don't quite agree with Sanne's assessment that it's fine for > getCache() to block for 5 minutes until the administrator allows the > new node to join. We should modify startCaches() instead to signal to > the coordinator that we are ready to receive data for one or all of > the defined caches, and wait with a customizable time limit until the > caches have properly joined the cluster. > > The getCache() timeout should not be increased at all. Instead I would > propose that getCache() returns a functional cache immediately, even > if the cache didn't receive any data, and it works solely as an L1 > cache until the administrator allows it to join. I'd even make it > possible to designate a cache as an L1-only cache, so it's never an > owner for any key. I agree that would be very nice, but makes it much more complex to implement in 5.2 as well: functional L1 means that the other nodes must accept this node as part of the grid, including for L1 invalidation purposes.

So my proposal on blocking until ready is to make a first step, and I think it would still be very useful for people wanting to boot some ~100 nodes. Blocking the application is not a big deal, as you're delaying boot of an application which was likely not even powered on before. When adding several new nodes, I just want them to "add all at once", so preventing intermediate rehashing: until all have joined you should block rehash - that's a manual (or more likely automated externally) step and will not be engaged for long, nor it would replace normal behaviour when disabled. Actually even more useful would be to start a node with such an no-rehash option enabled.

> > > For leaves, the main problem is that every node has to compute the > same primary owner for a key, at all times. So we need a 2PC cache > view installation immediately after any leave to ensure that every > node determines the primary owner in the same way - we can't coalesce > or postpone leaves. > > For 5.2 I will try to decouple the cache view installation from the > state transfer, so in theory we will be able to coalesce/postpone the > state transfer for leaves as well > (https://issues.jboss.org/browse/ISPN-1827). I'm kind of need it for > non-blocking state transfer, because with the current implementation a > leave forces us to cancel any state transfer in progress and restart > with the updated cache view - a state transfer rollback will be very > expensive with NBST. > > > Erik does raise a valid point - with TACH, if we bring up a node with > a different siteId, then it will be an owner for all the keys in the > cache. That node probably isn't provisioned to hold all the keys, so > it would very likely run out of memory or evict much of the data. I > guess that makes it a 5.2 issue? That's exactly my reasoning as well. There are situations in which you want to add several nodes "once", which is very different than "in rapid sequence" as that would storm the network with data shuffling in all directions.

> > Shutting down a site should be possible even with what we have now - > just insert a DISCARD protocol in the JGroups stack of all the nodes > that are shutting down, and when FD finally times out on the nodes in > the surviving datacenter they won't have any state transfer to do > (although it may cause a few failed state transfer attempts). We could > make it simpler though. Yes, use a pair of scissors ;-) --Sanne > > > Cheers > Dan > > > On Tue, Jan 31, 2012 at 6:21 PM, Erik Salter <an1310(a)hotmail.com&gt; wrote: >> ...such as bringing up a backup data center. >> >> -----Original Message----- >> From: infinispan-dev-bounces(a)lists.jboss.org >> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban >> Sent: Tuesday, January 31, 2012 11:18 AM >> To: infinispan-dev(a)lists.jboss.org >> Subject: Re: [infinispan-dev] Proposal: ISPN-1394 Manual rehashing in 5.2 >> >> I cannot volunteer either, but I find it important to be done in 5.2 ! >> >> Unless rehashing works flawlessly with a large number of nodes joining >> at the same time, I think manual rehashing is crucial... >> >> >> >> On 1/31/12 5:13 PM, Sanne Grinovero wrote: >>> On 31 January 2012 16:06, Bela Ban&lt;bban(a)redhat.com&gt;  wrote: >>>> This is essentially what I suggested at the Lisbon meeting, right ? >>> >>> Yes! >>> >>>> I think Dan had a design wiki on this somewhere... >>> >>> Just rising it here as it was moved to 6.0, while I think it deserves >>> a dedicated thread to better think about it. If it's not hard, I think >>> it should be done sooner. >>> But while I started the thread to wake up the brilliant minds, I can't >>> volunteer for this to make it happen. >>> >>> Sanne >>> >>>> >>>> >>>> On 1/31/12 4:53 PM, Sanne Grinovero wrote: >>>>> I think this is an important feature to have soon; >>>>> >>>>> My understanding of it: >>>>> >>>>> We default with the feature off, and newly discovered nodes are >>>>> added/removed as usual. With a JMX operatable switch, one can disable >>>>> this: >>>>> >>>>> If a remote node is joining the JGroups view, but rehash is off: it >>>>> will be added to a to-be-installed view, but this won't be installed >>>>> until rehash is enabled again. This gives time to add more changes >>>>> before starting the rehash, and would help a lot to start larger >>>>> clusters. >>>>> >>>>> If the [self] node is booting and joining a cluster with manual rehash >>>>> off, the start process and any getCache() invocation should block and >>>>> wait for it to be enabled. This would need of course to override the >>>>> usually low timeouts. >>>>> >>>>> When a node is suspected it's a bit a different story as we need to >>>>> make sure no data is lost. The principle is the same, but maybe we >>>>> should have two flags: one which is a "soft request" to avoid rehashes >>>>> of less than N members (and refuse N>=numOwners ?), one which is just >>>>> disable it and don't care: data might be in a cachestore, data might >>>>> not be important. Which reminds me, we should consider as well a JMX >>>>> command to flush the container to the CacheLoader. >>>>> >>>>> --Sanne >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Bela Ban >>>> Lead JGroups (http://www.jgroups.org) >>>> JBoss / Red Hat >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Bela Ban >> Lead JGroups (http://www.jgroups.org) >> JBoss / Red Hat >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> > The getCache() timeout should not be increased at all. Instead I would > propose that getCache() returns a functional cache immediately, even > if the cache didn't receive any data, and it works solely as an L1 > cache until the administrator allows it to join. I'd even make it > possible to designate a cache as an L1-only cache, so it's never an > owner for any key. I agree that would be very nice, but makes it much more complex to implement in 5.2 as well: functional L1 means that the other nodes must accept this node as part of the grid, including for L1 invalidation purposes. So my proposal on blocking until ready is to make a first step, and I think it would still be very useful for people wanting to boot some ~100 nodes. Blocking the application is not a big deal, as you're delaying boot of an application which was likely not even powered on before. When adding several new nodes, I just want them to "add all at once", so preventing intermediate rehashing: until all have joined you should block rehash - that's a manual (or more likely automated externally) step and will not be engaged for long, nor it would replace normal behaviour when disabled.

On 1 Feb 2012, at 12:23, Dan Berindei wrote: > Bela, you're right, this is essentially what we talked about in Lisbon: > https://community.jboss.org/wiki/AsymmetricCachesAndManualRehashingDesign > > For joins I actually started working on a policy of coalescing joins > that happen one after the other in a short time interval. The current > implementation is very primitive, as I shifted focus to stability, but > it does coalesce joins 1 second after another join started (or while > that join is still running). > > I don't quite agree with Sanne's assessment that it's fine for > getCache() to block for 5 minutes until the administrator allows the > new node to join. We should modify startCaches() instead to signal to > the coordinator that we are ready to receive data for one or all of > the defined caches, and wait with a customizable time limit until the > caches have properly joined the cluster. > > The getCache() timeout should not be increased at all. Instead I would > propose that getCache() returns a functional cache immediately, even > if the cache didn't receive any data, and it works solely as an L1 > cache until the administrator allows it to join. I'd even make it > possible to designate a cache as an L1-only cache, so it's never an > owner for any key. I presume this would be encoded in the Address?  That would make sense for a node permanently designated as an L1 node.  But then how would this work for a node temporarily acting as L1 only, until it has been allowed to join?  Change the Address instance on the fly?  A delegating Address?  :/

> For leaves, the main problem is that every node has to compute the > same primary owner for a key, at all times. So we need a 2PC cache > view installation immediately after any leave to ensure that every > node determines the primary owner in the same way - we can't coalesce > or postpone leaves. Yes, manual rehashing would probably just be for joins.  Controlled shutdown in itself is manual, and crashes, well, need to be dealt with immediately IMO.

> > For 5.2 I will try to decouple the cache view installation from the > state transfer, so in theory we will be able to coalesce/postpone the > state transfer for leaves as well > (https://issues.jboss.org/browse/ISPN-1827). I'm kind of need it for > non-blocking state transfer, because with the current implementation a > leave forces us to cancel any state transfer in progress and restart > with the updated cache view - a state transfer rollback will be very > expensive with NBST. > > > Erik does raise a valid point - with TACH, if we bring up a node with > a different siteId, then it will be an owner for all the keys in the > cache. That node probably isn't provisioned to hold all the keys, so > it would very likely run out of memory or evict much of the data. I > guess that makes it a 5.2 issue? Yes. > Shutting down a site should be possible even with what we have now - > just insert a DISCARD protocol in the JGroups stack of all the nodes > that are shutting down, and when FD finally times out on the nodes in > the surviving datacenter they won't have any state transfer to do > (although it may cause a few failed state transfer attempts). We could > make it simpler though. > > > Cheers > Dan > > > On Tue, Jan 31, 2012 at 6:21 PM, Erik Salter <an1310(a)hotmail.com&gt; wrote: >> ...such as bringing up a backup data center. >> >> -----Original Message----- >> From: infinispan-dev-bounces(a)lists.jboss.org >> [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Bela Ban >> Sent: Tuesday, January 31, 2012 11:18 AM >> To: infinispan-dev(a)lists.jboss.org >> Subject: Re: [infinispan-dev] Proposal: ISPN-1394 Manual rehashing in 5.2 >> >> I cannot volunteer either, but I find it important to be done in 5.2 ! >> >> Unless rehashing works flawlessly with a large number of nodes joining >> at the same time, I think manual rehashing is crucial... >> >> >> >> On 1/31/12 5:13 PM, Sanne Grinovero wrote: >>> On 31 January 2012 16:06, Bela Ban&lt;bban(a)redhat.com&gt;  wrote: >>>> This is essentially what I suggested at the Lisbon meeting, right ? >>> >>> Yes! >>> >>>> I think Dan had a design wiki on this somewhere... >>> >>> Just rising it here as it was moved to 6.0, while I think it deserves >>> a dedicated thread to better think about it. If it's not hard, I think >>> it should be done sooner. >>> But while I started the thread to wake up the brilliant minds, I can't >>> volunteer for this to make it happen. >>> >>> Sanne >>> >>>> >>>> >>>> On 1/31/12 4:53 PM, Sanne Grinovero wrote: >>>>> I think this is an important feature to have soon; >>>>> >>>>> My understanding of it: >>>>> >>>>> We default with the feature off, and newly discovered nodes are >>>>> added/removed as usual. With a JMX operatable switch, one can disable >>>>> this: >>>>> >>>>> If a remote node is joining the JGroups view, but rehash is off: it >>>>> will be added to a to-be-installed view, but this won't be installed >>>>> until rehash is enabled again. This gives time to add more changes >>>>> before starting the rehash, and would help a lot to start larger >>>>> clusters. >>>>> >>>>> If the [self] node is booting and joining a cluster with manual rehash >>>>> off, the start process and any getCache() invocation should block and >>>>> wait for it to be enabled. This would need of course to override the >>>>> usually low timeouts. >>>>> >>>>> When a node is suspected it's a bit a different story as we need to >>>>> make sure no data is lost. The principle is the same, but maybe we >>>>> should have two flags: one which is a "soft request" to avoid rehashes >>>>> of less than N members (and refuse N>=numOwners ?), one which is just >>>>> disable it and don't care: data might be in a cachestore, data might >>>>> not be important. Which reminds me, we should consider as well a JMX >>>>> command to flush the container to the CacheLoader. >>>>> >>>>> --Sanne >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Bela Ban >>>> Lead JGroups (http://www.jgroups.org) >>>> JBoss / Red Hat >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Bela Ban >> Lead JGroups (http://www.jgroups.org) >> JBoss / Red Hat >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Lead, Infinispan http://www.infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 5 Feb 2012, at 16:24, Dan Berindei wrote: > On Sat, Feb 4, 2012 at 4:49 PM, Manik Surtani <manik(a)jboss.org&gt; wrote: >> >> On 1 Feb 2012, at 12:23, Dan Berindei wrote: >> >>> Bela, you're right, this is essentially what we talked about in Lisbon: >>> https://community.jboss.org/wiki/AsymmetricCachesAndManualRehashingDesign >>> >>> For joins I actually started working on a policy of coalescing joins >>> that happen one after the other in a short time interval. The current >>> implementation is very primitive, as I shifted focus to stability, but >>> it does coalesce joins 1 second after another join started (or while >>> that join is still running). >>> >>> I don't quite agree with Sanne's assessment that it's fine for >>> getCache() to block for 5 minutes until the administrator allows the >>> new node to join. We should modify startCaches() instead to signal to >>> the coordinator that we are ready to receive data for one or all of >>> the defined caches, and wait with a customizable time limit until the >>> caches have properly joined the cluster. >>> >>> The getCache() timeout should not be increased at all. Instead I would >>> propose that getCache() returns a functional cache immediately, even >>> if the cache didn't receive any data, and it works solely as an L1 >>> cache until the administrator allows it to join. I'd even make it >>> possible to designate a cache as an L1-only cache, so it's never an >>> owner for any key. >> >> I presume this would be encoded in the Address?  That would make sense for a node permanently designated as an L1 node.  But then how would this work for a node temporarily acting as L1 only, until it has been allowed to join?  Change the Address instance on the fly?  A delegating Address?  :/ >> > > Nope, not in the Address. Since we now have virtual cache views, every > node has to explicitly request to join each cache. It would be quite > easy to add an "L1-only" flag to the join request, and nodes with that > flag would never be included in the proper cache view. They would > however get a copy of the cache view on join and on every cache view > installation so they could update their CH and send requests to the > proper owners. > > Nodes acting temporarily as L1-only would send a normal join request, > but they would also receive a copy of the cache view and getCache() > would return immediately instead of waiting for the node to receive > state. When the coordinator finally installs a cache view that > includes them, they will perform the initial state transfer as they do > now. Makes sense. > > >>> For leaves, the main problem is that every node has to compute the >>> same primary owner for a key, at all times. So we need a 2PC cache >>> view installation immediately after any leave to ensure that every >>> node determines the primary owner in the same way - we can't coalesce >>> or postpone leaves. >> >> Yes, manual rehashing would probably just be for joins.  Controlled shutdown in itself is manual, and crashes, well, need to be dealt with immediately IMO. >> > > We could extend the policy for craches as well, by adding a > minNumOwners setting and only triggering an automatic rehash when > there is a segment on the hash wheel with <= minNumOwners owners. > > We would have a properly installed CH, that guarantees at some point > in the past each key had numOwners owners, and a filter on top of it > that removes any leavers from the result of DM.locate(), > DM.getPrimaryLocation() etc. > > It would probably undo our recent optimizations around locate and > getPrimaryLocation, so it's slowing the normal case (without any > leavers) in order to make the exceptional case (organized shutdown or > a part of the cluster) faster. The question is how big the cluster has > to get before the exceptional case becomes common enough that it's > worth optimizing for… Well, we could just leave the crashed node in the owners list.  When any caller attempts to do something with this list, i.e., perform an RPC, don't we check again at that time and remove leavers from an RPC recipient list?

> We could extend the policy for craches as well, by adding a > minNumOwners setting and only triggering an automatic rehash when > there is a segment on the hash wheel with <= minNumOwners owners. minNumOwners should be at least 1, which in the default use case (numOwners==2) would mean that we react to every leave. Whilst this functinality is very nice, I don't think it should have a high prio as most of the use cases I'm aware of use numOwners=2.

> > We would have a properly installed CH, that guarantees at some point > in the past each key had numOwners owners, and a filter on top of it > that removes any leavers from the result of DM.locate(), > DM.getPrimaryLocation() etc. > > It would probably undo our recent optimizations around locate and > getPrimaryLocation, so it's slowing the normal case (without any > leavers) in order to make the exceptional case (organized shutdown or > a part of the cluster) faster. > The question is how big the cluster > has > to get before the exceptional case becomes common enough that it's > worth optimizing for... re:partial shutdown, due to consistency constraints, it won't be posible to controlled shutdow more than numOwners-1 nodes at any time, so not sure it this optimisation has a broad scope.

For total shutdown, I guess we can use other means that rehash, e.g. a specific command that would disable it and start flushing to the store.

> numOwners==2 is and will very likely remain the most common case, > particularly for small clusters. > > But if we have two sites, it makes sense to configure 2 owners per > site. If only one node goes down, the surviving owner will supply > state to the new owner. If both nodes go down, the new owners will > fetch the data from the other site. So while 2 nodes going down will > be quite costly, it should be infrequent enough that it's worth > optimizing for the more frequent "1 node goes down and than comes > back > up" case. > Agreed; this mixed batching (leaves with joins) makes sense for non-site clusters as well.

> > For total shutdown, I guess we can use other means that rehash, > > e.g. a specific command that would disable it and start flushing > > to the store. > > > > I think just stopping the cache is enough to get it to flush data to > the store with passivation enabled. ATM, wouldn't the shutdown of a cluster of servers trigger a rehash storm?

> But for now any data saved to a > private store in distributed mode is useless after restart, because > we > have no safe way to push data that we don't own to other nodes (and > by > safe I mean avoiding overwriting newer data or resurrecting deleted > data). I think that should work with a clustered cache store though.