Guys We've discussed ISPN-3140 elsewhere before, I'm brining it to this forum now. https://issues.jboss.org/browse/ISPN-3140 Any thoughts/concerns? Particularly looking to hear from Dan or Adrian about viability, complexity, ease of implementation. Thanks Manik -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

ISPN-3140 only mentions suspending state transfer. The partial cluster shutdown scenario is part of ISPN-1394. Doesn't it make sense to merge them?

Cheers Adi On 05/31/2013 03:52 PM, Dan Berindei wrote: > If we only want to deal with full cluster shutdown, then I think stopping all application requests, calling Cache.clear() on one node, and then shutting down all the nodes should be simpler. On start, assuming no cache store, the caches will start empty, so starting all the nodes at once and only allowing application requests when they've all joined should also work without extra work. > > If we only want to stop a part of the cluster, suppressing rebalancing would be better, because we wouldn't lose all the data. But we'd still lose the keys whose owners are all among the nodes we want to stop. I've discussed this with Adrian, and we think if we want to stop a part of the cluster without losing data we need a JMX operation on the coordinator that will "atomically" remove a set of nodes from the CH. After the operation completes, the user will know it's safe to stop those nodes without losing data. > > When it comes to starting a part of the cluster, a "pause rebalancing" option would probably be better - but again, on the coordinator, not on each joining node. And clearly, if more than numOwner nodes leave while rebalancing is suspended, data will be lost. > > Cheers > Dan > > > > On Fri, May 31, 2013 at 12:17 PM, Manik Surtani <msurtani(a)redhat.com&gt; wrote: > Guys > > We've discussed ISPN-3140 elsewhere before, I'm brining it to this forum now. > > https://issues.jboss.org/browse/ISPN-3140 > > Any thoughts/concerns? Particularly looking to hear from Dan or Adrian about viability, complexity, ease of implementation. > > Thanks > Manik > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Platform Architect, JBoss Data Grid > http://red.ht/data-grid > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 31 May 2013, at 13:52, Dan Berindei <dan.berindei(a)gmail.com <mailto:dan.berindei@gmail.com>> wrote: > If we only want to deal with full cluster shutdown, then I think > stopping all application requests, calling Cache.clear() on one node, > and then shutting down all the nodes should be simpler. On start, > assuming no cache store, the caches will start empty, so starting all > the nodes at once and only allowing application requests when they've > all joined should also work without extra work. > > If we only want to stop a part of the cluster, suppressing > rebalancing would be better, because we wouldn't lose all the data. > But we'd still lose the keys whose owners are all among the nodes we > want to stop. I've discussed this with Adrian, and we think if we > want to stop a part of the cluster without losing data we need a JMX > operation on the coordinator that will "atomically" remove a set of > nodes from the CH. After the operation completes, the user will know > it's safe to stop those nodes without losing data. I think the no-data-loss option is bigger scope, perhaps part of ISPN-1394. And that's not what I am asking about. > When it comes to starting a part of the cluster, a "pause > rebalancing" option would probably be better - but again, on the > coordinator, not on each joining node. And clearly, if more than > numOwner nodes leave while rebalancing is suspended, data will be lost. Yup. This sort of option would only be used where data loss isn't an issue (such as a distributed cache). Where data loss is an issue, we'd need more control - ISPN-1394. > > Cheers > Dan > > > > On Fri, May 31, 2013 at 12:17 PM, Manik Surtani <msurtani(a)redhat.com > <mailto:msurtani@redhat.com>> wrote: > > Guys > > We've discussed ISPN-3140 elsewhere before, I'm brining it to > this forum now. > > https://issues.jboss.org/browse/ISPN-3140 > > Any thoughts/concerns? Particularly looking to hear from Dan or > Adrian about viability, complexity, ease of implementation. > > Thanks > Manik > -- > Manik Surtani > manik(a)jboss.org <mailto:manik@jboss.org> > twitter.com/maniksurtani <http://twitter.com/maniksurtani> > > Platform Architect, JBoss Data Grid > http://red.ht/data-grid > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > <mailto:infinispan-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org <mailto:manik@jboss.org> twitter.com/maniksurtani <http://twitter.com/maniksurtani> Platform Architect, JBoss Data Grid http://red.ht/data-grid _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I see 2 potential issues: 1. How does the user know which node is the master to connect to, since the operations are a no-op on all the others? - instead of a no-op, what if the other nodes just forward the operation to the correct node? Then the user doesn't have to know who the current master is, and can just connect to any node. 2. What if the current master is one of the nodes being stopped? -Dennis On 05/31/2013 11:40 AM, Adrian Nistor wrote: > Yes, ISPN-1394 has a broader scope but the proposed solution for > ISPN-3140 solves quite a lot of ISPN-1394 and it's not complex. We > might not even need ISPN-1394 soon unless somebody really wants to > control data ownership down to segment granularity. If we only want > to batch joins/leaves and manually kick out nodes with or without > loosing their data then this proposal should be enough. This solution > should not prevent implementation of ISPN-1394 in future and will not > need to be removed/undone. > > Here are the details: > > 1. /Add a JMX writable attribute (or operation?) to > ClusterTopologyManager (name it suppressRehashing?) that is false by > default but should also be configurable via API or xml. While this > attribute is true the ClusterTopologyManager queues all > join/leave/exclude(see below) requests and does not execute them on > the spot as it would normally happen. The value of this attribute is > ignored on all nodes but the coordinator. When it is set back to > false all queued operations (except the ones that cancel eachother > out) are executed. The setter should be synchronous so when setting > is back to false it does not return until the queue is empty and all > rehashing was processed. / > > 2. /We add a JMX operation excludeNodes(list of addresses) to > ClusterTopologyManager. Calling this method on any node but the > coordinator is no-op. This operation removes the node from the > topology (almost as if it left) and forces a rebalance./ The node is > still present in the current CH but not in the pending CH. It's > basically disowned by all its data which is now being transferred to > other (not excluded) nodes. At the end of the rebalance the node is > removed from topology for good and can be shut down without loosing > data. Note that if suppressRehashing==false operation > excludeNodes(..) just queues them for later removal. We can batch > multiple such exclusions and then re-activate the rehashing. > > The parts that need to be implemented are written in italic above. > Everything else is already there. > > excludeNodes is a way of achieving a soft shutdown and should be used > only if we care about preserving data int the extreme case where the > nodes are the last/single owners. We can just kill the node directly > if we do not care about its data. > > suppressRehashing is a way of achieving some kind of batching of > topology changes. This should speed up state transfer a lot because > it avoids a lot of pointless reshuffling of data segments when we > have many successive joiners/leavers. > > So what happens if the current coordinator dies for whatever reason? > The new one will take control and will not have knowledge of the > existing rehash queue or the previous status of suppressRehashing > attribute so it will just get the current cache membership status > from all members of current view and proceed with the rehashing as > usual. If the user does not want this he can set a default value of > true for suppressRehashing. The admin has to interact now via JMX > with the new coordinator. But that's not as bad as the alternative > where all the nodes are involved in this jmx scheme :) I think having > only the coordinator involved in this is a plus. > > Manik, how does this fit for the full and partial shutdown? > > Cheers > Adi > > > On 05/31/2013 04:20 PM, Manik Surtani wrote: >> >> On 31 May 2013, at 13:52, Dan Berindei <dan.berindei(a)gmail.com >> <mailto:dan.berindei@gmail.com>> wrote: >> >>> If we only want to deal with full cluster shutdown, then I think >>> stopping all application requests, calling Cache.clear() on one >>> node, and then shutting down all the nodes should be simpler. On >>> start, assuming no cache store, the caches will start empty, so >>> starting all the nodes at once and only allowing application >>> requests when they've all joined should also work without extra work. >>> >>> If we only want to stop a part of the cluster, suppressing >>> rebalancing would be better, because we wouldn't lose all the data. >>> But we'd still lose the keys whose owners are all among the nodes >>> we want to stop. I've discussed this with Adrian, and we think if >>> we want to stop a part of the cluster without losing data we need a >>> JMX operation on the coordinator that will "atomically" remove a >>> set of nodes from the CH. After the operation completes, the user >>> will know it's safe to stop those nodes without losing data. >> >> I think the no-data-loss option is bigger scope, perhaps part of >> ISPN-1394. And that's not what I am asking about. >> >>> When it comes to starting a part of the cluster, a "pause >>> rebalancing" option would probably be better - but again, on the >>> coordinator, not on each joining node. And clearly, if more than >>> numOwner nodes leave while rebalancing is suspended, data will be lost. >> >> Yup. This sort of option would only be used where data loss isn't >> an issue (such as a distributed cache). Where data loss is an >> issue, we'd need more control - ISPN-1394. >> >>> >>> Cheers >>> Dan >>> >>> >>> >>> On Fri, May 31, 2013 at 12:17 PM, Manik Surtani >>> <msurtani(a)redhat.com <mailto:msurtani@redhat.com>> wrote: >>> >>> Guys >>> >>> We've discussed ISPN-3140 elsewhere before, I'm brining it to >>> this forum now. >>> >>> https://issues.jboss.org/browse/ISPN-3140 >>> >>> Any thoughts/concerns? Particularly looking to hear from Dan >>> or Adrian about viability, complexity, ease of implementation. >>> >>> Thanks >>> Manik >>> -- >>> Manik Surtani >>> manik(a)jboss.org <mailto:manik@jboss.org> >>> twitter.com/maniksurtani <http://twitter.com/maniksurtani> >>> >>> Platform Architect, JBoss Data Grid >>> http://red.ht/data-grid >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> <mailto:infinispan-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org <mailto:manik@jboss.org> >> twitter.com/maniksurtani <http://twitter.com/maniksurtani> >> >> Platform Architect, JBoss Data Grid >> http://red.ht/data-grid >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Yes, ISPN-1394 has a broader scope but the proposed solution for ISPN-3140 solves quite a lot of ISPN-1394 and it's not complex. We might not even need ISPN-1394 soon unless somebody really wants to control data ownership down to segment granularity. If we only want to batch joins/leaves and manually kick out nodes with or without loosing their data then this proposal should be enough. This solution should not prevent implementation of ISPN-1394 in future and will not need to be removed/undone. Here are the details: 1. Add a JMX writable attribute (or operation?) to ClusterTopologyManager (name it suppressRehashing?) that is false by default but should also be configurable via API or xml. While this attribute is true the ClusterTopologyManager queues all join/leave/exclude(see below) requests and does not execute them on the spot as it would normally happen. The value of this attribute is ignored on all nodes but the coordinator. When it is set back to false all queued operations (except the ones that cancel eachother out) are executed. The setter should be synchronous so when setting is back to false it does not return until the queue is empty and all rehashing was processed. 2. We add a JMX operation excludeNodes(list of addresses) to ClusterTopologyManager. Calling this method on any node but the coordinator is no-op. This operation removes the node from the topology (almost as if it left) and forces a rebalance. The node is still present in the current CH but not in the pending CH. It's basically disowned by all its data which is now being transferred to other (not excluded) nodes. At the end of the rebalance the node is removed from topology for good and can be shut down without loosing data. Note that if suppressRehashing==false operation excludeNodes(..) just queues them for later removal. We can batch multiple such exclusions and then re-activate the rehashing. The parts that need to be implemented are written in italic above. Everything else is already there. excludeNodes is a way of achieving a soft shutdown and should be used only if we care about preserving data int the extreme case where the nodes are the last/single owners. We can just kill the node directly if we do not care about its data. suppressRehashing is a way of achieving some kind of batching of topology changes. This should speed up state transfer a lot because it avoids a lot of pointless reshuffling of data segments when we have many successive joiners/leavers. So what happens if the current coordinator dies for whatever reason? The new one will take control and will not have knowledge of the existing rehash queue or the previous status of suppressRehashing attribute so it will just get the current cache membership status from all members of current view and proceed with the rehashing as usual. If the user does not want this he can set a default value of true for suppressRehashing. The admin has to interact now via JMX with the new coordinator. But that's not as bad as the alternative where all the nodes are involved in this jmx scheme :) I think having only the coordinator involved in this is a plus. Manik, how does this fit for the full and partial shutdown? Cheers Adi On 05/31/2013 04:20 PM, Manik Surtani wrote: > > On 31 May 2013, at 13:52, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > >> If we only want to deal with full cluster shutdown, then I think stopping all application requests, calling Cache.clear() on one node, and then shutting down all the nodes should be simpler. On start, assuming no cache store, the caches will start empty, so starting all the nodes at once and only allowing application requests when they've all joined should also work without extra work. >> >> If we only want to stop a part of the cluster, suppressing rebalancing would be better, because we wouldn't lose all the data. But we'd still lose the keys whose owners are all among the nodes we want to stop. I've discussed this with Adrian, and we think if we want to stop a part of the cluster without losing data we need a JMX operation on the coordinator that will "atomically" remove a set of nodes from the CH. After the operation completes, the user will know it's safe to stop those nodes without losing data. > > I think the no-data-loss option is bigger scope, perhaps part of ISPN-1394. And that's not what I am asking about. > >> When it comes to starting a part of the cluster, a "pause rebalancing" option would probably be better - but again, on the coordinator, not on each joining node. And clearly, if more than numOwner nodes leave while rebalancing is suspended, data will be lost. > > Yup. This sort of option would only be used where data loss isn't an issue (such as a distributed cache). Where data loss is an issue, we'd need more control - ISPN-1394. > >> >> Cheers >> Dan >> >> >> >> On Fri, May 31, 2013 at 12:17 PM, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >> Guys >> >> We've discussed ISPN-3140 elsewhere before, I'm brining it to this forum now. >> >> https://issues.jboss.org/browse/ISPN-3140 >> >> Any thoughts/concerns? Particularly looking to hear from Dan or Adrian about viability, complexity, ease of implementation. >> >> Thanks >> Manik >> -- >> Manik Surtani >> manik(a)jboss.org >> twitter.com/maniksurtani >> >> Platform Architect, JBoss Data Grid >> http://red.ht/data-grid >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > twitter.com/maniksurtani > > Platform Architect, JBoss Data Grid > http://red.ht/data-grid > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> If we only want to deal with full cluster shutdown, then I think stopping

> If we only want to stop a part of the cluster, suppressing rebalancing

I think the no-data-loss option is bigger scope, perhaps part of

> When it comes to starting a part of the cluster, a "pause rebalancing"

Yup. This sort of option would only be used where data loss isn't an

Erik, I think in your case you'd be better served by a ConsistentHashFactory that always assigns at most one owner from each machine for each segment. I guess the fix for ISPN-3140 should work as well, but it wouldn't be very straightforward: you'd have to keep the rebalancingEnabled attribute set to false by default, and you'd have to enable it temporarily every time you have a topology change that you do want to process.

It's certainly possible to do this automatically from your app or from a monitoring daemon, but I'm pretty sure an enhanced topology-aware CHF would be a better fit.

On Fri, Jun 7, 2013 at 1:45 PM, Erik Salter <an1310(a)hotmail.com&gt; wrote: I'd like something similar. If I have equal keys on two machines (given an orthogonal setup and a TACH), I'd like to suppress state transfer and run with only one copy until I can recover my machines. The business case is that in a degraded scenario, additional replicas aren't going to buy me anything, as a failure will most likely be at the machine level and will cause me to lose data. Once I've recovered the other machine, I can turn back on state transfer to get my data redundancy. Erik -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus Sent: Tuesday, June 04, 2013 5:44 AM To: infinispan -Dev List Subject: Re: [infinispan-dev] Suppressing state transfer via JMX Manik, what's wrong with Dan's suggestion with clearing the cache before shutdown? On 31 May 2013, at 14:20, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >> >> If we only want to deal with full cluster shutdown, then I think stopping all application requests, calling Cache.clear() on one node, and then shutting down all the nodes should be simpler. On start, assuming no cache store, the caches will start empty, so starting all the nodes at once and only allowing application requests when they've all joined should also work without extra work. >> >> If we only want to stop a part of the cluster, suppressing rebalancing would be better, because we wouldn't lose all the data. But we'd still lose the keys whose owners are all among the nodes we want to stop. I've discussed this with Adrian, and we think if we want to stop a part of the cluster without losing data we need a JMX operation on the coordinator that will "atomically" remove a set of nodes from the CH. After the operation completes, the user will know it's safe to stop those nodes without losing data. > > I think the no-data-loss option is bigger scope, perhaps part of ISPN-1394. And that's not what I am asking about. > >> When it comes to starting a part of the cluster, a "pause rebalancing" option would probably be better - but again, on the coordinator, not on each joining node. And clearly, if more than numOwner nodes leave while rebalancing is suspended, data will be lost. > > Yup. This sort of option would only be used where data loss isn't an issue (such as a distributed cache). Where data loss is an issue, we'd need more control - ISPN-1394. > Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Tue, Jun 11, 2013 at 2:01 PM, Manik Surtani <msurtani(a)redhat.com&gt; wrote: On 10 Jun 2013, at 15:12, Dan Berindei <dan.berindei(a)gmail.com&gt; wrote: > Erik, I think in your case you'd be better served by a ConsistentHashFactory that always assigns at most one owner from each machine for each segment. > > I guess the fix for ISPN-3140 should work as well, but it wouldn't be very straightforward: you'd have to keep the rebalancingEnabled attribute set to false by default, and you'd have to enable it temporarily every time you have a topology change that you do want to process. Why? Does the workflow detailed in ISPN-3140 not work? ISPN-3140 is geared toward planned shutdowns, my understanding was that Erik's scenario involves an unexpected failure. Say we have a cluster with 4 nodes spread on 2 machines: A(m1), B(m1), C(m2), D(m2). If m2 fails, rebalancing will start automatically and m1 will have 2 copies of each entry (one on A and one on B). Trying to suspend rebalancing after m2 has already failed won't have any effect - if state transfer is already in progress it won't be cancelled. In order to avoid the unnecessary transfers, rebalancing would have to be suspended before the failure - i.e. rebalancing should be suspended by default. > It's certainly possible to do this automatically from your app or from a monitoring daemon, but I'm pretty sure an enhanced topology-aware CHF would be a better fit. Do explain. A custom ConsistentHashFactory could distribute segments so that a machine never has more than 1 copy of each segment. If m2 failed, there would be just one machine in the cluster, and just one copy of each segment. The factory would not change the consistent hash, and there wouldn't be any state transfer.

It could be even simpler - the existing TopologyAwareConsistentHashFactory/TopologyAwareSyncConsistentHashFactory implementations already ensure just one copy per machine if the number of machines is >= numOwners. So a custom ConsistentHashFactory could just extend one of these and skip calling super.rebalance() when the number of machines in the cluster is < numOwners. > > > > On Fri, Jun 7, 2013 at 1:45 PM, Erik Salter <an1310(a)hotmail.com&gt; wrote: > I'd like something similar. If I have equal keys on two machines (given an > orthogonal setup and a TACH), I'd like to suppress state transfer and run > with only one copy until I can recover my machines. The business case is > that in a degraded scenario, additional replicas aren't going to buy me > anything, as a failure will most likely be at the machine level and will > cause me to lose data. Once I've recovered the other machine, I can turn > back on state transfer to get my data redundancy. > > Erik > > -----Original Message----- > From: infinispan-dev-bounces(a)lists.jboss.org > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus > Sent: Tuesday, June 04, 2013 5:44 AM > To: infinispan -Dev List > Subject: Re: [infinispan-dev] Suppressing state transfer via JMX > > Manik, what's wrong with Dan's suggestion with clearing the cache before > shutdown? > > On 31 May 2013, at 14:20, Manik Surtani <msurtani(a)redhat.com&gt; wrote: > > >> > >> If we only want to deal with full cluster shutdown, then I think stopping > all application requests, calling Cache.clear() on one node, and then > shutting down all the nodes should be simpler. On start, assuming no cache > store, the caches will start empty, so starting all the nodes at once and > only allowing application requests when they've all joined should also work > without extra work. > >> > >> If we only want to stop a part of the cluster, suppressing rebalancing > would be better, because we wouldn't lose all the data. But we'd still lose > the keys whose owners are all among the nodes we want to stop. I've > discussed this with Adrian, and we think if we want to stop a part of the > cluster without losing data we need a JMX operation on the coordinator that > will "atomically" remove a set of nodes from the CH. After the operation > completes, the user will know it's safe to stop those nodes without losing > data. > > > > I think the no-data-loss option is bigger scope, perhaps part of > ISPN-1394. And that's not what I am asking about. > > > >> When it comes to starting a part of the cluster, a "pause rebalancing" > option would probably be better - but again, on the coordinator, not on each > joining node. And clearly, if more than numOwner nodes leave while > rebalancing is suspended, data will be lost. > > > > Yup. This sort of option would only be used where data loss isn't an > issue (such as a distributed cache). Where data loss is an issue, we'd need > more control - ISPN-1394. > > > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Manik, what's wrong with Dan's suggestion with clearing the cache before shutdown? On 31 May 2013, at 14:20, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >> >> If we only want to deal with full cluster shutdown, then I think stopping all application requests, calling Cache.clear() on one node, and then shutting down all the nodes should be simpler. On start, assuming no cache store, the caches will start empty, so starting all the nodes at once and only allowing application requests when they've all joined should also work without extra work. >> >> If we only want to stop a part of the cluster, suppressing rebalancing would be better, because we wouldn't lose all the data. But we'd still lose the keys whose owners are all among the nodes we want to stop. I've discussed this with Adrian, and we think if we want to stop a part of the cluster without losing data we need a JMX operation on the coordinator that will "atomically" remove a set of nodes from the CH. After the operation completes, the user will know it's safe to stop those nodes without losing data. > > I think the no-data-loss option is bigger scope, perhaps part of ISPN-1394. And that's not what I am asking about. > >> When it comes to starting a part of the cluster, a "pause rebalancing" option would probably be better - but again, on the coordinator, not on each joining node. And clearly, if more than numOwner nodes leave while rebalancing is suspended, data will be lost. > > Yup. This sort of option would only be used where data loss isn't an issue (such as a distributed cache). Where data loss is an issue, we'd need more control - ISPN-1394. > Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev