<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 11, 2013 at 2:01 PM, Manik Surtani <span dir="ltr"><<a href="mailto:msurtani@redhat.com" target="_blank">msurtani@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div class="im"><div>On 10 Jun 2013, at 15:12, Dan Berindei <<a href="mailto:dan.berindei@gmail.com" target="_blank">dan.berindei@gmail.com</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr">Erik, I think in your case you'd be better served by a ConsistentHashFactory that always assigns at most one owner from each machine for each segment. <br><br>I guess the fix for ISPN-3140 should work as well, but it wouldn't be very straightforward: you'd have to keep the rebalancingEnabled attribute set to false by default, and you'd have to enable it temporarily every time you have a topology change that you do want to process. </div>
</blockquote><div><br></div></div><div>Why? Does the workflow detailed in ISPN-3140 not work?</div><div class="im"><br></div></div></div></blockquote><div><br></div><div>ISPN-3140 is geared toward planned shutdowns, my understanding was that Erik's scenario involves an unexpected failure.<br>
</div><div><br></div><div>Say we have a cluster with 4 nodes spread on 2 machines: A(m1), B(m1), C(m2), D(m2).<br></div><div>If m2 fails, rebalancing will start automatically and m1 will have 2 copies of each entry (one on A and one on B).<br>
</div><div>Trying to suspend rebalancing after m2 has already failed won't have any effect - if state transfer is already in progress it won't be cancelled.<br></div><div>In order to avoid the unnecessary transfers, rebalancing would have to be suspended before the failure - i.e. rebalancing should be suspended by default.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div class="im"><blockquote type="cite"><div dir="ltr">It's certainly possible to do this automatically from your app or from a monitoring daemon, but I'm pretty sure an enhanced topology-aware CHF would be a better fit.<br>
</div></blockquote><div><br></div></div><div>Do explain.</div><div><div class="h5"><div><br></div></div></div></div></div></blockquote><div><br></div><div>A custom ConsistentHashFactory could distribute segments so that a machine never has more than 1 copy of each segment. If m2 failed, there would be just one machine in the cluster, and just one copy of each segment. The factory would not change the consistent hash, and there wouldn't be any state transfer.<br>
<br></div><div>It could be even simpler - the existing TopologyAwareConsistentHashFactory/TopologyAwareSyncConsistentHashFactory implementations already ensure just one copy per machine if the number of machines is >= numOwners. So a custom ConsistentHashFactory could just extend one of these and skip calling super.rebalance() when the number of machines in the cluster is < numOwners.<br>
</div><div> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div><div class="h5"><div></div><br><blockquote type="cite">
<div dir="ltr">
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 7, 2013 at 1:45 PM, Erik Salter <span dir="ltr"><<a href="mailto:an1310@hotmail.com" target="_blank">an1310@hotmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'd like something similar. If I have equal keys on two machines (given an<br>
orthogonal setup and a TACH), I'd like to suppress state transfer and run<br>
with only one copy until I can recover my machines. The business case is<br>
that in a degraded scenario, additional replicas aren't going to buy me<br>
anything, as a failure will most likely be at the machine level and will<br>
cause me to lose data. Once I've recovered the other machine, I can turn<br>
back on state transfer to get my data redundancy.<br>
<span><font color="#888888"><br>
Erik<br>
</font></span><div><div><br>
-----Original Message-----<br>
From: <a href="mailto:infinispan-dev-bounces@lists.jboss.org" target="_blank">infinispan-dev-bounces@lists.jboss.org</a><br>
[mailto:<a href="mailto:infinispan-dev-bounces@lists.jboss.org" target="_blank">infinispan-dev-bounces@lists.jboss.org</a>] On Behalf Of Mircea Markus<br>
Sent: Tuesday, June 04, 2013 5:44 AM<br>
To: infinispan -Dev List<br>
Subject: Re: [infinispan-dev] Suppressing state transfer via JMX<br>
<br>
Manik, what's wrong with Dan's suggestion with clearing the cache before<br>
shutdown?<br>
<br>
On 31 May 2013, at 14:20, Manik Surtani <<a href="mailto:msurtani@redhat.com" target="_blank">msurtani@redhat.com</a>> wrote:<br>
<br>
>><br>
>> If we only want to deal with full cluster shutdown, then I think stopping<br>
all application requests, calling Cache.clear() on one node, and then<br>
shutting down all the nodes should be simpler. On start, assuming no cache<br>
store, the caches will start empty, so starting all the nodes at once and<br>
only allowing application requests when they've all joined should also work<br>
without extra work.<br>
>><br>
>> If we only want to stop a part of the cluster, suppressing rebalancing<br>
would be better, because we wouldn't lose all the data. But we'd still lose<br>
the keys whose owners are all among the nodes we want to stop. I've<br>
discussed this with Adrian, and we think if we want to stop a part of the<br>
cluster without losing data we need a JMX operation on the coordinator that<br>
will "atomically" remove a set of nodes from the CH. After the operation<br>
completes, the user will know it's safe to stop those nodes without losing<br>
data.<br>
><br>
> I think the no-data-loss option is bigger scope, perhaps part of<br>
ISPN-1394. And that's not what I am asking about.<br>
><br>
>> When it comes to starting a part of the cluster, a "pause rebalancing"<br>
option would probably be better - but again, on the coordinator, not on each<br>
joining node. And clearly, if more than numOwner nodes leave while<br>
rebalancing is suspended, data will be lost.<br>
><br>
> Yup. This sort of option would only be used where data loss isn't an<br>
issue (such as a distributed cache). Where data loss is an issue, we'd need<br>
more control - ISPN-1394.<br>
><br>
<br>
Cheers,<br>
--<br>
Mircea Markus<br>
Infinispan lead (<a href="http://www.infinispan.org/" target="_blank">www.infinispan.org</a>)<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div>
_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div></div></div><br><div class="im"><div>
<span style="border-collapse:separate;border-spacing:0px"><div style="word-wrap:break-word"><span style="border-spacing:0px;text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<span style="border-spacing:0px;text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<div>--</div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div><a href="http://twitter.com/maniksurtani" target="_blank">twitter.com/maniksurtani</a></div><div><br>
</div><div><div>Platform Architect, JBoss Data Grid</div><div><a href="http://red.ht/data-grid" target="_blank">http://red.ht/data-grid</a></div></div></div></span></div></span></div></span>
</div>
<br></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div></div>