<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Yes, ISPN-1394 has a broader scope but
the proposed solution for ISPN-3140 solves quite a lot of
ISPN-1394 and it's not complex. We might not even need ISPN-1394
soon unless somebody really wants to control data ownership down
to segment granularity. If we only want to batch joins/leaves and
manually kick out nodes with or without loosing their data then
this proposal should be enough. This solution should not prevent
implementation of ISPN-1394 in future and will not need to be
removed/undone.<br>
<br>
Here are the details:<br>
<br>
1. <i>Add a JMX writable attribute (or operation?) to
ClusterTopologyManager (name it suppressRehashing?) that is
false by default but should also be configurable via API or xml.
While this attribute is true the ClusterTopologyManager queues
all join/leave/exclude(see below) requests and does not execute
them on the spot as it would normally happen. The value of this
attribute is ignored on all nodes but the coordinator. When it
is set back to false all queued operations (except the ones that
cancel eachother out) are executed. The setter should be
synchronous so when setting is back to false it does not return
until the queue is empty and all rehashing was processed. </i><br>
<br>
2. <i>We add a JMX operation excludeNodes(list of addresses) to
ClusterTopologyManager. Calling this method on any node but the
coordinator is no-op. This operation removes the node from the
topology (almost as if it left) and forces a rebalance.</i> The
node is still present in the current CH but not in the pending CH.
It's basically disowned by all its data which is now being
transferred to other (not excluded) nodes. At the end of the
rebalance the node is removed from topology for good and can be
shut down without loosing data. Note that if
suppressRehashing==false operation excludeNodes(..) just queues
them for later removal. We can batch multiple such exclusions and
then re-activate the rehashing.<br>
<br>
The parts that need to be implemented are written in italic above.
Everything else is already there.<br>
<br>
excludeNodes is a way of achieving a soft shutdown and should be
used only if we care about preserving data int the extreme case
where the nodes are the last/single owners. We can just kill the
node directly if we do not care about its data. <br>
<br>
suppressRehashing is a way of achieving some kind of batching of
topology changes. This should speed up state transfer a lot
because it avoids a lot of pointless reshuffling of data segments
when we have many successive joiners/leavers.<br>
<br>
So what happens if the current coordinator dies for whatever
reason? The new one will take control and will not have knowledge
of the existing rehash queue or the previous status of
suppressRehashing attribute so it will just get the current cache
membership status from all members of current view and proceed
with the rehashing as usual. If the user does not want this he can
set a default value of true for suppressRehashing. The admin has
to interact now via JMX with the new coordinator. But that's not
as bad as the alternative where all the nodes are involved in this
jmx scheme :) I think having only the coordinator involved in this
is a plus.<br>
<br>
Manik, how does this fit for the full and partial shutdown?<br>
<br>
Cheers<br>
Adi<br>
<br>
<br>
On 05/31/2013 04:20 PM, Manik Surtani wrote:<br>
</div>
<blockquote
cite="mid:BDFDAFF8-7620-49E5-9CAF-1B28DB795C78@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<br>
<div>
<div>On 31 May 2013, at 13:52, Dan Berindei <<a
moz-do-not-send="true" href="mailto:dan.berindei@gmail.com">dan.berindei@gmail.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="ltr">
<div>
<div>If we only want to deal with full cluster shutdown,
then I think stopping all application requests, calling
Cache.clear() on one node, and then shutting down all
the nodes should be simpler. On start, assuming no cache
store, the caches will start empty, so starting all the
nodes at once and only allowing application requests
when they've all joined should also work without extra
work.<br>
<br>
</div>
<div>If we only want to stop a part of the cluster,
suppressing rebalancing would be better, because we
wouldn't lose all the data. But we'd still lose the keys
whose owners are all among the nodes we want to stop.
I've discussed this with Adrian, and we think if we want
to stop a part of the cluster without losing data we
need a JMX operation on the coordinator that will
"atomically" remove a set of nodes from the CH. After
the operation completes, the user will know it's safe to
stop those nodes without losing data.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I think the no-data-loss option is bigger scope, perhaps
part of ISPN-1394. And that's not what I am asking about.</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>When it comes to starting a part of the cluster, a
"pause rebalancing" option would probably be better -
but again, on the coordinator, not on each joining node.
And clearly, if more than numOwner nodes leave while
rebalancing is suspended, data will be lost.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yup. This sort of option would only be used where data
loss isn't an issue (such as a distributed cache). Where data
loss is an issue, we'd need more control - ISPN-1394.</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Cheers<br>
</div>
<div>Dan<br>
<br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 31, 2013 at 12:17 PM,
Manik Surtani <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:msurtani@redhat.com" target="_blank">msurtani@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Guys<br>
<br>
We've discussed ISPN-3140 elsewhere before, I'm brining
it to this forum now.<br>
<br>
<a moz-do-not-send="true"
href="https://issues.jboss.org/browse/ISPN-3140"
target="_blank">https://issues.jboss.org/browse/ISPN-3140</a><br>
<br>
Any thoughts/concerns? Particularly looking to hear
from Dan or Adrian about viability, complexity, ease of
implementation.<br>
<br>
Thanks<br>
Manik<br>
--<br>
Manik Surtani<br>
<a moz-do-not-send="true" href="mailto:manik@jboss.org">manik@jboss.org</a><br>
<a moz-do-not-send="true"
href="http://twitter.com/maniksurtani" target="_blank">twitter.com/maniksurtani</a><br>
<br>
Platform Architect, JBoss Data Grid<br>
<a moz-do-not-send="true" href="http://red.ht/data-grid"
target="_blank">http://red.ht/data-grid</a><br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/infinispan-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div>
<br>
<div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate;
border-spacing: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; "><span
class="Apple-style-span" style="border-collapse: separate;
color: rgb(0, 0, 0); font-family: Helvetica; font-style:
normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; border-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; font-size: medium; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space; "><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family: Helvetica;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
border-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size: medium; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space; ">
<div>--</div>
<div>Manik Surtani</div>
<div><a moz-do-not-send="true"
href="mailto:manik@jboss.org">manik@jboss.org</a></div>
<div><a moz-do-not-send="true"
href="http://twitter.com/maniksurtani">twitter.com/maniksurtani</a></div>
<div><br>
</div>
<div>
<div>Platform Architect, JBoss Data Grid</div>
<div><a moz-do-not-send="true"
href="http://red.ht/data-grid">http://red.ht/data-grid</a></div>
</div>
</div>
</span></div>
</span></div>
</span>
</div>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
infinispan-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
</body>
</html>