<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">The coordinator is the first member of
the jgroups view, which is easy to obtain via jmx from any node,
but you do make a valid point. We should remove this strain from
the admin and automate a bit. I like the forwarding idea. <br>
<br>
Excluding the coordinator should not be a problem. After this is
performed there will be no running caches left on the coordinator
but there will still be a <i> </i>ClusterTopologyManager
instance running on the coordinator which will continue to manage
the topology of the cluster. This was made possible (in 5.1?) when
the concept of asymetric caches was implemented. After this
coordinator is excluded it can still work until it is killed and a
new coordinator is elected and will take control.<br>
<br>
Thanks for feedback Dennis!<br>
<br>
On 05/31/2013 07:52 PM, Dennis Reed wrote:<br>
</div>
<blockquote cite="mid:51A8D548.1060203@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">I see 2 potential issues:<br>
<br>
1. How does the user know which node is the master to connect
to, since the operations are a no-op on all the others?<br>
<br>
- instead of a no-op, what if the other nodes just forward
the operation to the correct node?<br>
Then the user doesn't have to know who the current master
is, and can just connect to any node.<br>
<br>
2. What if the current master is one of the nodes being
stopped?<br>
<br>
-Dennis<br>
<br>
On 05/31/2013 11:40 AM, Adrian Nistor wrote:<br>
</div>
<blockquote cite="mid:51A8D295.8070506@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Yes, ISPN-1394 has a broader scope
but the proposed solution for ISPN-3140 solves quite a lot of
ISPN-1394 and it's not complex. We might not even need
ISPN-1394 soon unless somebody really wants to control data
ownership down to segment granularity. If we only want to
batch joins/leaves and manually kick out nodes with or without
loosing their data then this proposal should be enough. This
solution should not prevent implementation of ISPN-1394 in
future and will not need to be removed/undone.<br>
<br>
Here are the details:<br>
<br>
1. <i>Add a JMX writable attribute (or operation?) to
ClusterTopologyManager (name it suppressRehashing?) that is
false by default but should also be configurable via API or
xml. While this attribute is true the ClusterTopologyManager
queues all join/leave/exclude(see below) requests and does
not execute them on the spot as it would normally happen.
The value of this attribute is ignored on all nodes but the
coordinator. When it is set back to false all queued
operations (except the ones that cancel eachother out) are
executed. The setter should be synchronous so when setting
is back to false it does not return until the queue is empty
and all rehashing was processed. </i><br>
<br>
2. <i>We add a JMX operation excludeNodes(list of addresses)
to ClusterTopologyManager. Calling this method on any node
but the coordinator is no-op. This operation removes the
node from the topology (almost as if it left) and forces a
rebalance.</i> The node is still present in the current CH
but not in the pending CH. It's basically disowned by all its
data which is now being transferred to other (not excluded)
nodes. At the end of the rebalance the node is removed from
topology for good and can be shut down without loosing data.
Note that if suppressRehashing==false operation
excludeNodes(..) just queues them for later removal. We can
batch multiple such exclusions and then re-activate the
rehashing.<br>
<br>
The parts that need to be implemented are written in italic
above. Everything else is already there.<br>
<br>
excludeNodes is a way of achieving a soft shutdown and should
be used only if we care about preserving data int the extreme
case where the nodes are the last/single owners. We can just
kill the node directly if we do not care about its data. <br>
<br>
suppressRehashing is a way of achieving some kind of batching
of topology changes. This should speed up state transfer a lot
because it avoids a lot of pointless reshuffling of data
segments when we have many successive joiners/leavers.<br>
<br>
So what happens if the current coordinator dies for whatever
reason? The new one will take control and will not have
knowledge of the existing rehash queue or the previous status
of suppressRehashing attribute so it will just get the current
cache membership status from all members of current view and
proceed with the rehashing as usual. If the user does not want
this he can set a default value of true for suppressRehashing.
The admin has to interact now via JMX with the new
coordinator. But that's not as bad as the alternative where
all the nodes are involved in this jmx scheme :) I think
having only the coordinator involved in this is a plus.<br>
<br>
Manik, how does this fit for the full and partial shutdown?<br>
<br>
Cheers<br>
Adi<br>
<br>
<br>
On 05/31/2013 04:20 PM, Manik Surtani wrote:<br>
</div>
<blockquote
cite="mid:BDFDAFF8-7620-49E5-9CAF-1B28DB795C78@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<br>
<div>
<div>On 31 May 2013, at 13:52, Dan Berindei <<a
moz-do-not-send="true"
href="mailto:dan.berindei@gmail.com">dan.berindei@gmail.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="ltr">
<div>
<div>If we only want to deal with full cluster
shutdown, then I think stopping all application
requests, calling Cache.clear() on one node, and
then shutting down all the nodes should be simpler.
On start, assuming no cache store, the caches will
start empty, so starting all the nodes at once and
only allowing application requests when they've all
joined should also work without extra work.<br>
<br>
</div>
<div>If we only want to stop a part of the cluster,
suppressing rebalancing would be better, because we
wouldn't lose all the data. But we'd still lose the
keys whose owners are all among the nodes we want to
stop. I've discussed this with Adrian, and we think
if we want to stop a part of the cluster without
losing data we need a JMX operation on the
coordinator that will "atomically" remove a set of
nodes from the CH. After the operation completes,
the user will know it's safe to stop those nodes
without losing data.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I think the no-data-loss option is bigger scope,
perhaps part of ISPN-1394. And that's not what I am
asking about.</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>When it comes to starting a part of the cluster,
a "pause rebalancing" option would probably be
better - but again, on the coordinator, not on each
joining node. And clearly, if more than numOwner
nodes leave while rebalancing is suspended, data
will be lost.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yup. This sort of option would only be used where data
loss isn't an issue (such as a distributed cache). Where
data loss is an issue, we'd need more control - ISPN-1394.</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Cheers<br>
</div>
<div>Dan<br>
<br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 31, 2013 at 12:17
PM, Manik Surtani <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:msurtani@redhat.com" target="_blank">msurtani@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Guys<br>
<br>
We've discussed ISPN-3140 elsewhere before, I'm
brining it to this forum now.<br>
<br>
<a moz-do-not-send="true"
href="https://issues.jboss.org/browse/ISPN-3140"
target="_blank">https://issues.jboss.org/browse/ISPN-3140</a><br>
<br>
Any thoughts/concerns? Particularly looking to hear
from Dan or Adrian about viability, complexity, ease
of implementation.<br>
<br>
Thanks<br>
Manik<br>
--<br>
Manik Surtani<br>
<a moz-do-not-send="true"
href="mailto:manik@jboss.org">manik@jboss.org</a><br>
<a moz-do-not-send="true"
href="http://twitter.com/maniksurtani"
target="_blank">twitter.com/maniksurtani</a><br>
<br>
Platform Architect, JBoss Data Grid<br>
<a moz-do-not-send="true"
href="http://red.ht/data-grid" target="_blank">http://red.ht/data-grid</a><br>
<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true"
href="https://lists.jboss.org/mailman/listinfo/infinispan-dev"
target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote>
</div>
<br>
<div apple-content-edited="true"> <span
class="Apple-style-span" style="border-collapse: separate;
border-spacing: 0px; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space; "><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family: Helvetica;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent:
0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; border-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size: medium; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space; "><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family:
Helvetica; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
border-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size: medium;
">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; ">
<div>--</div>
<div>Manik Surtani</div>
<div><a moz-do-not-send="true"
href="mailto:manik@jboss.org">manik@jboss.org</a></div>
<div><a moz-do-not-send="true"
href="http://twitter.com/maniksurtani">twitter.com/maniksurtani</a></div>
<div><br>
</div>
<div>
<div>Platform Architect, JBoss Data Grid</div>
<div><a moz-do-not-send="true"
href="http://red.ht/data-grid">http://red.ht/data-grid</a></div>
</div>
</div>
</span></div>
</span></div>
</span> </div>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
infinispan-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
infinispan-dev mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
infinispan-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></pre>
</blockquote>
<br>
</body>
</html>