<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 17, 2013 at 1:28 PM, Bela Ban <span dir="ltr"><<a href="mailto:bban@redhat.com" target="_blank">bban@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Well, first of all, we won't *have* any conflicting topology IDs, as the<br>
minority partitions don't change them after becoming minority.<br>
<br></blockquote><div><br>We don't have the notion of
"conflicting topology IDs" with the current algorithm, either. After a merge, it doesn't
matter which partition had the highest topology id before, we just pick a
topology id that we know wasn't used in any of the partitions. <br><br>Then we assume that each node has latest data in the
segments that it owned in its pre-merge consistent hash. Obviously, if any value changed while the partitions were separated, we
would have lost consistency - hence the chaos that Adrian mentioned.<br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Secondly, we can end up with the coordinator of a minority partition<br>
becoming the coordinator of the new merged partition, so we shouldn't<br>
rely on that (but I don't think we do so anyway?).<br>
<br></blockquote><div><br></div><div>No, we don't care what partition the merge coordinator was in, we tread all partitions the same (with some extra work for overlapping partitions).<br><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
On a merge, everyone knows whether it came from a minority or majority<br>
partition, and the algorithm for state transfer should always clear the<br>
state in members in the minority partition and overwrite it from members<br>
of the primary partition.<br></blockquote><div><br></div><div>Actually, the merge coordinator is the only one that has to know which node is from a minority or a majority partition.<br><br></div><div>I like the idea of always clearing the state in members of the minority partition(s), but one problem with that is that there may be some keys that only had owners in the minority partition(s). If we wiped the state of the minority partition members, those keys would be lost.<br>
<br></div><div>Of course, you could argue that the cluster already lost those keys when we allowed the majority partition to continue working without having those keys... We could also rely on the topology information, and say that we only support partitioning when numOwners >= numSites (or numRacks, if there is only one site, or numMachines, if there is a single rack).<br>
<br></div><div>One other option is to perform a more complicated post-merge state transfer, in which each partition sends all the data it has to all the other partitions, and on the receiving end each node has a "conflict resolution" component that can merge two values. That is definitely more complicated than just going with a primary partition, though.<br>
</div><div><br></div><div>One final point... when a node comes back online and it has a local cache store, it is very much as if we had a merge view. The current approach is to join as if the node didn't have any data, then delete everything from the cache store that is not mapped to the node in the consistent hash. Obviously that can lead to consistency problems, just like our current merge algorithm. It would be nice if we could handle both these cases the same way.<br>
</div><div><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class=""><div class="h5"><br>
On 4/17/13 10:58 AM, Radim Vansa wrote:<br>
> And the nice behaviour is that if we have partitions P1 and P2 with latest common topology 20 , when P2 increased it's topology to, say 40, while P1 only to 30, when a new coordinator from P1 will be elected it will try to compare these topology ids directly (assuming which one is newer or older) which won't end up well.<br>
><br>
> Radim<br>
><br>
> ----- Original Message -----<br>
> | From: "Adrian Nistor" <<a href="mailto:anistor@redhat.com">anistor@redhat.com</a>><br>
> | To: "infinispan -Dev List" <<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a>><br>
> | Cc: "Manik Surtani" <<a href="mailto:msurtani@redhat.com">msurtani@redhat.com</a>><br>
> | Sent: Wednesday, April 17, 2013 10:31:39 AM<br>
> | Subject: Re: [infinispan-dev] ISPN-263 and handling partitions<br>
> |<br>
> | In case of MergeView the cluster topology manager running on (the new)<br>
> | coordinator will request the current cache topology from all members and<br>
> | will compute a new topology as the union of all. The new topology id is<br>
> | computed as the max + 2 of the existing topology ids. Any currently<br>
> | pending rebalance in any subpartition is ended now and a new rebalance<br>
> | is triggered for the new cluster. No data version conflict resolution is<br>
> | performed => chaos :)<br>
> |<br>
> | On 04/16/2013 10:05 PM, Manik Surtani wrote:<br>
> | > Guys - I've started documenting this here [1] and will put together a<br>
> | > prototype this week.<br>
> | ><br>
> | > One question though, perhaps one for Dan/Adrian - is there any special<br>
> | > handling for state transfer if a MergeView is detected?<br>
> | ><br>
> | > - M<br>
> | ><br>
> | > [1] <a href="https://community.jboss.org/wiki/DesignDealingWithNetworkPartitions" target="_blank">https://community.jboss.org/wiki/DesignDealingWithNetworkPartitions</a><br>
> | ><br>
> | > On 6 Apr 2013, at 04:26, Bela Ban <<a href="mailto:bban@redhat.com">bban@redhat.com</a>> wrote:<br>
> | ><br>
> | >><br>
> | >> On 4/5/13 3:53 PM, Manik Surtani wrote:<br>
> | >>> Guys,<br>
> | >>><br>
> | >>> So this is what I have in mind for this, looking for opinions.<br>
> | >>><br>
> | >>> 1. We write a SplitBrainListener which is registered when the<br>
> | >>> channel connects. The aim of this listener is to identify when we<br>
> | >>> have a partition. This can be identified when a view change is<br>
> | >>> detected, and the new view is significantly smaller than the old<br>
> | >>> view. Easier to detect for large clusters, but smaller clusters will<br>
> | >>> be harder - trying to decide between a node leaving vs a partition.<br>
> | >>> (Any better ideas here?)<br>
> | >>><br>
> | >>> 2. The SBL flips a switch in an interceptor<br>
> | >>> (SplitBrainHandlerInterceptor?) which switches the node to be<br>
> | >>> read-only (reject invocations that change the state of the local<br>
> | >>> node) if it is in the smaller partition (newView.size < oldView.size<br>
> | >>> / 2). Only works reliably for odd-numbered cluster sizes, and the<br>
> | >>> issues with small clusters seen in (1) will affect here as well.<br>
> | >>><br>
> | >>> 3. The SBL can flip the switch in the interceptor back to normal<br>
> | >>> operation once a MergeView is detected.<br>
> | >>><br>
> | >>> It's no way near perfect but at least it means that we can recommend<br>
> | >>> enabling this and setting up an odd number of nodes, with a cluster<br>
> | >>> size of at least N if you want to reduce inconsistency in your grid<br>
> | >>> during partitions.<br>
> | >>><br>
> | >>> Is this even useful?<br>
> | >><br>
> | >> So I assume this is to shut down (or make read-only) non primary<br>
> | >> partitions. I'd go with an approach similar to [1] section 5.6.2, which<br>
> | >> makes a partition read-only once it drops below a certain number of nodes<br>
> | >> N.<br>
> | >><br>
> | >><br>
> | >>> Bela, is there a more reliable mechanism to detect a split in (1)?<br>
> | >> I'm afraid no. We never know whether a large number of members being<br>
> | >> removed from the view means that they left, or that we have a partition,<br>
> | >> e.g. because a switch crashed.<br>
> | >><br>
> | >> One thing you could do though is for members who are about to leave<br>
> | >> regularly to broadcast a LEAVE messages, so that when the view is<br>
> | >> received, the SBL knows those members, and might be able to determine<br>
> | >> better whether we have a partition, or not.<br>
> | >><br>
> | >> [1] <a href="http://www.jgroups.org/manual-3.x/html/user-advanced.html" target="_blank">http://www.jgroups.org/manual-3.x/html/user-advanced.html</a>, section<br>
> | >> 5.6.2<br>
> | >><br>
> | >> --<br>
> | >> Bela Ban, JGroups lead (<a href="http://www.jgroups.org" target="_blank">http://www.jgroups.org</a>)<br>
> | >> _______________________________________________<br>
> | >> infinispan-dev mailing list<br>
> | >> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> | >> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
> | > --<br>
> | > Manik Surtani<br>
> | > <a href="mailto:manik@jboss.org">manik@jboss.org</a><br>
> | > <a href="http://twitter.com/maniksurtani" target="_blank">twitter.com/maniksurtani</a><br>
> | ><br>
> | > Platform Architect, JBoss Data Grid<br>
> | > <a href="http://red.ht/data-grid" target="_blank">http://red.ht/data-grid</a><br>
> | ><br>
> | ><br>
> | > _______________________________________________<br>
> | > infinispan-dev mailing list<br>
> | > <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> | > <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
> |<br>
> | _______________________________________________<br>
> | infinispan-dev mailing list<br>
> | <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> | <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
> |<br>
> _______________________________________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
><br>
<br>
--<br>
Bela Ban, JGroups lead (<a href="http://www.jgroups.org" target="_blank">http://www.jgroups.org</a>)<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>