Re: [infinispan-dev] ISPN-263 and handling partitions

And the nice behaviour is that if we have partitions P1 and P2 with latest common topology 20 , when P2 increased it's topology to, say 40, while P1 only to 30, when a new coordinator from P1 will be elected it will try to compare these topology ids directly (assuming which one is newer or older) which won't end up well. Radim ----- Original Message ----- | From: "Adrian Nistor" <anistor(a)redhat.com&gt; | To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org&gt; | Cc: "Manik Surtani" <msurtani(a)redhat.com&gt; | Sent: Wednesday, April 17, 2013 10:31:39 AM | Subject: Re: [infinispan-dev] ISPN-263 and handling partitions | | In case of MergeView the cluster topology manager running on (the new) | coordinator will request the current cache topology from all members and | will compute a new topology as the union of all. The new topology id is | computed as the max + 2 of the existing topology ids. Any currently | pending rebalance in any subpartition is ended now and a new rebalance | is triggered for the new cluster. No data version conflict resolution is | performed => chaos :) | | On 04/16/2013 10:05 PM, Manik Surtani wrote: | > Guys - I've started documenting this here [1] and will put together a | > prototype this week. | > | > One question though, perhaps one for Dan/Adrian - is there any special | > handling for state transfer if a MergeView is detected? | > | > - M | > | > [1] https://community.jboss.org/wiki/DesignDealingWithNetworkPartitions | > | > On 6 Apr 2013, at 04:26, Bela Ban <bban(a)redhat.com&gt; wrote: | > | >> | >> On 4/5/13 3:53 PM, Manik Surtani wrote: | >>> Guys, | >>> | >>> So this is what I have in mind for this, looking for opinions. | >>> | >>> 1. We write a SplitBrainListener which is registered when the | >>> channel connects. The aim of this listener is to identify when we | >>> have a partition. This can be identified when a view change is | >>> detected, and the new view is significantly smaller than the old | >>> view. Easier to detect for large clusters, but smaller clusters will | >>> be harder - trying to decide between a node leaving vs a partition. | >>> (Any better ideas here?) | >>> | >>> 2. The SBL flips a switch in an interceptor | >>> (SplitBrainHandlerInterceptor?) which switches the node to be | >>> read-only (reject invocations that change the state of the local | >>> node) if it is in the smaller partition (newView.size < oldView.size | >>> / 2). Only works reliably for odd-numbered cluster sizes, and the | >>> issues with small clusters seen in (1) will affect here as well. | >>> | >>> 3. The SBL can flip the switch in the interceptor back to normal | >>> operation once a MergeView is detected. | >>> | >>> It's no way near perfect but at least it means that we can recommend | >>> enabling this and setting up an odd number of nodes, with a cluster | >>> size of at least N if you want to reduce inconsistency in your grid | >>> during partitions. | >>> | >>> Is this even useful? | >> | >> So I assume this is to shut down (or make read-only) non primary | >> partitions. I'd go with an approach similar to [1] section 5.6.2, which | >> makes a partition read-only once it drops below a certain number of nodes | >> N. | >> | >> | >>> Bela, is there a more reliable mechanism to detect a split in (1)? | >> I'm afraid no. We never know whether a large number of members being | >> removed from the view means that they left, or that we have a partition, | >> e.g. because a switch crashed. | >> | >> One thing you could do though is for members who are about to leave | >> regularly to broadcast a LEAVE messages, so that when the view is | >> received, the SBL knows those members, and might be able to determine | >> better whether we have a partition, or not. | >> | >> [1] http://www.jgroups.org/manual-3.x/html/user-advanced.html, section | >> 5.6.2 | >> | >> -- | >> Bela Ban, JGroups lead (http://www.jgroups.org) | >> _______________________________________________ | >> infinispan-dev mailing list | >> infinispan-dev(a)lists.jboss.org | >> https://lists.jboss.org/mailman/listinfo/infinispan-dev | > -- | > Manik Surtani | > manik(a)jboss.org | > twitter.com/maniksurtani | > | > Platform Architect, JBoss Data Grid | > http://red.ht/data-grid | > | > | > _______________________________________________ | > infinispan-dev mailing list | > infinispan-dev(a)lists.jboss.org | > https://lists.jboss.org/mailman/listinfo/infinispan-dev | | _______________________________________________ | infinispan-dev mailing list | infinispan-dev(a)lists.jboss.org | https://lists.jboss.org/mailman/listinfo/infinispan-dev | _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev