[infinispan-dev] ISPN-263 and handling partitions

[Dennis Reed]

Some other alternatives for detecting a split:

- a hard-coded number of nodes required for a quorum configured by the 
customer.
   This is what we recommend for HASIngletons when the customer wants to 
guarantee that something is running at most once in the cluster.

- an alternative communication channel
   JBoss Messaging added this a year or two ago, where it uses a 
database in addition to JGroups to keep track of cluster membership and 
detect discrepencies between the two
   (I don't know all the details of the implementation)

- something in JGroups to notify Infinispan of the difference between a 
normal node leave and being kicked
   (easily detectable inside JGroups, but I don't think this is really 
exposed to apps?)

#1 should probably be pluggable, as a particular strategy may make sense 
for specific use cases.

-Dennis

On 04/05/2013 08:53 AM, Manik Surtani wrote:
> Guys,
>
> So this is what I have in mind for this, looking for opinions.
>
> 1.  We write a SplitBrainListener which is registered when the channel connects.  The aim of this listener is to identify when we have a partition.  This can be identified when a view change is detected, and the new view is significantly smaller than the old view.  Easier to detect for large clusters, but smaller clusters will be harder - trying to decide between a node leaving vs a partition.  (Any better ideas here?)
>
> 2.  The SBL flips a switch in an interceptor (SplitBrainHandlerInterceptor?) which switches the node to be read-only (reject invocations that change the state of the local node) if it is in the smaller partition (newView.size < oldView.size / 2).  Only works reliably for odd-numbered cluster sizes, and the issues with small clusters seen in (1) will affect here as well.
>
> 3.  The SBL can flip the switch in the interceptor back to normal operation once a MergeView is detected.
>
> It's no way near perfect but at least it means that we can recommend enabling this and setting up an odd number of nodes, with a cluster size of at least N if you want to reduce inconsistency in your grid during partitions.
>
> Is this even useful?
>
> Bela, is there a more reliable mechanism to detect a split in (1)?
>
> Cheers
> Manik
>
> --
> Manik Surtani
> manik at jboss.org
> twitter.com/maniksurtani
>
> Platform Architect, JBoss Data Grid
> http://red.ht/data-grid
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev