Re: [infinispan-dev] ISPN-263 and handling partitions
Some other alternatives for detecting a split:
- a hard-coded number of nodes required for a quorum configured by the
customer.
This is what we recommend for HASIngletons when the customer wants to
guarantee that something is running at most once in the cluster.
- an alternative communication channel
JBoss Messaging added this a year or two ago, where it uses a
database in addition to JGroups to keep track of cluster membership and
detect discrepencies between the two
(I don't know all the details of the implementation)
- something in JGroups to notify Infinispan of the difference between a
normal node leave and being kicked
(easily detectable inside JGroups, but I don't think this is really
exposed to apps?)
#1 should probably be pluggable, as a particular strategy may make sense
for specific use cases.
-Dennis
On 04/05/2013 08:53 AM, Manik Surtani wrote:
Guys,
So this is what I have in mind for this, looking for opinions.
1. We write a SplitBrainListener which is registered when the channel connects. The aim
of this listener is to identify when we have a partition. This can be identified when a
view change is detected, and the new view is significantly smaller than the old view.
Easier to detect for large clusters, but smaller clusters will be harder - trying to
decide between a node leaving vs a partition. (Any better ideas here?)
2. The SBL flips a switch in an interceptor (SplitBrainHandlerInterceptor?) which
switches the node to be read-only (reject invocations that change the state of the local
node) if it is in the smaller partition (newView.size < oldView.size / 2). Only works
reliably for odd-numbered cluster sizes, and the issues with small clusters seen in (1)
will affect here as well.
3. The SBL can flip the switch in the interceptor back to normal operation once a
MergeView is detected.
It's no way near perfect but at least it means that we can recommend enabling this
and setting up an odd number of nodes, with a cluster size of at least N if you want to
reduce inconsistency in your grid during partitions.
Is this even useful?
Bela, is there a more reliable mechanism to detect a split in (1)?
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Platform Architect, JBoss Data Grid
http://red.ht/data-grid
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev