Since it seems I can't comment on the wiki itself, I am just replying here. I wonder if the third option 'Primary partition' is desirable. I think availability in some cases would be harmed more than we would like. Lets say you have a 5 node cluster where 3 of the nodes are behind the same router and the remaining 2 are behind a different one. If the router crashes, power loss etc. for the 3 and are no longer addressable you have your 2 partitions (possibly 1 or even 4). When this occurs the other 2 nodes would go into read only mode since they lost the quorum check.

But the 3 nodes that are "writable" can't be accessed any longer and thus no writes can be performed on the cluster. It seems we would still want to allow writes to provide as high of availability as possible.

Also if we did have read only, what criteria would cause those nodes to be writeable again?

There is no guarantee when the other nodes will ever come back up or if there will ever be additional ones anytime soon. - Will > [1] https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions

On 10/30/2013 08:46 PM, Mircea Markus wrote: > On Oct 30, 2013, at 7:28 PM, William Burns <mudokonman(a)gmail.com&gt; wrote: > >> Since it seems I can't comment on the wiki itself, I am just replying here. >> >> I wonder if the third option 'Primary partition' is desirable. I >> think availability in some cases would be harmed more than we would >> like. >> >> Lets say you have a 5 node cluster where 3 of the nodes are behind the >> same router and the remaining 2 are behind a different one. If the >> router crashes, power loss etc. for the 3 and are no longer >> addressable you have your 2 partitions (possibly 1 or even 4). When >> this occurs the other 2 nodes would go into read only mode since they >> lost the quorum check. > agreed. > >> But the 3 nodes that are "writable" can't be >> accessed any longer and thus no writes can be performed on the >> cluster. It seems we would still want to allow writes to provide as >> high of availability as possible. > we actually don't take the decision for the user but to plug in his own PartitionHandlingStrategy to make a wiser decision based on their network specifics. > The quorum approach written there is just a suggestion, I'll make that clearer. > >> Also if we did have read only, what criteria would cause those nodes >> to be writeable again? > Changing the availability status is possible through JMX, so either manual intervention or some MergeListeners that do that automatically. You should probably outline the MergeListeners on wiki as well. I believe that automatic merge is highly desirable, because for example long GC may cause partition more likely than network failure - and you don't want to require some monkey pushing JMX after every long GC.

On 31 Oct 2013, at 07:18, Bela Ban <bban(a)redhat.com&gt; wrote: > On 10/30/13 8:28 PM, William Burns wrote: > Since it seems I can't comment on the wiki itself, I am just replying here. > > I wonder if the third option 'Primary partition' is desirable. I > think availability in some cases would be harmed more than we would > like. > > Lets say you have a 5 node cluster where 3 of the nodes are behind the > same router and the remaining 2 are behind a different one. If the > router crashes, power loss etc. for the 3 and are no longer > addressable you have your 2 partitions (possibly 1 or even 4). When > this occurs the other 2 nodes would go into read only mode since they > lost the quorum check. Yes, this is intended. Actually, the minority partition {D,E} might even become totally inaccessible, ie. rejecting *all* requests (also reads). This is in line with the Primary Partition approach where a majority partition is allowed to make progress, and all minority partitions shut down. In terms of CAP, we're sacrificing availabilty here in favor of consistency. > But the 3 nodes that are "writable" can't be > accessed any longer and thus no writes can be performed on the cluster. You mean some clients cannot access {A,B,C} ? Sure, then so be it, but at least we don't have any inconsistent state. Again, PP is *one* tool we give to th user to handle partitions. > It seems we would still want to allow writes to provide as > high of availability as possible. PP is *not* about availability, it is about consistency.

Good for some apps, bad for others. If you pick PP, you lose availability. > Also if we did have read only, what criteria would cause those nodes > to be writeable again? Once you become the primary partition, e.g. when a view is received where view.size() >= N where N is a predefined threshold. Can be different, as long as it is deterministic. > There is no guarantee when the other nodes > will ever come back up or if there will ever be additional ones anytime soon. If a system picks the Primary Partition approach, then it can become completely inaccessible (read-only). In this case, I envisage that a sysadmin will be notified, who can then start additional nodes for the system to acquire primary partition and become accessible again. -- Bela Ban, JGroups lead (http://www.jgroups.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

>>> On 10/30/13 8:28 PM, William Burns wrote: Since it seems I can't >>> comment on the wiki itself, I am just replying here. >>> >>> I wonder if the third option 'Primary partition' is desirable. >>> I think availability in some cases would be harmed more than we >>> would like. >>> >>> Lets say you have a 5 node cluster where 3 of the nodes are >>> behind the same router and the remaining 2 are behind a different >>> one. If the router crashes, power loss etc. for the 3 and are no >>> longer addressable you have your 2 partitions (possibly 1 or even >>> 4). When this occurs the other 2 nodes would go into read only >>> mode since they lost the quorum check. >> >> Yes, this is intended. Actually, the minority partition {D,E} might >> even become totally inaccessible, ie. rejecting *all* requests >> (also reads). >> >> This is in line with the Primary Partition approach where a >> majority partition is allowed to make progress, and all minority >> partitions shut down. In terms of CAP, we're sacrificing >> availabilty here in favor of consistency. >> >>> But the 3 nodes that are "writable" can't be accessed any longer >>> and thus no writes can be performed on the cluster. >> >> You mean some clients cannot access {A,B,C} ? Sure, then so be it, >> but at least we don't have any inconsistent state. Again, PP is >> *one* tool we give to th user to handle partitions. >> >>> It seems we would still want to allow writes to provide as high >>> of availability as possible. >> >> PP is *not* about availability, it is about consistency. > > I think it's about availability as well, as the primary partition is still available. Note that with a Primary Partition approach, *no* partition might be the primary partition and thus availablity would be impacted.

On 10/31/2013 02:18 AM, Bela Ban wrote: > >> Also if we did have read only, what criteria would cause those nodes >> to be writeable again? > Once you become the primary partition, e.g. when a view is received > where view.size() >= N where N is a predefined threshold. Can be > different, as long as it is deterministic. > >> There is no guarantee when the other nodes >> will ever come back up or if there will ever be additional ones anytime soon. > If a system picks the Primary Partition approach, then it can become > completely inaccessible (read-only). In this case, I envisage that a > sysadmin will be notified, who can then start additional nodes for the > system to acquire primary partition and become accessible again. There should be a way to manually modify the primary partition status. So if the admin knows the nodes will never return, they can manually enable the partition.

Also, the PartitionContext should know whether the nodes left normally or not. If you have 5 nodes in a cluster, and you shut down 3 of them, you'll want the remaining two to remain available. But if there was a network partition, you wouldn't. So it needs to know the difference.

On 31 October 2013 20:07, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On Oct 31, 2013, at 3:45 PM, Dennis Reed <dereed(a)redhat.com&gt; wrote: > >> On 10/31/2013 02:18 AM, Bela Ban wrote: >>> >>>> Also if we did have read only, what criteria would cause those nodes >>>> to be writeable again? >>> Once you become the primary partition, e.g. when a view is received >>> where view.size() >= N where N is a predefined threshold. Can be >>> different, as long as it is deterministic. >>> >>>> There is no guarantee when the other nodes >>>> will ever come back up or if there will ever be additional ones anytime soon. >>> If a system picks the Primary Partition approach, then it can become >>> completely inaccessible (read-only). In this case, I envisage that a >>> sysadmin will be notified, who can then start additional nodes for the >>> system to acquire primary partition and become accessible again. >> >> There should be a way to manually modify the primary partition status. >> So if the admin knows the nodes will never return, they can manually >> enable the partition. > > The status will be exposed through JMX at any point, disregarding if there's a split brain going on or not. > >> >> Also, the PartitionContext should know whether the nodes left normally >> or not. >> If you have 5 nodes in a cluster, and you shut down 3 of them, you'll >> want the remaining two to remain available. >> But if there was a network partition, you wouldn't. So it needs to know >> the difference. > > very good point again. > Thank you Dennis! Let's clarify. If 3 nodes out of 5 are killed without a reconfiguration, you do NOT want the remaining two to remain available unless explicitly told so by an admin. It is not possible to automatically make a distinction between 3 nodes being shut down vs. 3 crashed nodes.

In our face to face meeting we did point out that an admin needs hooks to be able to: - specify how many nodes are expected in the full system (and adapt dynamically) - some admin command to "clean shutdown" a node (which was also discussed as a strong requirement in scope of CacheStores so I'm assuming the operation is defined already) The design Wiki has captured the API we discussed around the PartitionHandlingStrategy but is missing the details about these operations, that should probably be added to the PartitionContext as well. Also in the scope of CacheStore consistency we had discussed the need to store the expected nodes to be in the View: for example when the grid is started and all nodes are finding each other, the Cache shall not be considered started until all required nodes have joined.

On 31 October 2013 20:07, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On Oct 31, 2013, at 3:45 PM, Dennis Reed <dereed(a)redhat.com&gt; wrote: > >> On 10/31/2013 02:18 AM, Bela Ban wrote: >>> >>>> Also if we did have read only, what criteria would cause those nodes >>>> to be writeable again? >>> Once you become the primary partition, e.g. when a view is received >>> where view.size() >= N where N is a predefined threshold. Can be >>> different, as long as it is deterministic. >>> >>>> There is no guarantee when the other nodes >>>> will ever come back up or if there will ever be additional ones anytime soon. >>> If a system picks the Primary Partition approach, then it can become >>> completely inaccessible (read-only). In this case, I envisage that a >>> sysadmin will be notified, who can then start additional nodes for the >>> system to acquire primary partition and become accessible again. >> >> There should be a way to manually modify the primary partition status. >> So if the admin knows the nodes will never return, they can manually >> enable the partition. > > The status will be exposed through JMX at any point, disregarding if there's a split brain going on or not. > >> >> Also, the PartitionContext should know whether the nodes left normally >> or not. >> If you have 5 nodes in a cluster, and you shut down 3 of them, you'll >> want the remaining two to remain available. >> But if there was a network partition, you wouldn't. So it needs to know >> the difference. > > very good point again. > Thank you Dennis! Let's clarify. If 3 nodes out of 5 are killed without a reconfiguration, you do NOT want the remaining two to remain available unless explicitly told so by an admin. It is not possible to automatically make a distinction between 3 nodes being shut down vs. 3 crashed nodes.

In our face to face meeting we did point out that an admin needs hooks to be able to: - specify how many nodes are expected in the full system (and adapt dynamically)

- some admin command to "clean shutdown" a node (which was also discussed as a strong requirement in scope of CacheStores so I'm assuming the operation is defined already) The design Wiki has captured the API we discussed around the PartitionHandlingStrategy but is missing the details about these operations, that should probably be added to the PartitionContext as well.

Also in the scope of CacheStore consistency we had discussed the need to store the expected nodes to be in the View: for example when the grid is started and all nodes are finding each other, the Cache shall not be considered started until all required nodes have joined.

Cheers, Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 1 Nov 2013, at 16:05, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > On 1 November 2013 11:56, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > >> On Oct 31, 2013, at 10:20 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: >> >>> On 31 October 2013 20:07, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>> >>>> On Oct 31, 2013, at 3:45 PM, Dennis Reed <dereed(a)redhat.com&gt; wrote: >>>> >>>>> On 10/31/2013 02:18 AM, Bela Ban wrote: >>>>> >>>>>> Also if we did have read only, what criteria would cause those nodes >>>>>> to be writeable again? >>>>> Once you become the primary partition, e.g. when a view is received >>>>> where view.size() >= N where N is a predefined threshold. Can be >>>>> different, as long as it is deterministic. >>>>> >>>>>> There is no guarantee when the other nodes >>>>>> will ever come back up or if there will ever be additional ones anytime soon. >>>>> If a system picks the Primary Partition approach, then it can become >>>>> completely inaccessible (read-only). In this case, I envisage that a >>>>> sysadmin will be notified, who can then start additional nodes for the >>>>> system to acquire primary partition and become accessible again. >>>> >>>> There should be a way to manually modify the primary partition status. >>>> So if the admin knows the nodes will never return, they can manually >>>> enable the partition. >>> >>> The status will be exposed through JMX at any point, disregarding if there's a split brain going on or not. >>> >>>> >>>> Also, the PartitionContext should know whether the nodes left normally >>>> or not. >>>> If you have 5 nodes in a cluster, and you shut down 3 of them, you'll >>>> want the remaining two to remain available. >>>> But if there was a network partition, you wouldn't. So it needs to know >>>> the difference. >>> >>> very good point again. >>> Thank you Dennis! >> >> Let's clarify. If 3 nodes out of 5 are killed without a >> reconfiguration, you do NOT want the remaining two to remain available >> unless explicitly told so by an admin. It is not possible to >> automatically make a distinction between 3 nodes being shut down vs. 3 >> crashed nodes. > > I'm not sure you can make this generalization: it's really up to the implementor of PartitionHandlingStrategy to decide that. Knowing whether it was a clean shutdown or not might be relevant to that decision. I think the focus of this functionality should be on how exactly to react to partitions happening, but provide the hooks for the user to make that decision and act on system's availability. We're on the same page on that, I'm just stressing that there is not automatic way that we can make a distinction between crash or intentional shutdown, if we don't have the "clean shutdown" method like Bela also reminded. > >> >> In our face to face meeting we did point out that an admin needs hooks >> to be able to: >> - specify how many nodes are expected in the full system (and adapt >> dynamically) > > yes, that's an custom implementation of PartitionHandlingStrategy. One we might provide out of the box. Right it could be part of the default PartitionHandlingStrategy but I think all strategies might be interested in this, and that it's Infinispan (core) responsibility to also provide ways to admin the expected view at runtime. >> - some admin command to "clean shutdown" a node (which was also >> discussed as a strong requirement in scope of CacheStores so I'm >> assuming the operation is defined already) >> >> The design Wiki has captured the API we discussed around the >> PartitionHandlingStrategy but is missing the details about these >> operations, that should probably be added to the PartitionContext as >> well. > > The PartitionContext allows a partition to be marked as unavailable, I think that should do. You also need the "clean shutdown", very likely with the RPC suggested by Bela. >> Also in the scope of CacheStore consistency we had discussed the need >> to store the expected nodes to be in the View: for example when the >> grid is started and all nodes are finding each other, the Cache shall >> not be considered started until all required nodes have joined. > > the discussion is here: https://community.jboss.org/wiki/ControlledClusterShutdownWithDataRestore... > >> >> Cheers, >> Sanne >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 10/31/13 4:45 PM, Dennis Reed wrote: > On 10/31/2013 02:18 AM, Bela Ban wrote: >> >>> Also if we did have read only, what criteria would cause those nodes >>> to be writeable again? >> Once you become the primary partition, e.g. when a view is received >> where view.size() >= N where N is a predefined threshold. Can be >> different, as long as it is deterministic. >> >>> There is no guarantee when the other nodes >>> will ever come back up or if there will ever be additional ones anytime soon. >> If a system picks the Primary Partition approach, then it can become >> completely inaccessible (read-only). In this case, I envisage that a >> sysadmin will be notified, who can then start additional nodes for the >> system to acquire primary partition and become accessible again. > > There should be a way to manually modify the primary partition status. > So if the admin knows the nodes will never return, they can manually > enable the partition. > > Also, the PartitionContext should know whether the nodes left normally > or not. > If you have 5 nodes in a cluster, and you shut down 3 of them, you'll > want the remaining two to remain available. > But if there was a network partition, you wouldn't. So it needs to know > the difference. JGroups won't tell you, and I don't want to add a flag to each member of a view telling you whether it was a graceful or crash-leave. However, you (Infinispan) could send a LEAVE message shortly before leaving, which is stored by everyone (or only the coord?). When the view is received, we should be able to determine who left and who crashed.

Ups, wrong URL. This is the design for cluster events: https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions On Oct 30, 2013, at 7:05 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > Hi, > > I've added a wiki page[1] capturing our discussions around cluster events. > Any feedback welcomed! > > [1] https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org)

Just to clarify, can you comment on whether the statements below are true ? #1 When mode=repl is used, the event itself is never broadcast; it is always local and no communication is needed

#2 when mode=dist, is the event broadcast synchronously or asynchronously, can this be configured ? I think it doesn't make sense to broadcast the event synchronously in dist-async, for instance.

Regarding #2: if this mechanism is used unwisely, a user can destroy all the advantages (perf) brought by DIST by causing a broadcast for every update. Is there anything to prevent this (e.g. a sanity check at startup) ?

On 10/30/13 8:33 PM, Mircea Markus wrote: > Okay, and now the good URL (3rd try): https://github.com/infinispan/infinispan/wiki/Clustered-listeners > > On Oct 30, 2013, at 7:31 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > >> Ups, wrong URL. >> >> This is the design for cluster events: https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions >> >> >> On Oct 30, 2013, at 7:05 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >> >>> Hi, >>> >>> I've added a wiki page[1] capturing our discussions around cluster events. >>> Any feedback welcomed! >>> >>> [1] https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions >>> >>> Cheers, >>> -- >>> Mircea Markus >>> Infinispan lead (www.infinispan.org) >>> >>> >>> >>> >> Cheers, >> -- >> Mircea Markus >> Infinispan lead (www.infinispan.org) >> >> >> >> > Cheers, >

Just to clarify, can you comment on whether the statements below are true ? #1 When mode=repl is used, the event itself is never broadcast; it is always local and no communication is needed

#2 when mode=dist, is the event broadcast synchronously or asynchronously, can this be configured ? I think it doesn't make sense to broadcast the event synchronously in dist-async, for instance.

Regarding #2: if this mechanism is used unwisely, a user can destroy all the advantages (perf) brought by DIST by causing a broadcast for every update. Is there anything to prevent this (e.g. a sanity check at startup) ?

On 10/31/13 1:05 PM, Mircea Markus wrote: > > On Oct 31, 2013, at 7:10 AM, Bela Ban <bban(a)redhat.com&gt; wrote: > >> Just to clarify, can you comment on whether the statements below are true ? >> >> #1 When mode=repl is used, the event itself is never broadcast; it is >> always local and no communication is needed > > yes. for replicated caches we should handle this as a local listener. > >> >> #2 when mode=dist, is the event broadcast synchronously or >> asynchronously, can this be configured ? I think it doesn't make sense >> to broadcast the event synchronously in dist-async, for instance. > > there are two levels of asynchronicity: > - the cluster is configured to be asyn > - the listeners can be configured to be notified in an async thread (@Listener async=true) > In both situations the notification from the event should be broadcasted asynchronously Ah, ok, so the events are *always* async ?

+1 in general, but there's a point where you mentioned ordering, and ordering might get destroyed if you switch to async.

On 10/30/2013 02:33 PM, Mircea Markus wrote: > Okay, and now the good URL (3rd try): https://github.com/infinispan/infinispan/wiki/Clustered-listeners There are a couple of things missing in the Filter and Convertor: - What type of event? - isPre? For instance, for CacheEntryModified the listener may only want the filter to return true for only the isPre=true event. Or they may want a different converter when isPre=false.

Mircea, I haven't read this fully, but I want to clarify something about the discussion we had on this particular topic, before commenting on it. Didn't we agree when we discussed it with Sanne and Adrian that this would not be a good idea?

Discussing this I came to the conclusion that to build the CEP or CQ functionality, what you'd really want is remote events for Hot Rod, which then led to me explaining what to do with remote Hot Rod events.

Cheers, On Oct 30, 2013, at 8:33 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > Okay, and now the good URL (3rd try): https://github.com/infinispan/infinispan/wiki/Clustered-listeners > > On Oct 30, 2013, at 7:31 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > >> Ups, wrong URL. >> >> This is the design for cluster events: https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions >> >> >> On Oct 30, 2013, at 7:05 PM, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >> >>> Hi, >>> >>> I've added a wiki page[1] capturing our discussions around cluster events. >>> Any feedback welcomed! >>> >>> [1] https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions >>> >>> Cheers, >>> -- >>> Mircea Markus >>> Infinispan lead (www.infinispan.org) >>> >>> >>> >>> >> >> Cheers, >> -- >> Mircea Markus >> Infinispan lead (www.infinispan.org) >> >> >> >> > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Thanks for the wiki and the discussion. Since I'm encountering this in the wild, allow me to offer my thoughts. Basically my feeling is that no matter what you do in split-brain handling, it's going to be wrong. In my use case, I have individual blades where each blade runs a suite of application nodes; one of which is a data grid node. Each node is single-homed. And they wire into the same switch. This setup is orthogonal across a data center (WAN). In this deployment, these two DCs make up a single cluster. There is a concept of a set of keys for my caches being "owned" by a site, i.e. only one set of clients will access these keys. These keys are striped across the WAN with a TACH. So a split brain on a local data center only can occur when a NIC on one of the blades goes bad and the node is still running. The merge will always be of the [subgroups=N-1, 1] variety, where N is the number of running nodes in the cluster. Since these nodes are single-homed, they cannot receive requests if they are "offline" from the NIC. I don't have to worry about state collision, but I DO have to worry about stale state from the merged node.

In this case, it's easy to tell when I might be in a split-brain. The FD protocol will suspect and exclude a node. Currently, though, I have no way of knowing how or why a node was excluded.

If the WAN goes down, I have a rather large problem.First off is detection. If there's an ACL blockage, or worse, a unidirectional outage (i.e. east can see west, but not vice-versa), it takes the cluster a minute (really, about 60 seconds) to figure things out.

One side will have spurious MERGEs, the other side will have leaves from either FD_SOCK or FD.

So again, there's a problem of detection. It's a pretty safe bet, though, that if I don't see nodes from the other data center, I'm in a split brain. But I still can't differentiate between a normal leave, a leave caused by a socket error, or a leave caused by an FD exclusion.

But if a WAN goes down, both sides are available to their respective clients.

Since my use case is such that the keys are unique to each side, all the CRUD operations of a key come from a client on one side of the WAN, even though the primary owner of that key may be on the other side. Ex: East and West are a single cluster. East makes modifications to Key K that has its primary owner in West (suboptimal, but it works). The WAN splits apart. Clients from East continue to make modifications to K. If clients fail over to West, they still are the only ones making modifications to K.

The real problem occurs when the WAN event is over and the clusters merge. The merge handling has no way of knowing where to source K from. So I think we're missing vector clocking.

And taking it back a step, this doesn't have to be a WAN. The use case is: split brain, but clients can only write to one side. I would tend to think that's pretty common. This can be solved with vector clocking (or any other versioning scheme).

In my use case, it would be "pick the latest". I mentioned vector clocking because if you detected a split brain event at time T, and you were trying to merge keys with modifications after this time, this would be some application-level interface that would need to be called. When we talk about controlling availability, I tend to think users really care about when the event started and how to detect it. Availability is a "damned if you do, damned if you don't" scenario, due to the complex networking scenarios you can envision. From my perspective, I really care about "how do I detect it?" Then it becomes "how do I recover?" Just some disparate thoughts, Erik P.S. (Yes, I know about XSite. It has some limitations that necessitated the architecture above) -----Original Message----- From: infinispan-dev-bounces(a)lists.jboss.org [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus Sent: Wednesday, October 30, 2013 3:05 PM To: infinispan -Dev List Subject: [infinispan-dev] design for cluster events (wiki page) Hi, I've added a wiki page[1] capturing our discussions around cluster events. Any feedback welcomed! [1] https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions Cheers,

Thanks for the wiki and the discussion. Since I'm encountering this in the wild, allow me to offer my thoughts. Basically my feeling is that no matter what you do in split-brain handling, it's going to be wrong. In my use case, I have individual blades where each blade runs a suite of application nodes; one of which is a data grid node. Each node is single-homed. And they wire into the same switch. This setup is orthogonal across a data center (WAN). In this deployment, these two DCs make up a single cluster. There is a concept of a set of keys for my caches being "owned" by a site, i.e. only one set of clients will access these keys. These keys are striped across the WAN with a TACH. So a split brain on a local data center only can occur when a NIC on one of the blades goes bad and the node is still running. The merge will always be of the [subgroups=N-1, 1] variety, where N is the number of running nodes in the cluster. Since these nodes are single-homed, they cannot receive requests if they are "offline" from the NIC. I don't have to worry about state collision, but I DO have to worry about stale state from the merged node. In this case, it's easy to tell when I might be in a split-brain. The FD protocol will suspect and exclude a node. Currently, though, I have no way of knowing how or why a node was excluded. If the WAN goes down, I have a rather large problem. First off is detection. If there's an ACL blockage, or worse, a unidirectional outage (i.e. east can see west, but not vice-versa), it takes the cluster a minute (really, about 60 seconds) to figure things out. One side will have spurious MERGEs, the other side will have leaves from either FD_SOCK or FD. So again, there's a problem of detection. It's a pretty safe bet, though, that if I don't see nodes from the other data center, I'm in a split brain. But I still can't differentiate between a normal leave, a leave caused by a socket error, or a leave caused by an FD exclusion. But if a WAN goes down, both sides are available to their respective clients. Since my use case is such that the keys are unique to each side, all the CRUD operations of a key come from a client on one side of the WAN, even though the primary owner of that key may be on the other side. Ex: East and West are a single cluster. East makes modifications to Key K that has its primary owner in West (suboptimal, but it works). The WAN splits apart. Clients from East continue to make modifications to K. If clients fail over to West, they still are the only ones making modifications to K. The real problem occurs when the WAN event is over and the clusters merge. The merge handling has no way of knowing where to source K from. So I think we're missing vector clocking. And taking it back a step, this doesn't have to be a WAN. The use case is: split brain, but clients can only write to one side. I would tend to think that's pretty common. This can be solved with vector clocking (or any other versioning scheme). In my use case, it would be "pick the latest". I mentioned vector clocking because if you detected a split brain event at time T, and you were trying to merge keys with modifications after this time, this would be some application-level interface that would need to be called. When we talk about controlling availability, I tend to think users really care about when the event started and how to detect it. Availability is a "damned if you do, damned if you don't" scenario, due to the complex networking scenarios you can envision. From my perspective, I really care about "how do I detect it?" Then it becomes "how do I recover?" Just some disparate thoughts, Erik P.S. (Yes, I know about XSite. It has some limitations that necessitated the architecture above)