[infinispan-dev] design for cluster events (wiki page)

[Erik Salter]

Thanks for the wiki and the discussion.  Since I'm encountering this in the
wild, allow me to offer my thoughts.

Basically my feeling is that no matter what you do in split-brain handling,
it's going to be wrong.

In my use case, I have individual blades where each blade runs a suite of
application nodes; one of which is a data grid node.  Each node is
single-homed.  And they wire into the same switch.  This setup is orthogonal
across a data center (WAN).  In this deployment, these two DCs make up a
single cluster.  There is a concept of a set of keys for my caches being
"owned" by a site, i.e. only one set of clients will access these keys.
These keys are striped across the WAN with a TACH.

So a split brain on a local data center only can occur when a NIC on one of
the blades goes bad and the node is still running.  The merge will always be
of the [subgroups=N-1, 1] variety, where N is the number of running nodes in
the cluster.  Since these nodes are single-homed, they cannot receive
requests if they are "offline" from the NIC.  I don't have to worry about
state collision, but I DO have to worry about stale state from the merged
node.

In this case, it's easy to tell when I might be in a split-brain.  The FD
protocol will suspect and exclude a node.  Currently, though, I have no way
of knowing how or why a node was excluded.

If the WAN goes down, I have a rather large problem.  First off is
detection.  If there's an ACL blockage, or worse, a unidirectional outage
(i.e. east can see west, but not vice-versa), it takes the cluster a minute
(really, about 60 seconds) to figure things out.  One side will have
spurious MERGEs, the other side will have leaves from either FD_SOCK or FD.
So again, there's a problem of detection.  It's a pretty safe bet, though,
that if I don't see nodes from the other data center, I'm in a split brain.
But I still can't differentiate between a normal leave, a leave caused by a
socket error, or a leave caused by an FD exclusion.

But if a WAN goes down, both sides are available to their respective
clients.

Since my use case is such that the keys are unique to each side,  all the
CRUD operations of a key come from a client on one side of the WAN, even
though the primary owner of that key may be on the other side.  Ex: East and
West are a single cluster.  East makes modifications to Key K that has its
primary owner in West (suboptimal, but it works).  The WAN splits apart.
Clients from East continue to make modifications to K.  If clients fail over
to West, they still are the only ones making modifications to K.

The real problem occurs when the WAN event is over and the clusters merge.
The merge handling has no way of knowing where to source K from.  

So I think we're missing vector clocking.

And taking it back a step, this doesn't have to be a WAN.  The use case is:
split brain, but clients can only write to one side.  I would tend to think
that's pretty common.  This can be solved with vector clocking (or any other
versioning scheme).  In my use case, it would be "pick the latest".  I
mentioned vector clocking because if you detected a split brain event at
time T, and you were trying to merge keys with modifications after this
time, this would be some application-level interface that would need to be
called.

When we talk about controlling availability, I tend to think users really
care about when the event started and how to detect it.   Availability is a
"damned if you do, damned if you don't" scenario, due to the complex
networking scenarios you can envision.  From my perspective, I really care
about "how do I detect it?"  Then it becomes "how do I recover?"  

Just some disparate thoughts,

Erik

P.S.  (Yes, I know about XSite.  It has some limitations that necessitated
the architecture above)


-----Original Message-----
From: infinispan-dev-bounces at lists.jboss.org
[mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Mircea Markus
Sent: Wednesday, October 30, 2013 3:05 PM
To: infinispan -Dev List
Subject: [infinispan-dev] design for cluster events (wiki page)

Hi,

I've added a wiki page[1] capturing our discussions around cluster events. 
Any feedback welcomed!  

[1]
https://github.com/infinispan/infinispan/wiki/Handling-cluster-partitions

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)




_______________________________________________
infinispan-dev mailing list
infinispan-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev