10:16 p.m.
Some clarifications:
- All data centers own a copy of the data.
- +1 is an additional copy at the local data center, while 1 replica at each
of the other data centers. In the other thread, Bela specified this as a
"numPrimarySiteOwners=2 and numBackupSiteOwners=1" configuration.
- There are aspects of the custom bridge, like site-aware key affinity,
state transfer of offline data center, etc., that are simpler vs. a virtual
cluster. But a customer may expect that configuration and management of the
data centers be managed as if it were a virtual cluster.
- My comment about a quorum has to do with the physical network being down
between data centers in a virtual cluster scenario. In my use case, the
clients would only be able to talk to their local data centers anyway. If I
have appropriate coverage on my local site (and any other data center I'm
connected to), I don't necessarily want to start a state transfer. Any
deltas can be pushed to the other sites when they come back online.
Thanks,
Erik
-----Original Message-----
From: infinispan-dev-bounces(a)lists.jboss.org
[mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Mircea Markus
Sent: Monday, February 13, 2012 1:47 PM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] Use cases for x-datacenter replication
----- Original Message -----
From: "Erik Salter" <an1310(a)hotmail.com>
To: "infinispan -Dev List" <infinispan-dev(a)lists.jboss.org>
Sent: Monday, February 13, 2012 6:07:37 PM
Subject: Re: [infinispan-dev] Use cases for x-datacenter replication
Hi all,
Since I'm deploying my applications into 3 data centers, I commented
at the
wiki site. Since the bulk of the discussion seems to be here, I'll
offer
some additional thoughts.
- First and foremost, I have three data centers (JGRP-1313). All of
my data
centers follow a model where the data is partitioned between them
with
geographic failover.
how does the backup work in your use case: A backup data to B, B to C and C
to A. Or everybody is a data owner?
- These data centers have about 50-70 ms latency (RTT) between them.
That
means I (and the customer) can live with synchronous replication for
the
time being, especially for the transactional scenario. In fact, in
one of
my cache use cases, this is almost required.
- In the current state transfer model, a node joining means that all
the
nodes from all the sites are involved in rehashing. The rehash
operation
should be enhanced in this case to prefer using the local nodes as
state
providers. I know that this cannot be necessarily guaranteed, as a
loss of
multiple nodes at a site might mean the state only exists on another
data
center. But for a minimum of 10M records, I can't afford the
existing
limitations on state transfer. (And it goes without saying the NBST
is an
absolute must)
Indeed that would be the case if we decide to go with the #1
solution
(virtual nodes) as described here:
https://community.jboss.org/wiki/CrossDatacenterReplication-Design
Othewise the resilience to local failures should be handled through
numOwners.
- I will need a custom consistent hash that is data center-aware
that
can
group keys into a local site. All of my data is site-specific.
seems like
you've decided to implement your use case based onsolution #1. If
#2 or #3 is used you don't have this custom CH problem.
- I mentioned a +1 model for local key ownership on the wiki,
"+1" means that main datacenter has numOwners copies and the backup
datacenter has always 1?
Taking
that a
step further, has there been any thought to a quorum model? Here's
what I'm
concerned about. Most of the time, a data center won't fail --
rather
there'll be some intermittent packet loss between the two.
the bridge between
the datacenters should take care of retransmission and
fault tolerance of the bridge-ends. I don't think that packet loss is
possible..
If I have
2
owners on the key's local data center and 1 each on the backup sites,
I may
not want to rehash if I can't reach a data center. I can understand
if a
prepared tx fails -- those I can retry in application code.
you shouldn't have
to. the bridge should do the retry/retransmission of the
lost packets transparently.
I'm sure I'll have more =)
Thanks again for your input Erik!
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev