[infinispan-dev] Use cases for x-datacenter replication

[Erik Salter]

Some clarifications:

- All data centers own a copy of the data.  
- +1 is an additional copy at the local data center, while 1 replica at each
of the other data centers.  In the other thread, Bela specified this as a
"numPrimarySiteOwners=2 and numBackupSiteOwners=1" configuration.  
- There are aspects of the custom bridge, like site-aware key affinity,
state transfer of offline data center, etc., that are simpler vs. a virtual
cluster.  But a customer may expect that configuration and management of the
data centers be managed as if it were a virtual cluster.
- My comment about a quorum has to do with the physical network being down
between data centers in a virtual cluster scenario.  In my use case, the
clients would only be able to talk to their local data centers anyway.  If I
have appropriate coverage on my local site (and any other data center I'm
connected to), I don't necessarily want to start a state transfer.  Any
deltas can be pushed to the other sites when they come back online. 

Thanks,

Erik
  
-----Original Message-----
From: infinispan-dev-bounces at lists.jboss.org
[mailto:infinispan-dev-bounces at lists.jboss.org] On Behalf Of Mircea Markus
Sent: Monday, February 13, 2012 1:47 PM
To: infinispan -Dev List
Subject: Re: [infinispan-dev] Use cases for x-datacenter replication



----- Original Message -----
> From: "Erik Salter" <an1310 at hotmail.com>
> To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>
> Sent: Monday, February 13, 2012 6:07:37 PM
> Subject: Re: [infinispan-dev] Use cases for x-datacenter replication
> 
> Hi all,
> 
> Since I'm deploying my applications into 3 data centers, I commented
> at the
> wiki site.  Since the bulk of the discussion seems to be here, I'll
> offer
> some additional thoughts.
> 
> 
> - First and foremost, I have three data centers (JGRP-1313).  All of
> my data
> centers follow a model where the data is partitioned between them
> with
> geographic failover.

how does the backup work in your use case: A backup data to B, B to C and C
to A. Or everybody is a data owner? 

> - These data centers have about 50-70 ms latency (RTT) between them.
>  That
> means I (and the customer) can live with synchronous replication for
> the
> time being, especially for the transactional scenario.  In fact, in
> one of
> my cache use cases, this is almost required.
> - In the current state transfer model, a node joining means that all
> the
> nodes from all the sites are involved in rehashing.  The rehash
> operation
> should be enhanced in this case to prefer using the local nodes as
> state
> providers.  I know that this cannot be necessarily guaranteed, as a
> loss of
> multiple nodes at a site might mean the state only exists on another
> data
> center.  But for a minimum of 10M records, I can't afford the
> existing
> limitations on state transfer.  (And it goes without saying the NBST
> is an
> absolute must)
Indeed that would be the case if we decide to go with the #1 solution
(virtual nodes) as described here:
https://community.jboss.org/wiki/CrossDatacenterReplication-Design  
Othewise the resilience to local failures should be handled through
numOwners. 

> - I will need a custom consistent hash that is data center-aware that
> can
> group keys into a local site.  All of my data is site-specific.
seems like you've decided to implement your use case based onsolution #1. If
#2 or #3 is used you don't have this custom CH problem.  

> - I mentioned a +1 model for local key ownership on the wiki,
"+1" means that main datacenter has numOwners copies and the backup
datacenter has always 1? 
 
  Taking
> that a
> step further, has there been any thought to a quorum model?  Here's
> what I'm
> concerned about.  Most of the time, a data center won't fail --
> rather
> there'll be some intermittent packet loss between the two.
the bridge between the datacenters should take care of retransmission and
fault tolerance of the bridge-ends. I don't think that packet loss is
possible.. 

  If I have
> 2
> owners on the key's local data center and 1 each on the backup sites,
> I may
> not want to rehash if I can't reach a data center.  I can understand
> if a
> prepared tx fails -- those I can retry in application code.
you  shouldn't have to. the bridge should do the retry/retransmission of the
lost packets transparently.
> 
> I'm sure I'll have more =)

Thanks again for your input Erik!
_______________________________________________
infinispan-dev mailing list
infinispan-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev