2:29 a.m.
The other day I had an idea how we could use Infinispan across data centers:
Say we have 2 data centers: NYC and SFO. There are nodes A,B,C,D in NYC
and W,X,Y,Z in SFO.
We'd like to use Infinispan in DIST mode with numOwners == 2, and
ideally place 1 copy of a data in the NYC center and one in the SFO
center. (Of course, this would also work with numOwners > 2, we'd just
have to make sure the copies are distributed more or less evenly between
the 2 data centers).
To make this work, there are 2 things that need to be done:
1. The nodes in NYC and SFO have to be in the same JGroups cluster.
This way Infinispan has insight into the membership and gets
notified of changes. The cluster would usually be TCP based, but
if there's multicasting available between the centers, then of
course multicasting could be used, too.
2. The nodes in NYC would have to have a logical name starting with
"nyc-", the ones in SFO "sfo-".
#2 is needed by Infinispan as input into its consistent hash function in
order to determine the nodes to store a given key/value. Say if we have
a view of
{nyc-D, sfo-Y, sfo-W, nyc-B, nyc-C, sfo-Z, nyc-A, sfo-X}, then we could
come up with a consistent hash function which knows that there are 2
data centers and could place the nodes onto 2 different hash wheels and
pick one from each (if numOwners=2).
Yes, I know #1 sucks, but until we've implemented the more advanced
solutions (e.g. RELAY in JGroups, data center replication in
Infinispan), this seems to me a workable solution that could be
implemented in a short time.
WDYT ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
4:45 a.m.
On top of the issues you mentioned, you force all to have same communications layer, TCP
or UDP.
Did you have a look at my Hot Rod presentation for JUDCon?
http://www.jboss.org/events/JUDCon/presentations.html. You might check out the last bit of
my presentation where I explained very briefly how Infinispan accross data centres could
potentially be written with RemoteCacheStore hooking up different data centres via Hot
Rod. Each dc could be configured with JGroups TCP/UDP while the inter dc traffic goes over
Netty based TCP-NIO. An active/passive kind of layout would be relatively easy to achieve
this way.
On Aug 5, 2010, at 9:29 AM, Bela Ban wrote:
The other day I had an idea how we could use Infinispan across data
centers:
Say we have 2 data centers: NYC and SFO. There are nodes A,B,C,D in NYC
and W,X,Y,Z in SFO.
We'd like to use Infinispan in DIST mode with numOwners == 2, and
ideally place 1 copy of a data in the NYC center and one in the SFO
center. (Of course, this would also work with numOwners > 2, we'd just
have to make sure the copies are distributed more or less evenly between
the 2 data centers).
To make this work, there are 2 things that need to be done:
1. The nodes in NYC and SFO have to be in the same JGroups cluster.
This way Infinispan has insight into the membership and gets
notified of changes. The cluster would usually be TCP based, but
if there's multicasting available between the centers, then of
course multicasting could be used, too.
2. The nodes in NYC would have to have a logical name starting with
"nyc-", the ones in SFO "sfo-".
#2 is needed by Infinispan as input into its consistent hash function in
order to determine the nodes to store a given key/value. Say if we have
a view of
{nyc-D, sfo-Y, sfo-W, nyc-B, nyc-C, sfo-Z, nyc-A, sfo-X}, then we could
come up with a consistent hash function which knows that there are 2
data centers and could place the nodes onto 2 different hash wheels and
pick one from each (if numOwners=2).
Yes, I know #1 sucks, but until we've implemented the more advanced
solutions (e.g. RELAY in JGroups, data center replication in
Infinispan), this seems to me a workable solution that could be
implemented in a short time.
WDYT ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
5:17 a.m.
On 5 Aug 2010, at 10:45, Galder Zamarreño wrote:
On top of the issues you mentioned, you force all to have same
communications layer, TCP or UDP.
Did you have a look at my Hot Rod presentation for JUDCon?
http://www.jboss.org/events/JUDCon/presentations.html. You might check out the last bit of
my presentation where I explained very briefly how Infinispan accross data centres could
potentially be written with RemoteCacheStore hooking up different data centres via Hot
Rod. Each dc could be configured with JGroups TCP/UDP while the inter dc traffic goes over
Netty based TCP-NIO. An active/passive kind of layout would be relatively easy to achieve
this way.
Right, that's active/passive. I think Bela's referring to active/active.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
6:02 a.m.
Galder Zamarreño wrote:
On top of the issues you mentioned, you force all to have same
communications layer, TCP or UDP.
Correct
Did you have a look at my Hot Rod presentation for JUDCon?
http://www.jboss.org/events/JUDCon/presentations.html. You might check
out the last bit of my presentation where I explained very briefly how
Infinispan accross data centres could potentially be written with
RemoteCacheStore hooking up different data centres via Hot Rod. Each
dc could be configured with JGroups TCP/UDP while the inter dc traffic
goes over Netty based TCP-NIO. An active/passive kind of layout would
be relatively easy to achieve this way.
Yes, this would probably work, too. However, there are a few drawbacks:
* You have to configure an initial list of Hot Rod end points (in
the example, in Bilbao). I guess this is minor though; once
connected, clients get updated with the new membership list
* When the coordinator in Bilbao crashes, someone else has to take
over. During this transition you have to ensure that there are no
lost or duplicate messages. This can be done with queuing and
sequence numbers, plus retransmission, but then again, is this
something you want to do in Hot Rod ?
* Assume Boston is up and running, then Bilbao is started. How does
Bilbao get the initial state of Boston ?
An unrelated question I have is what happens when the switch between
Boston and Bilbao crashes ? I assume Hot Rod requires TCP underneath, so
does it (or Netty) implement some heartbeating over the link ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
2:59 a.m.
On Aug 5, 2010, at 1:02 PM, Bela Ban wrote:
Galder Zamarreño wrote:
> On top of the issues you mentioned, you force all to have same
> communications layer, TCP or UDP.
Correct
> Did you have a look at my Hot Rod presentation for JUDCon?
> http://www.jboss.org/events/JUDCon/presentations.html. You might check
> out the last bit of my presentation where I explained very briefly how
> Infinispan accross data centres could potentially be written with
> RemoteCacheStore hooking up different data centres via Hot Rod. Each
> dc could be configured with JGroups TCP/UDP while the inter dc traffic
> goes over Netty based TCP-NIO. An active/passive kind of layout would
> be relatively easy to achieve this way.
Yes, this would probably work, too. However, there are a few drawbacks:
* You have to configure an initial list of Hot Rod end points (in
the example, in Bilbao). I guess this is minor though; once
connected, clients get updated with the new membership list
Indeed
* When the coordinator in Bilbao crashes, someone else has to
take
over. During this transition you have to ensure that there are no
lost or duplicate messages. This can be done with queuing and
sequence numbers, plus retransmission, but then again, is this
something you want to do in Hot Rod ?
I don't think there's a need to worry about this. The client side, Boston, can
keep track of a number of Hot Rod servers in Bilbao. IOW, you can have all nodes in Bilbao
running Hot Rod servers. So, if the first one fails, it sends it to the second one. I
don't see potential for loss/duplicate at all. It's the client driving it and as
long as the servers are there and the client has the view, it's all fine.
* Assume Boston is up and running, then Bilbao is started. How
does
Bilbao get the initial state of Boston ?
Yup, that's missing. I think you'd need Bilbao to be Hot Rod client to Boston and
do a bulk get.
An unrelated question I have is what happens when the switch between
Boston and Bilbao crashes ? I assume Hot Rod requires TCP underneath, so
does it (or Netty) implement some heartbeating over the link ?
I'm not aware of Netty sending any hearbeats, but Hot Rod has a ping command which can
be used for that.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
2:13 a.m.
Galder Zamarreño wrote:
> * When the coordinator in Bilbao crashes, someone else has to take
> over. During this transition you have to ensure that there are no
> lost or duplicate messages. This can be done with queuing and
> sequence numbers, plus retransmission, but then again, is this
> something you want to do in Hot Rod ?
I don't think there's a need to worry about this. The client side,
Boston, can keep track of a number of Hot Rod servers in Bilbao. IOW,
you can have all nodes in Bilbao running Hot Rod servers. So, if the
first one fails, it sends it to the second one. I don't see potential
for loss/duplicate at all. It's the client driving it and as long as
the servers are there and the client has the view, it's all fine.
What happens if a client thinks the first server failed, and start
forwarding messages to the second server, but the first server still did
receive a few messages ? Then we could end up with duplicate messages...
Also, the clients can have different 'views' of the cluster, due to the
ping protocol having different latencies across different network links.
This way, we can end up with different views thus the consistent hash
function can pick different servers for a given key.
Definitely worth a design / brainstorming session once we start tackling
distribution across data centers...
> * Assume Boston is up and running, then Bilbao is started. How
does
> Bilbao get the initial state of Boston ?
Yup, that's missing. I think you'd need Bilbao to be Hot Rod client to
Boston and do a bulk get.
Probably something equivalent to the non-blocking state transfer already
present in JBossCache / Infinispan...
> An unrelated question I have is what happens when the switch
between
> Boston and Bilbao crashes ? I assume Hot Rod requires TCP underneath, so
> does it (or Netty) implement some heartbeating over the link ?
I'm not aware of Netty sending any hearbeats, but Hot Rod has a ping
command which can be used for that.
Still another point to consider.
My overall argument is that, while HotRod is fine to access servers from
clients, it needs work to connect servers or server islands. Or it may
not be the right tool to do that, and JGroups serves you better in
solving issues like the ones mentioned above.
I'll be looking into issues revolving around wide area data replication
/ distribution on the JGroups level shortly, to see if there's a
solution requiring minimal or no changes in Infinispan. Although I doubt
this, at least for distribution; there an Infinispan centered solution
is probaby better. But we actually do have a design for this, that we
came up with at our last Neuchatel meeting, what about it ? Still relevant ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
6:58 a.m.
On Aug 10, 2010, at 9:13 AM, Bela Ban wrote:
Galder Zamarreño wrote:
>
>> * When the coordinator in Bilbao crashes, someone else has to take
>> over. During this transition you have to ensure that there are no
>> lost or duplicate messages. This can be done with queuing and
>> sequence numbers, plus retransmission, but then again, is this
>> something you want to do in Hot Rod ?
>
> I don't think there's a need to worry about this. The client side,
> Boston, can keep track of a number of Hot Rod servers in Bilbao. IOW,
> you can have all nodes in Bilbao running Hot Rod servers. So, if the
> first one fails, it sends it to the second one. I don't see potential
> for loss/duplicate at all. It's the client driving it and as long as
> the servers are there and the client has the view, it's all fine.
What happens if a client thinks the first server failed, and start
forwarding messages to the second server, but the first server still did
receive a few messages ? Then we could end up with duplicate messages...
Is that possible with client sending over TCP connections? The failure is based on the
inability to send the message over TCP. So, if TCP says that it couldn't send it, I
wonder how the message could have maybe reached. That would break TCP itself, wouldn't
it? In any case, I can only see duplicates having issues with return values. For example,
if you remove a key from the cache and it returns the previous value. If you remove twice
the same key, the second time it would return null.
Also, the clients can have different 'views' of the cluster, due to the
ping protocol having different latencies across different network links.
This way, we can end up with different views thus the consistent hash
function can pick different servers for a given key.
Hmmmm, clients don't use ping to find out what the cluster view. The cluster view is
maintained at the Hot Rod server side, which is passed back to the clients. If anything,
clients might have a stale view, but this will be updated the moment the client sees that
a client has a stale view. I don't see how clients might end up with different views.
Eventually they'll have the same.
Definitely worth a design / brainstorming session once we start tackling
distribution across data centers...
>> * Assume Boston is up and running, then Bilbao is started. How does
>> Bilbao get the initial state of Boston ?
>
> Yup, that's missing. I think you'd need Bilbao to be Hot Rod client to
> Boston and do a bulk get.
Probably something equivalent to the non-blocking state transfer already
present in JBossCache / Infinispan...
>> An unrelated question I have is what happens when the switch between
>> Boston and Bilbao crashes ? I assume Hot Rod requires TCP underneath, so
>> does it (or Netty) implement some heartbeating over the link ?
>
> I'm not aware of Netty sending any hearbeats, but Hot Rod has a ping
> command which can be used for that.
Still another point to consider.
My overall argument is that, while HotRod is fine to access servers from
clients, it needs work to connect servers or server islands. Or it may
not be the right tool to do that, and JGroups serves you better in
solving issues like the ones mentioned above.
I agree that a Hot Rod based solution might not be ideal and has some weaknesses such as
the issue of Bilbao getting state from Boston. However, I think it can get you a
relatively workable solution for an active/passive scenario.
I'll be looking into issues revolving around wide area data replication
/ distribution on the JGroups level shortly, to see if there's a
solution requiring minimal or no changes in Infinispan. Although I doubt
this, at least for distribution; there an Infinispan centered solution
is probaby better. But we actually do have a design for this, that we
came up with at our last Neuchatel meeting, what about it ? Still relevant ?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
5:16 a.m.
On 5 Aug 2010, at 08:29, Bela Ban wrote:
Yes, I know #1 sucks, but until we've implemented the more advanced
solutions (e.g. RELAY in JGroups, data center replication in
Infinispan), this seems to me a workable solution that could be
implemented in a short time.
Would this be usable though? The thing with #1 is that you will force all traffic over a
WAN link. So even if you have 4 owners, 2 in each DC, the "local" replication,
there will be no way to write to the local backup synchronously and the remote backups
asynchronously. I.e., either all comms are sync or all comms are async.
Maybe this is something we can add in Infinispan. I.e., with node hints
(https://jira.jboss.org/browse/ISPN-180) (*) we could detect which recipients are local
and which are remote, and accordingly split the RPC into 2: a sync local RPC and an async
remote RPC.
Cheers
Manik
(*) Node hinting isn't strictly necessary; the naming convention you mentioned earlier
would work in this regard as well, although I think these hints is probably a better
universal solution since we need this for other things anyway.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
6:10 a.m.
Manik Surtani wrote:
On 5 Aug 2010, at 08:29, Bela Ban wrote:
> Yes, I know #1 sucks, but until we've implemented the more advanced
> solutions (e.g. RELAY in JGroups, data center replication in
> Infinispan), this seems to me a workable solution that could be
> implemented in a short time.
Would this be usable though? The thing with #1 is that you will force
all traffic over a WAN link.
Yes. I know of several customers who have set up and run clusters over
WAN links. Most of them use TCP, but I know of 2 customers who run IP
multicasting across data centers over OC-48 fiber.
Until a few years ago, I've always recommened against this (I mean
clusters across WANs), but it seems it works for folks, and so I'll be
working on making this a better choice. Some of the things that will
help are relaying traffic between separate islands (RELAY), daisy
chaining and optimizing traffic across the WAN link etc.
So even if you have 4 owners, 2 in each DC, the "local"
replication,
there will be no way to write to the local backup synchronously and
the remote backups asynchronously. I.e., either all comms are sync or
all comms are async.
Yes, correct. However, this is something which could be implemented in
Infinispan, ie. configure a mode (DIST_SYNC_ASYNC ?) where the primary
owner of a key is updated synchronously and the other(s) asynchronously.
Apart from data center replication, this might even be a nice feature to
have in Infinispan proper ?
Maybe this is something we can add in Infinispan. I.e., with node
hints (https://jira.jboss.org/browse/ISPN-180) (*) we could detect
which recipients are local and which are remote, and accordingly split
the RPC into 2: a sync local RPC and an async remote RPC.
+1
(*) Node hinting isn't strictly necessary; the naming convention
you
mentioned earlier would work in this regard as well, although I think
these hints is probably a better universal solution since we need this
for other things anyway.
+1 again
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
6:20 a.m.
On 5 Aug 2010, at 12:10, Bela Ban wrote:
Yes, correct. However, this is something which could be implemented in
Infinispan, ie. configure a mode (DIST_SYNC_ASYNC ?) where the primary
owner of a key is updated synchronously and the other(s) asynchronously.
Apart from data center replication, this might even be a nice feature to
have in Infinispan proper ?
> Maybe this is something we can add in Infinispan. I.e., with node
> hints (https://jira.jboss.org/browse/ISPN-180) (*) we could detect
> which recipients are local and which are remote, and accordingly split
> the RPC into 2: a sync local RPC and an async remote RPC.
+1
Yeah I think this would definitely be a nice-to-have.
See https://jira.jboss.org/browse/ISPN-573
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
5696
days inactive
5721
days old
infinispan-dev@lists.jboss.org
9 comments
3 participants
participants (3)
-
Bela Ban -
Galder Zamarreño -
Manik Surtani