[infinispan-dev] Use of Inifnispan across data centers

[Galder Zamarreño]

On Aug 10, 2010, at 9:13 AM, Bela Ban wrote:

> 
> 
> Galder Zamarreño wrote:
>> 
>>> * When the coordinator in Bilbao crashes, someone else has to take
>>> over. During this transition you have to ensure that there are no
>>> lost or duplicate messages. This can be done with queuing and
>>> sequence numbers, plus retransmission, but then again, is this
>>> something you want to do in Hot Rod ?
>> 
>> I don't think there's a need to worry about this. The client side, 
>> Boston, can keep track of a number of Hot Rod servers in Bilbao. IOW, 
>> you can have all nodes in Bilbao running Hot Rod servers. So, if the 
>> first one fails, it sends it to the second one. I don't see potential 
>> for loss/duplicate at all. It's the client driving it and as long as 
>> the servers are there and the client has the view, it's all fine.
> 
> What happens if a client thinks the first server failed, and start 
> forwarding messages to the second server, but the first server still did 
> receive a few messages ? Then we could end up with duplicate messages...

Is that possible with client sending over TCP connections? The failure is based on the inability to send the message over TCP. So, if TCP says that it couldn't send it, I wonder how the message could have maybe reached. That would break TCP itself, wouldn't it? In any case, I can only see duplicates having issues with return values. For example, if you remove a key from the cache and it returns the previous value. If you remove twice the same key, the second time it would return null.

> 
> Also, the clients can have different 'views' of the cluster, due to the 
> ping protocol having different latencies across different network links. 
> This way, we can end up with different views thus the consistent hash 
> function can pick different servers for a given key.

Hmmmm, clients don't use ping to find out what the cluster view. The cluster view is maintained at the Hot Rod server side, which is passed back to the clients. If anything, clients might have a stale view, but this will be updated the moment the client sees that a client has a stale view. I don't see how clients might end up with different views. Eventually they'll have the same.

> 
> Definitely worth a design / brainstorming session once we start tackling 
> distribution across data centers...
> 
> 
>>> * Assume Boston is up and running, then Bilbao is started. How does
>>> Bilbao get the initial state of Boston ?
>> 
>> Yup, that's missing. I think you'd need Bilbao to be Hot Rod client to 
>> Boston and do a bulk get.
> 
> Probably something equivalent to the non-blocking state transfer already 
> present in JBossCache / Infinispan...
> 
>>> An unrelated question I have is what happens when the switch between
>>> Boston and Bilbao crashes ? I assume Hot Rod requires TCP underneath, so
>>> does it (or Netty) implement some heartbeating over the link ?
>> 
>> I'm not aware of Netty sending any hearbeats, but Hot Rod has a ping 
>> command which can be used for that.
> 
> Still another point to consider.
> 
> My overall argument is that, while HotRod is fine to access servers from 
> clients, it needs work to connect servers or server islands. Or it may 
> not be the right tool to do that, and JGroups serves you better in 
> solving issues like the ones mentioned above.

I agree that a Hot Rod based solution might not be ideal and has some weaknesses such as the issue of Bilbao getting state from Boston. However, I think it can get you a relatively workable solution for an active/passive scenario.  

> 
> I'll be looking into issues revolving around wide area data replication 
> / distribution on the JGroups level shortly, to see if there's a 
> solution requiring minimal or no changes in Infinispan. Although I doubt 
> this, at least for distribution; there an Infinispan centered solution 
> is probaby better. But we actually do have a design for this, that we 
> came up with at our last Neuchatel meeting, what about it ? Still relevant ?
> 
> -- 
> Bela Ban
> Lead JGroups / Clustering Team
> JBoss
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache