1:21 a.m.
Hi,
Here is some communication between loleary and me on the topic of supporting unicast for
Teiid server clusters by default, please read it and let me know you opinion. If you do
not want read the whole thing it can be summarized as
The user would have those three deployment options. Our default would be Unicast over TCP
using an embedded router that supports fail-over. Option two would be Multicast over UDP
(what we do now by default) and option three would be Unicast over UDP or TCP using
GossipRouter (like we offer now).
Thanks
Ramesh..
(10:16:45 AM) ramesh1: I am looking into unicast; and I remember you saying this needs to
be default
(10:17:24 AM) ramesh1: also, there are two types of unicast JGroups provides UDP based and
TCP based
(10:17:42 AM) ramesh1: both requires some kind of gossip router
(10:17:53 AM) loleary: I figured as much.
(10:18:05 AM) ramesh1: JGroups recommends UDP in LAN and TCP in WAN
(10:18:21 AM) loleary: Sounds right.
(10:18:26 AM) loleary: We should use UDP by defult.
(10:18:29 AM) ramesh1: also, when unicast the traffic for each message n(n-1)
(10:18:39 AM) ramesh1: which is understandable
(10:19:01 AM) ramesh1: so, do customers do not like to use TCP?
(10:19:29 AM) loleary: I can't say they do not like to use TCP. I think TCP would be
most common but also adds overhead.
(10:19:44 AM) ramesh1: We were also thinking, how to get rid of the separate gossip
router
(10:20:06 AM) loleary: Is there a way we can embed gossip router into the host controller?
(10:20:40 AM) ramesh1: if we can assume the availability of gossip router some where like
on port x + range then we can embed in some process
(10:20:42 AM) loleary: I believe, with the latest release of JGroups, GossipRouter now
supports fail-over (maybe load balancing but not sure).
(10:21:03 AM) ramesh1: they call it TCPGOSSIP, which is only on TCP
(10:21:33 AM) loleary: Well, when the embedded GossipRouter starts, can't it just
write its listening port information to the configuration?
(10:21:44 AM) loleary: Ahh..
(10:22:17 AM) ramesh1: it could, but that may be error prone as system state
(10:22:43 AM) ramesh1: i.e. when system shuts down it needs to remove it self
(10:22:54 AM) ramesh1: may be it can shun itself I think
(10:23:26 AM) ramesh1: also, when using UDP, this notion multiple gossip routers does not
exist
(10:23:36 AM) ramesh1: there has to be only one.
(10:23:56 AM) loleary: Yes. I can see that to be an issue.
(10:24:14 AM) ramesh1: so, I am torn between, what we would want to support by default
(10:24:49 AM) loleary: Well, we know multicast is an issue in most deployments.
(10:25:09 AM) loleary: We know that UDP is preferred for performance.
(10:25:22 AM) ramesh1: Yes, we want unicast by default; multicast only by more config
(10:25:46 AM) loleary: By using Unicast, we introduce GossipRouter.
(10:25:51 AM) ramesh1: and resource consumption; port availability
(10:25:59 AM) loleary: GossipRouter with UDP can only run as a single instance.
(10:26:01 AM) ramesh1: firewall issues
(10:26:28 AM) ramesh1: that is correct
(10:26:39 AM) loleary: So, although I do not like introducing a dedicated administration
server in the cluster, it almost seems that we do not have a choice.
(10:27:19 AM) loleary: A single server must be determined to be the administrative
(primary) server upon start-up if unicast/udp configuration is selected.
(10:28:33 AM) loleary: Now, it would be nice if the primary/administrative server could be
dynamic.
(10:29:01 AM) loleary: In other words, the first server I start in the cluster becomes the
primary. And, the primary server could change at any time.
(10:29:30 AM) ramesh1: so, when the second server comes up it would tunnel?
(10:29:33 AM) loleary: For that to work, each server in the cluster would have to rely on
configuration information/state to determine who is the primary.
(10:30:38 AM) ramesh1: how would we transfer the administrative stuff to one to another
(10:30:49 AM) ramesh1: when primary goes down?
(10:31:20 AM) loleary: One of the remaining servers would take over and update the
configuration.
(10:31:30 AM) ramesh1: I wish UDP supported multiple gossip, then this would not have been
a issue
(10:31:39 AM) loleary: For this to work, we would have to lock/synch the configuration to
prevent another node from doing the same.
(10:31:51 AM) loleary: Agreed.
(10:32:11 AM) ramesh1: but for that to work JGroups channel needs to be recycled with new
info
(10:32:21 AM) loleary: Correct.
(10:32:48 AM) loleary: So each node of the cluster much periodically check to see who is
the primary server (or do it as soon as communication fails with the old primary).
(10:33:21 AM) loleary: Once they determine that the new primary is different from their
existing primary router, they dump their old channels and start/create new ones with the
new primary.
(10:33:34 AM) ramesh1: that may not be possible; may cause loss of data; unless we write
some elaborate code to move the data
(10:33:52 AM) loleary: True.
(10:34:26 AM) loleary: Well, if we are using UDP, do we have message delivery
confirmation/ack?
(10:34:46 AM) ramesh1: you can configure that in the stack
(10:34:58 AM) ramesh1: TCP you do not
(10:35:10 AM) loleary: :)
(10:35:19 AM) loleary: Right... which is why UDP is better for performance.
(10:35:35 AM) loleary: So, by introducing delivery confirmation, we might as well be using
TCP.
(10:35:50 AM) ramesh1: since our typical customer cluster sizes are small; why not use
TCP?
(10:36:10 AM) ramesh1: little overhead, sure but lot more flexibility
(10:36:28 AM) loleary: It is very tempting.
(10:36:59 AM) loleary: Seems like a good solution for the time being.
(10:37:26 AM) loleary: If it is to network intense (which I doubt it would be) we can work
on implementing a usable UDP version in a later release.
(10:38:00 AM) ramesh1: actually, we have UDP now
(10:38:19 AM) ramesh1: with separate gossip server
(10:38:21 AM) loleary: Well yeah... but it uses multicast.
(10:38:29 AM) loleary: True.
(10:38:56 AM) loleary: So, embedded router would be unicast with TCP to support
fail-over.
(10:39:26 AM) ramesh1: so, we can keep what we have; if we do not want that then embedded
router is TCP
(10:39:32 AM) ramesh1: true
(10:39:46 AM) loleary: Or, no router would use multicast with UDP or TCP... and a third
option would be unicast with UDP or TCP using GossipRouter.
(10:39:55 AM) loleary: Right.
(10:39:57 AM) loleary: Sounds good.
(10:41:39 AM) ramesh1: I am leaning towards TCP one; I will get feedback from others
(10:43:04 AM) ramesh1: thanks
(10:43:33 AM) loleary: Right. That is what I am saying... the user would have those three
deployment options. Our default would be Unicast over TCP using an embedded router that
supports fail-over. Option two would be Multicast over UDP (what we do now by default)
and option three would be Unicast over UDP or TCP using GossipRouter (like we offer now).
(10:44:32 AM) ramesh1: yep, got to think about the configuration options, but sounds
good.