Hi, Here is some communication between loleary and me on the topic of supporting unicast for Teiid server clusters by default, please read it and let me know you opinion. If you do not want read the whole thing it can be summarized as The user would have those three deployment options. Our default would be Unicast over TCP using an embedded router that supports fail-over. Option two would be Multicast over UDP (what we do now by default) and option three would be Unicast over UDP or TCP using GossipRouter (like we offer now). Thanks Ramesh.. (10:16:45 AM) ramesh1: I am looking into unicast; and I remember you saying this needs to be default (10:17:24 AM) ramesh1: also, there are two types of unicast JGroups provides UDP based and TCP based (10:17:42 AM) ramesh1: both requires some kind of gossip router (10:17:53 AM) loleary: I figured as much. (10:18:05 AM) ramesh1: JGroups recommends UDP in LAN and TCP in WAN (10:18:21 AM) loleary: Sounds right. (10:18:26 AM) loleary: We should use UDP by defult. (10:18:29 AM) ramesh1: also, when unicast the traffic for each message n(n-1) (10:18:39 AM) ramesh1: which is understandable (10:19:01 AM) ramesh1: so, do customers do not like to use TCP? (10:19:29 AM) loleary: I can't say they do not like to use TCP. I think TCP would be most common but also adds overhead. (10:19:44 AM) ramesh1: We were also thinking, how to get rid of the separate gossip router (10:20:06 AM) loleary: Is there a way we can embed gossip router into the host controller? (10:20:40 AM) ramesh1: if we can assume the availability of gossip router some where like on port x + range then we can embed in some process (10:20:42 AM) loleary: I believe, with the latest release of JGroups, GossipRouter now supports fail-over (maybe load balancing but not sure). (10:21:03 AM) ramesh1: they call it TCPGOSSIP, which is only on TCP (10:21:33 AM) loleary: Well, when the embedded GossipRouter starts, can't it just write its listening port information to the configuration? (10:21:44 AM) loleary: Ahh.. (10:22:17 AM) ramesh1: it could, but that may be error prone as system state (10:22:43 AM) ramesh1: i.e. when system shuts down it needs to remove it self (10:22:54 AM) ramesh1: may be it can shun itself I think (10:23:26 AM) ramesh1: also, when using UDP, this notion multiple gossip routers does not exist (10:23:36 AM) ramesh1: there has to be only one. (10:23:56 AM) loleary: Yes. I can see that to be an issue. (10:24:14 AM) ramesh1: so, I am torn between, what we would want to support by default (10:24:49 AM) loleary: Well, we know multicast is an issue in most deployments. (10:25:09 AM) loleary: We know that UDP is preferred for performance. (10:25:22 AM) ramesh1: Yes, we want unicast by default; multicast only by more config (10:25:46 AM) loleary: By using Unicast, we introduce GossipRouter. (10:25:51 AM) ramesh1: and resource consumption; port availability (10:25:59 AM) loleary: GossipRouter with UDP can only run as a single instance. (10:26:01 AM) ramesh1: firewall issues (10:26:28 AM) ramesh1: that is correct (10:26:39 AM) loleary: So, although I do not like introducing a dedicated administration server in the cluster, it almost seems that we do not have a choice. (10:27:19 AM) loleary: A single server must be determined to be the administrative (primary) server upon start-up if unicast/udp configuration is selected. (10:28:33 AM) loleary: Now, it would be nice if the primary/administrative server could be dynamic. (10:29:01 AM) loleary: In other words, the first server I start in the cluster becomes the primary. And, the primary server could change at any time. (10:29:30 AM) ramesh1: so, when the second server comes up it would tunnel? (10:29:33 AM) loleary: For that to work, each server in the cluster would have to rely on configuration information/state to determine who is the primary. (10:30:38 AM) ramesh1: how would we transfer the administrative stuff to one to another (10:30:49 AM) ramesh1: when primary goes down? (10:31:20 AM) loleary: One of the remaining servers would take over and update the configuration. (10:31:30 AM) ramesh1: I wish UDP supported multiple gossip, then this would not have been a issue (10:31:39 AM) loleary: For this to work, we would have to lock/synch the configuration to prevent another node from doing the same. (10:31:51 AM) loleary: Agreed. (10:32:11 AM) ramesh1: but for that to work JGroups channel needs to be recycled with new info (10:32:21 AM) loleary: Correct. (10:32:48 AM) loleary: So each node of the cluster much periodically check to see who is the primary server (or do it as soon as communication fails with the old primary). (10:33:21 AM) loleary: Once they determine that the new primary is different from their existing primary router, they dump their old channels and start/create new ones with the new primary. (10:33:34 AM) ramesh1: that may not be possible; may cause loss of data; unless we write some elaborate code to move the data (10:33:52 AM) loleary: True. (10:34:26 AM) loleary: Well, if we are using UDP, do we have message delivery confirmation/ack? (10:34:46 AM) ramesh1: you can configure that in the stack (10:34:58 AM) ramesh1: TCP you do not (10:35:10 AM) loleary: :) (10:35:19 AM) loleary: Right... which is why UDP is better for performance. (10:35:35 AM) loleary: So, by introducing delivery confirmation, we might as well be using TCP. (10:35:50 AM) ramesh1: since our typical customer cluster sizes are small; why not use TCP? (10:36:10 AM) ramesh1: little overhead, sure but lot more flexibility (10:36:28 AM) loleary: It is very tempting. (10:36:59 AM) loleary: Seems like a good solution for the time being. (10:37:26 AM) loleary: If it is to network intense (which I doubt it would be) we can work on implementing a usable UDP version in a later release. (10:38:00 AM) ramesh1: actually, we have UDP now (10:38:19 AM) ramesh1: with separate gossip server (10:38:21 AM) loleary: Well yeah... but it uses multicast. (10:38:29 AM) loleary: True. (10:38:56 AM) loleary: So, embedded router would be unicast with TCP to support fail-over. (10:39:26 AM) ramesh1: so, we can keep what we have; if we do not want that then embedded router is TCP (10:39:32 AM) ramesh1: true (10:39:46 AM) loleary: Or, no router would use multicast with UDP or TCP... and a third option would be unicast with UDP or TCP using GossipRouter. (10:39:55 AM) loleary: Right. (10:39:57 AM) loleary: Sounds good. (10:41:39 AM) ramesh1: I am leaning towards TCP one; I will get feedback from others (10:43:04 AM) ramesh1: thanks (10:43:33 AM) loleary: Right. That is what I am saying... the user would have those three deployment options. Our default would be Unicast over TCP using an embedded router that supports fail-over. Option two would be Multicast over UDP (what we do now by default) and option three would be Unicast over UDP or TCP using GossipRouter (like we offer now). (10:44:32 AM) ramesh1: yep, got to think about the configuration options, but sounds good.