<div dir="ltr"><div>Adding Bela to the thread...</div><div><br></div>The POC looks really nice to me. I could try take it from herre and finish WFLY-1066 implementation to see how everything works together.<div><br></div><div>The only thing that comes into my mind is whether we should (or or not) add capability and server group information to it? I think most of the subsystems would be interested in that. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 11, 2016 at 6:57 PM, Brian Stansberry <span dir="ltr"><<a href="mailto:brian.stansberry@redhat.com" target="_blank">brian.stansberry@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Just an FYI: I spent a couple days and worked up a POC[1] of creating a<br>
JGroups-based reliable group communication mesh over the sockets our<br>
Host Controllers use for intra-domain management communications.<br>
<br>
Currently those sockets are used to form a tree of connections; master<br>
HC to slave HCs and then HCs to their servers. Slave HCs don't talk to<br>
each other. That kind of topology works fine for our current use cases,<br>
but not for other use cases, where a full communication mesh is more<br>
appropriate.<br>
<br>
2 use cases led me to explore this:<br>
<br>
1) A longstanding request to have automatic failover of the master HC to<br>
a backup. There are different ways to do this, but group communication<br>
based leader election is a possible solution. My preference, really.<br>
<br>
2) <a href="https://issues.jboss.org/browse/WFLY-1066" rel="noreferrer" target="_blank">https://issues.jboss.org/browse/WFLY-1066</a>, which has led to various<br>
design alternatives, one of which is a distributed cache of topology<br>
information, available via each HC. See [2] for some of that discussion.<br>
<br>
I don't know if this kind of communication is a good idea, or if it's<br>
the right solution to either of these use cases. Lots of things need<br>
careful thought!! But I figured it was worth some time to experiment.<br>
And it worked in at least a basic POC way, hence this FYI.<br>
<br>
If you're interested in details, here are some Q&A:<br>
<br>
Q: Why JGroups?<br>
<br>
A: Because 1) I know it well 2) I trust it and 3) it's already used for<br>
this kind of group communications in full WildFly.<br>
<br>
Q: Why the management sockets? Why not other sockets?<br>
<br>
A: Slave HCs already need configuration for how to discover the master.<br>
Using the same sockets lets us reuse that discovery configuration for<br>
the JGroups communications as well. If we're going to use this kind of<br>
communication in an serious way, the configuration needs to be as easy<br>
as possible.<br>
<br>
Q: How does it work?<br>
<br>
A: JGroups is based on a stack of "protocols" each of which handles one<br>
aspect of reliable group communications. The POC creates and uses a<br>
standard protocol stack, except it replaces two standard protocols with<br>
custom ones:<br>
<br>
a) JGroups has various "Discovery" protocols which are used to find<br>
possible peers. I implemented one that integrates with the HC's domain<br>
controller discovery logic. It's basically a copy of the oft used<br>
TCPPING protocol with about 10-15 lines of code changed.<br>
<br>
b) JGroups has various "Transport" protocols which are responsible for<br>
actually sending/receiving over the network. I created a new one of<br>
those that knows how to use the WF management comms stuff built on JBoss<br>
Remoting. JGroups provides a number of base classes to use in this<br>
transport area, so I was able to rely on a lot of existing functionality<br>
and could just focus on the details specific to this case.<br>
<br>
Q: What have you done using the POC?<br>
<br>
A: I created a master HC and a slave on my laptop and saw them form a<br>
cluster and exchange messages. Typical stuff like starting and stopping<br>
the HCs worked. I see no reason why having multiple slaves wouldn't have<br>
worked too; I just didn't do it.<br>
<br>
Q: What's next?<br>
<br>
A: Nothing really. We have a couple concrete use cases we're looking to<br>
solve. We need to figure out the best solution for those use cases. If<br>
this kind of thing is useful in that, great. If not, it was a fun POC.<br>
<br>
[1]<br>
<a href="https://github.com/wildfly/wildfly-core/compare/master...bstansberry:jgroups-dc" rel="noreferrer" target="_blank">https://github.com/wildfly/wildfly-core/compare/master...bstansberry:jgroups-dc</a><br>
. See the commit message on the single commit to learn a bit more.<br>
<br>
[2] <a href="https://developer.jboss.org/wiki/ADomainManagedServiceRegistry" rel="noreferrer" target="_blank">https://developer.jboss.org/wiki/ADomainManagedServiceRegistry</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Brian Stansberry<br>
Senior Principal Software Engineer<br>
JBoss by Red Hat<br>
_______________________________________________<br>
wildfly-dev mailing list<br>
<a href="mailto:wildfly-dev@lists.jboss.org">wildfly-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/wildfly-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/wildfly-dev</a><br>
</font></span></blockquote></div><br></div>