<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>I've also been working on setting up clustered Keycloak on Docker containers in EC2 and would be interested in any potential solutions for this configuration. </div>
<div><br>
</div>
<div>Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested in hearing about the issues with this configuration.</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Scott Rossillo <<a href="mailto:srossillo@smartling.com">srossillo@smartling.com</a>><br>
<span style="font-weight:bold">Date: </span>Mon, 14 Dec 2015 18:31:30 -0500<br>
<span style="font-weight:bold">To: </span>Marek Posolda <<a href="mailto:mposolda@redhat.com">mposolda@redhat.com</a>>, <<a href="mailto:afield@redhat.com">afield@redhat.com</a>><br>
<span style="font-weight:bold">Cc: </span>keycloak-user <<a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">There are two issues:</div>
<div class=""><br class="">
</div>
<div class="">1. Infinispan relies on JGroups, which is difficult to configure correctly with the various ping techniques that aren�t UDP multicast. I can elaborate on each one that we tested but it�s just generally complex to get right. That�s not to say it�s
impossible or the biggest reason this is complicated on ECS or _insert container service here_, see #2 for that.</div>
<div class=""><br class="">
</div>
<div class="">2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged Docker instances - the default and recommend type - do not implicitly know their host�s IP. This causes IP mismatches between what JGroups thinks the machine�s
IP is and what it actually is when connecting to hosts on different machines. This is the main issue and it�s not the fault of JGroups per se, but there�s no simple work around.</div>
<div class=""><br class="">
</div>
<div class="">Take for example a simple 2 node cluster:</div>
<div class=""><br class="">
</div>
<div class="">Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The host A IP is 10.10.0.100.</div>
<div class="">Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The host B IP is 10.10.0.108.</div>
<div class=""><br class="">
</div>
<div class="">The 172.16 network is not routable between hosts (by design). Docker does port forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster traffic.</div>
<div class=""><br class="">
</div>
<div class="">So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises 172.16.0.8. The two cannot talk to each other by default. However, using the hard coded IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr
on Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to discover each other. However, they will not form a cluster. The nodes will reject the handshake thinking they�re not actually 10.10.0.100 or 10.10.0.108
respectively.</div>
<div class=""><br class="">
</div>
<div class="">I�d like to discuss further and I can share where we�ve gotten so far with workarounds to this but it may be better to get into the weeds on another list.</div>
<div class=""><br class="">
</div>
<div class="">Let me know what you think.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Scott</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">Scott Rossillo</div>
<div class="">Smartling | Senior Software Engineer</div>
<div class=""><a href="mailto:srossillo@smartling.com" class="">srossillo@smartling.com</a></div>
<div class=""><br class="">
</div>
<div class=""><span style="color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""></span>
<div id="watermark" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class="">
<a href="http://www.sigstr.com/" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;" class=""><img alt="Powered by Sigstr" border="0" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" class=""></a></div>
</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Dec 14, 2015, at 5:32 PM, Marek Posolda <<a href="mailto:mposolda@redhat.com" class="">mposolda@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">CCing Alan Field from RH Infinispan team and forwarding his question:
<br class="">
<pre wrap="" class="">I'd like to know which configuration files you are using and why is is
harder to use with Amazon�s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
</pre>
<br class="">
On 14/12/15 22:24, Scott Rossillo wrote:<br class="">
</div>
<blockquote cite="mid:622AE9A5-3E81-4CA5-B4B6-CACD84051DB2@smartling.com" type="cite" class="">
AWS was why we didn�t use Infinispan to begin with. That and it�s even more complicated when you deploy using Amazon�s Docker service (ECS) or Beanstalk.
<div class=""><br class="">
</div>
<div class="">It�s too bad Infinispan / JGroups are beasts when the out of the box configuration can�t be used. I�m planning to document this as we fix but I�d avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak DB, unless you�re using
Mongo and it�s easier to test locally.</div>
<div class=""><br class="">
</div>
<div class="">TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it does occasionally w/ECS or Beanstalk).</div>
<div class=""><br class="">
</div>
<div class="">Best, </div>
<div class="">Scott</div>
<div class=""><br class="">
<div class="">
<div class="">Scott Rossillo</div>
<div class="">Smartling | Senior Software Engineer</div>
<div class=""><a moz-do-not-send="true" href="mailto:srossillo@smartling.com" class="">srossillo@smartling.com</a></div>
<div class=""><br class="">
</div>
<div class=""><a moz-do-not-send="true" href="http://www.sigstr.com/" style="font-family: gesta,
Arial, Helvetica, sans-serif; font-size: 14px; widows: 1;
box-sizing: border-box; color: rgb(0, 124, 194);
text-decoration: none; outline: 0px !important;" class=""><img moz-do-not-send="true" alt="Powered by Sigstr" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark" style="box-sizing: border-box; border: 0px;
vertical-align: top; max-width: 100%; height: auto;
width: inherit; color: rgb(99, 99, 99); font-family:
Helvetica; font-size: 11px;" class="" border="0"></a></div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Dec 14, 2015, at 10:59 AM, Marek Posolda <<a moz-do-not-send="true" href="mailto:mposolda@redhat.com" class="">mposolda@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On 14/12/15 16:55, Marek Posolda wrote:<br class="">
<blockquote type="cite" class="">On 14/12/15 15:58, Bill Burke wrote:<br class="">
<blockquote type="cite" class="">On 12/14/2015 5:01 AM, Niko K�bler wrote:<br class="">
<blockquote type="cite" class="">Hi Marek,<br class="">
<br class="">
<blockquote type="cite" class="">Am 14.12.2015 um 08:50 schrieb Marek Posolda <<a moz-do-not-send="true" href="mailto:mposolda@redhat.com" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:mposolda@redhat.com">mposolda@redhat.com</a><br class="">
<<a moz-do-not-send="true" href="mailto:mposolda@redhat.com" class="">mailto:mposolda@redhat.com</a>>>:<br class="">
<br class="">
Btv. what's your motivation to not use infinispan? If you afraid of<br class="">
cluster communication, you don't need to worry much about it, because<br class="">
if you run single keycloak through standalone.xml, the infinispan<br class="">
automatically works in LOCAL mode and there is no any cluster<br class="">
communication at all.<br class="">
</blockquote>
My current customer is running his apps in AWS. As known, multicast is<br class="">
not available in cloud infrastructures. Wildfly/Infinispan Cluster works<br class="">
pretty well with multicast w/o having to know too much about JGroups<br class="">
config. S3_PING seams to be a viable way to get a cluster running in AWS.<br class="">
But additionally, my customer doesn�t have any (deep) knowledge about<br class="">
JBoss infrastructures and so I�m looking for a way to be able to run<br class="">
Keycloak in a cluster in AWS without the need to build up deeper<br class="">
knowlegde of JGroups config, for example in getting rid of Infinispan.<br class="">
But I do understand all the concerns in doing this.<br class="">
I still have to test S3_PING, if it works as easy as multicast. If yes,<br class="">
we can use it, if no� I don�t know yet. But this gets offtopic for<br class="">
Keycloak mailinglist, it�s more related to pure Wildfly/Infinispan.<br class="">
<br class="">
</blockquote>
seems to me it would be much easier to get Infinispan working on AWS<br class="">
than to write and maintain an entire new caching mechanism and hope we<br class="">
don't refactor the cache SPI.<br class="">
<br class="">
<br class="">
</blockquote>
+1<br class="">
<br class="">
I am sure infinispan/JGroups has possibility to run in non-multicast<br class="">
environment. You may just need to figure how exactly to configure it. So<br class="">
I agree that this issue is more related to Wildfly/Infinispan itself<br class="">
than to Keycloak.<br class="">
<br class="">
You may need to use jgroups protocols like TCP instead of default UDP<br class="">
and maybe TCPPING (this requires to manually list all your cluster<br class="">
nodes. But still, it's much better option IMO than rewriting UserSession<br class="">
SPI)<br class="">
</blockquote>
Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING <br class="">
<a moz-do-not-send="true" href="http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100" class="">http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100</a> , but it's
<br class="">
not official part of jgroups.<br class="">
<br class="">
Marek<br class="">
<blockquote type="cite" class=""><br class="">
Marek<br class="">
_______________________________________________<br class="">
keycloak-user mailing list<br class="">
<a moz-do-not-send="true" href="mailto:keycloak-user@lists.jboss.org" class="">keycloak-user@lists.jboss.org</a><br class="">
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
keycloak-user mailing list<br class="">
<a moz-do-not-send="true" href="mailto:keycloak-user@lists.jboss.org" class="">keycloak-user@lists.jboss.org</a><br class="">
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________ keycloak-user mailing list <a href="mailto:keycloak-user@lists.jboss.org">
keycloak-user@lists.jboss.org</a> <a href="https://lists.jboss.org/mailman/listinfo/keycloak-user">
https://lists.jboss.org/mailman/listinfo/keycloak-user</a></span>
</body>
</html>