<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">There are two issues:</div><div class=""><br class=""></div><div class="">1. Infinispan relies on JGroups, which is difficult to configure correctly with the various ping techniques that aren’t UDP multicast. I can elaborate on each one that we tested but it’s just generally complex to get right. That’s not to say it’s impossible or the biggest reason this is complicated on ECS or _insert container service here_, see #2 for that.</div><div class=""><br class=""></div><div class="">2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged Docker instances - the default and recommend type - do not implicitly know their host’s IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it actually is when connecting to hosts on different machines. This is the main issue and it’s not the fault of JGroups per se, but there’s no simple work around.</div><div class=""><br class=""></div><div class="">Take for example a simple 2 node cluster:</div><div class=""><br class=""></div><div class="">Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The host A IP is 10.10.0.100.</div><div class="">Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The host B IP is 10.10.0.108.</div><div class=""><br class=""></div><div class="">The 172.16 network is not routable between hosts (by design). Docker does port forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster traffic.</div><div class=""><br class=""></div><div class="">So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises 172.16.0.8. The two cannot talk to each other by default. However, using the hard coded IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to discover each other. However, they will not form a cluster. The nodes will reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108 respectively.</div><div class=""><br class=""></div><div class="">I’d like to discuss further and I can share where we’ve gotten so far with workarounds to this but it may be better to get into the weeds on another list.</div><div class=""><br class=""></div><div class="">Let me know what you think.</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Scott</div><div class=""><br class=""></div><div class="">
<div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><span style="color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""></span><div id="watermark" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""><a href="http://www.sigstr.com/" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;" class=""><img alt="Powered by Sigstr" border="0" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" class=""></a></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Dec 14, 2015, at 5:32 PM, Marek Posolda <<a href="mailto:mposolda@redhat.com" class="">mposolda@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">CCing Alan Field from RH Infinispan
team and forwarding his question: <br class="">
<pre wrap="" class="">I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
</pre>
<br class="">
On 14/12/15 22:24, Scott Rossillo wrote:<br class="">
</div>
<blockquote cite="mid:622AE9A5-3E81-4CA5-B4B6-CACD84051DB2@smartling.com" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
AWS was why we didn’t use Infinispan to begin with. That and it’s
even more complicated when you deploy using Amazon’s Docker
service (ECS) or Beanstalk.
<div class=""><br class="">
</div>
<div class="">It’s too bad Infinispan / JGroups are beasts when
the out of the box configuration can’t be used. I’m planning to
document this as we fix but I’d avoid S3_PING and use JDBC_PING.
You already need JDBC for the Keycloak DB, unless you’re using
Mongo and it’s easier to test locally.</div>
<div class=""><br class="">
</div>
<div class="">TCPPING will bite you on AWS if Amazon decides to
replace one of your instances (which it does occasionally w/ECS
or Beanstalk).</div>
<div class=""><br class="">
</div>
<div class="">Best, </div>
<div class="">Scott</div>
<div class=""><br class="">
<div class="">
<div class="">Scott Rossillo</div>
<div class="">Smartling | Senior Software Engineer</div>
<div class=""><a moz-do-not-send="true" href="mailto:srossillo@smartling.com" class="">srossillo@smartling.com</a></div>
<div class=""><br class="">
</div>
<div class=""><a moz-do-not-send="true" href="http://www.sigstr.com/" style="font-family: gesta,
Arial, Helvetica, sans-serif; font-size: 14px; widows: 1;
box-sizing: border-box; color: rgb(0, 124, 194);
text-decoration: none; outline: 0px !important;" class=""><img moz-do-not-send="true" alt="Powered by Sigstr" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark" style="box-sizing: border-box; border: 0px;
vertical-align: top; max-width: 100%; height: auto;
width: inherit; color: rgb(99, 99, 99); font-family:
Helvetica; font-size: 11px;" class="" border="0"></a></div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Dec 14, 2015, at 10:59 AM, Marek Posolda
<<a moz-do-not-send="true" href="mailto:mposolda@redhat.com" class="">mposolda@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On 14/12/15 16:55, Marek Posolda wrote:<br class="">
<blockquote type="cite" class="">On 14/12/15 15:58, Bill
Burke wrote:<br class="">
<blockquote type="cite" class="">On 12/14/2015 5:01
AM, Niko Köbler wrote:<br class="">
<blockquote type="cite" class="">Hi Marek,<br class="">
<br class="">
<blockquote type="cite" class="">Am 14.12.2015 um
08:50 schrieb Marek Posolda <<a moz-do-not-send="true" href="mailto:mposolda@redhat.com" class=""></a><a class="moz-txt-link-abbreviated" href="mailto:mposolda@redhat.com">mposolda@redhat.com</a><br class="">
<<a moz-do-not-send="true" href="mailto:mposolda@redhat.com" class="">mailto:mposolda@redhat.com</a>>>:<br class="">
<br class="">
Btv. what's your motivation to not use
infinispan? If you afraid of<br class="">
cluster communication, you don't need to worry
much about it, because<br class="">
if you run single keycloak through
standalone.xml, the infinispan<br class="">
automatically works in LOCAL mode and there is
no any cluster<br class="">
communication at all.<br class="">
</blockquote>
My current customer is running his apps in AWS. As
known, multicast is<br class="">
not available in cloud infrastructures.
Wildfly/Infinispan Cluster works<br class="">
pretty well with multicast w/o having to know too
much about JGroups<br class="">
config. S3_PING seams to be a viable way to get a
cluster running in AWS.<br class="">
But additionally, my customer doesn’t have any
(deep) knowledge about<br class="">
JBoss infrastructures and so I’m looking for a way
to be able to run<br class="">
Keycloak in a cluster in AWS without the need to
build up deeper<br class="">
knowlegde of JGroups config, for example in
getting rid of Infinispan.<br class="">
But I do understand all the concerns in doing
this.<br class="">
I still have to test S3_PING, if it works as easy
as multicast. If yes,<br class="">
we can use it, if no… I don’t know yet. But this
gets offtopic for<br class="">
Keycloak mailinglist, it’s more related to pure
Wildfly/Infinispan.<br class="">
<br class="">
</blockquote>
seems to me it would be much easier to get
Infinispan working on AWS<br class="">
than to write and maintain an entire new caching
mechanism and hope we<br class="">
don't refactor the cache SPI.<br class="">
<br class="">
<br class="">
</blockquote>
+1<br class="">
<br class="">
I am sure infinispan/JGroups has possibility to run in
non-multicast<br class="">
environment. You may just need to figure how exactly
to configure it. So<br class="">
I agree that this issue is more related to
Wildfly/Infinispan itself<br class="">
than to Keycloak.<br class="">
<br class="">
You may need to use jgroups protocols like TCP instead
of default UDP<br class="">
and maybe TCPPING (this requires to manually list all
your cluster<br class="">
nodes. But still, it's much better option IMO than
rewriting UserSession<br class="">
SPI)<br class="">
</blockquote>
Btv. if TCPPING or S3_PING is an issue, there is also
AWS_PING <br class="">
<a moz-do-not-send="true" href="http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100" class="">http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100</a>
, but it's <br class="">
not official part of jgroups.<br class="">
<br class="">
Marek<br class="">
<blockquote type="cite" class=""><br class="">
Marek<br class="">
_______________________________________________<br class="">
keycloak-user mailing list<br class="">
<a moz-do-not-send="true" href="mailto:keycloak-user@lists.jboss.org" class="">keycloak-user@lists.jboss.org</a><br class="">
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
keycloak-user mailing list<br class="">
<a moz-do-not-send="true" href="mailto:keycloak-user@lists.jboss.org" class="">keycloak-user@lists.jboss.org</a><br class="">
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></body></html>