<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">I actually set the jgroups.bind_addr to global. I need the EC2 instance's address for jgroups.external.addr, see:</div><div class=""><br class=""></div><div class=""><a href="https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh" class="">https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh</a></div><div class=""><br class=""><div class="">
<div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><span style="color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""></span><div id="watermark" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""><a href="http://www.sigstr.com/" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;" class=""><img alt="Powered by Sigstr" border="0" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" class=""></a></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Dec 16, 2015, at 4:21 PM, Alan Field <<a href="mailto:afield@redhat.com" class="">afield@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class="">Almost...<br class=""></div><div class=""><br class=""></div><div class="">I guess if the EC2 instance IP works for the bind address, why do you need to set external_addr?</div><div class=""><br class=""></div><div class="">Thanks for bearing with me on this! :-)</div><div class=""><br class=""></div><div class="">Alan</div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">From: </b>"Scott Rossillo" <<a href="mailto:srossillo@smartling.com" class="">srossillo@smartling.com</a>><br class=""><b class="">To: </b>"Alan Field" <<a href="mailto:afield@redhat.com" class="">afield@redhat.com</a>><br class=""><b class="">Cc: </b>"Niko Köbler" <<a href="mailto:niko@n-k.de" class="">niko@n-k.de</a>>, "keycloak-user" <<a href="mailto:keycloak-user@lists.jboss.org" class="">keycloak-user@lists.jboss.org</a>><br class=""><b class="">Sent: </b>Wednesday, December 16, 2015 4:17:29 PM<br class=""><b class="">Subject: </b>Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?<br class=""><div class=""><br class=""></div>Ah, sorry, my originally contrived example wasn’t using Amazon but just my local Docker machine IP.<div class=""><br class=""></div><div class="">In the case of my ECS tests, 172.16.0.0/16 is the Docker network’s IP, which is local to the machine / EC2 instance. Using ECS, my VPC has an IP range of 172.31.0.0/16, so the bind_addr has to be on this network. On my small cluster, that’s either 172.31.44.109 or 172.31.45.191.</div><div class=""><br class=""></div><div class="">Does that clear it up?</div><div class=""><br class=""><div class="">
<div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><span style="color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""></span><div id="watermark" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""><a href="http://www.sigstr.com/" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;" class="" target="_blank"><img alt="Powered by Sigstr" border="0" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" class="" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark"></a></div></div>
</div>
<br class=""><div class=""><blockquote class=""><div class="">On Dec 16, 2015, at 4:11 PM, Alan Field <<a href="mailto:afield@redhat.com" class="" target="_blank">afield@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class="">Hey Scott,<br class=""></div><div class=""><br class=""></div><div class="">Thanks, I think you answered all of my questions, but I'm confused by something you said in your first email:</div><div class=""><br class=""></div><div class="">"<div style="font-family: Calibri, sans-serif; font-size: 14px;" class="">The 172.16 network is not routable between hosts (by design). Docker does port forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster traffic.</div><div style="font-family: Calibri, sans-serif; font-size: 14px;" class=""><br class=""></div><div style="font-family: Calibri, sans-serif; font-size: 14px;" class="">So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises 172.16.0.8. The two cannot talk to each other by default. </div>"</div><div class=""><br class=""></div><div class="">My understanding is that the 172.16 addresses are the Amazon EC2 instance’s internal IP, so I'm confused why this didn't work for you before. Is the difference that you were setting jgroups.bind_addr to this address and now you are setting it to global and setting external_addr to the instance’s internal IP? Just trying to understand what the problem was and how you fixed it!</div><div class=""><br class=""></div><div class="">Thanks again,</div><div class="">Alan</div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">From: </b>"Scott Rossillo" <<a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a>><br class=""><b class="">To: </b>"Alan Field" <<a href="mailto:afield@redhat.com" class="" target="_blank">afield@redhat.com</a>><br class=""><b class="">Cc: </b>"Niko Köbler" <<a href="mailto:niko@n-k.de" class="" target="_blank">niko@n-k.de</a>>, "keycloak-user" <<a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a>><br class=""><b class="">Sent: </b>Wednesday, December 16, 2015 3:45:40 PM<br class=""><b class="">Subject: </b>Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?<br class=""><div class=""><br class=""></div>Hi Alan,<div class=""><br class=""></div><div class="">> It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I understand not wanting to have to setup and maintain the extra gossip router processes.</div><div class=""><br class=""></div><div class="">True, it’s mainly about maintaining extra components.</div><div class=""><br class=""></div><div class="">> Which IP address from your example is retrieved with this command:</div><div class="">> EXTERNAL_HOST_IP=$(curl <a href="http://169.254.169.254/latest/meta-data/local-ipv4" class="" target="_blank">http://169.254.169.254/latest/meta-data/local-ipv4</a>)”</div><div class=""><br class=""></div><div class="">I get the Amazon EC2 instance’s internal IP. This is what I want. There’s another endpoint for public but I don’t want to use it. What’s good about this is when called from inside a Docker container, I manage to get the actual internal IP for the EC2 instance.</div><div class=""><br class=""></div><div class="">> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?</div><div class=""><br class=""></div><div class="">Since this was a test with just 2 known hosts, I injected them as a Docker environment variable with two fixed IPs. Once we switch to JDBC_PING, this will be removed.</div><div class=""><br class=""></div><div class="">> For my curiosity, can you tell me more about why you don't want to use S3_PING? Is it the cost or something else? Just wondering and JDBC_PING should work fine.</div><div class=""><br class=""></div><div class="">S3_PING, like Gossip Router adds an external dependency on another service. S3 has had consistency issues 3 times in 2015 (at least in US East). I don’t want to rely another component when I already need the database to be up. Less components, less chance of failure. Also, there are ton of variables to set with S3 and it requires preliminary work. I want something that scales well from dev to QA to prod. JDBC_PING has a datasource_jndi_name property. I can just reuse the data source I set up for Keycloak.</div><div class=""><br class=""></div><div class="">I hope I got all your questions.</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Scott</div><div class=""><br class=""><div class="">
<div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><span style="color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""></span><div id="watermark" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);" class=""><a href="http://www.sigstr.com/" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;" class="" target="_blank"><img alt="Powered by Sigstr" border="0" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" class="" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark"></a></div></div>
</div>
<br class=""><div class=""><blockquote class=""><div class="">On Dec 16, 2015, at 3:33 PM, Alan Field <<a href="mailto:afield@redhat.com" class="" target="_blank">afield@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class="">Hey Scott,<br class=""></div><div class=""><br class=""></div><div class="">Thanks for following up and showing me your code. I have some questions inline for you:</div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">From:<span class="Apple-converted-space"> </span></b>"Scott Rossillo" <<a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a>><br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Alan Field" <<a href="mailto:afield@redhat.com" class="" target="_blank">afield@redhat.com</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>"Niko Köbler" <<a href="mailto:niko@n-k.de" class="" target="_blank">niko@n-k.de</a>>, "keycloak-user" <<a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a>><br class=""><b class="">Sent:<span class="Apple-converted-space"> </span></b>Wednesday, December 16, 2015 2:19:27 PM<br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?<br class=""><div class=""><br class=""></div>Hi Alan,<div class=""><br class=""></div><div class="">Thanks for the informative email. The steps you outlined are similar to what I’ve tested with ECS. The gossip router is definitely a no-go for production since it’s a single point of failure.</div></blockquote><div class="">It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I understand not wanting to have to setup and maintain the extra gossip router processes.</div><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><div class=""><br class=""></div><div class="">I am testing this down at the JGroups level right now and got it working with ECS. There were two issues. On TCP you have to specify the external_addr to match the EC2 host otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a random port. With Docker instances, this fails. Using a known client_bind_port works well.</div></blockquote><div class="">Which IP address from your example is retrieved with this command:</div><div class=""><br class=""></div><div class=""><table class="highlight tab-size js-file-line-container mceItemTable" style="box-sizing: border-box; border-collapse: collapse; border-spacing: 0px; tab-size: 8; color: rgb(51, 51, 51); font-family: Helvetica, arial, nimbussansl, liberationsans, freesans, clean, sans-serif, 'Apple Color Emoji', 'Segoe UI Emoji', 'Segoe UI Symbol'; font-size: 13px; line-height: 20.2222px; background-color: rgb(255, 255, 255);"><tbody style="box-sizing: border-box;" class=""><tr style="box-sizing: border-box;" class=""><td id="LC7" class="js-file-line blob-code-inner blob-code" style="box-sizing: border-box; padding: 0px 10px; position: relative; vertical-align: top; font-family: Consolas, 'Liberation Mono', Menlo, Courier, monospace; font-size: 12px; white-space: pre; overflow: visible; word-wrap: normal;"><div class="">EXTERNAL_HOST_IP=<span class="pl-s" style="box-sizing: border-box; color: rgb(24, 54, 145);"><span class="pl-pds" style="box-sizing: border-box;">$(</span>curl <a href="http://169.254.169.254/latest/meta-data/local-ipv4" class="" target="_blank">http://169.254.169.254/latest/meta-data/local-ipv4</a><span class="pl-pds" style="box-sizing: border-box;">)</span></span><span style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="">"</span></div><div class=""><span style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><br class=""></span></div><div class=""><span style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="">Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in EC2, I get the internal IP address for the instance, but not the public IP address. In your example, that would be the <span style="line-height: 20.2222px; white-space: pre; background-color: rgb(255, 255, 255);" class="">172.16.0.4 address. Also which address is used for the bind_addr when you use -Djgroups.bind_addr=global? </span></span></div></td></tr></tbody></table></div><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><div class=""><br class=""></div><div class="">Here’s the code I’m testing with: <a href="https://github.com/foo4u/aws-infinispan-poc" class="" target="_blank">https://github.com/foo4u/aws-infinispan-poc</a></div><div class=""><br class=""></div><div class="">Most interesting are probably:</div><div class=""><br class=""></div><div class=""><a href="https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh" class="" target="_blank">https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh</a></div></blockquote><div class="">How are you setting the JGROUPS_INITIAL_HOSTS environment variable?</div><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><div class=""><a href="https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/src/main/resources/tcp.xml" class="" target="_blank">https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/src/main/resources/tcp.xml</a></div><div class=""><br class=""></div><div class="">With this set up the nodes on different machines communicate without issue. I still have to add in something other than TCP_PING, but that wasn’t the main issue. Will use JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for Keycloak.</div></blockquote><div class="">For my curiosity, can you tell me more about why you don't want to use S3_PING? Is it the cost or something else? Just wondering and JDBC_PING should work fine.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Alan</div><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><div class=""><br class=""></div><div class=""><div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><a href="https://app.sigstr.com/uc/55e5d41c6533390d03580000" id="campaignblock" target="_blank" class="" style="box-sizing: border-box; color: rgb(0, 75, 118); outline-offset: -2px; font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255); outline: 0px !important;"><img alt="Latest News + Events" border="0" class="" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: blue; font-family: Helvetica; font-size: 12px;" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/img"></a><div id="watermark" class="" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);"><a href="http://www.sigstr.com/" class="" target="_blank" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;"><img alt="Powered by Sigstr" border="0" class="" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark"></a></div></div></div><br class=""><div class=""><blockquote class=""><div class="">On Dec 15, 2015, at 2:13 PM, Alan Field <<a href="mailto:afield@redhat.com" class="" target="_blank">afield@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div class="" style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">Just to be clear, I have successfully tested Infinispan library and server mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud providers support multicast. The Docker case is a little different though, because of the issues with getting access to the IP address. <br class=""></div><div class=""><br class=""></div><div class="">Thanks,<br class="">Alan</div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote class="" style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">From:<span class="Apple-converted-space"> </span></b>"Niko Köbler" <<a href="mailto:niko@n-k.de" class="" target="_blank">niko@n-k.de</a>><br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>"Paul Blair" <<a href="mailto:pblair@clearme.com" class="" target="_blank">pblair@clearme.com</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>"keycloak-user" <<a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a>><br class=""><b class="">Sent:<span class="Apple-converted-space"> </span></b>Tuesday, December 15, 2015 1:53:18 PM<br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?<br class=""><div class=""><br class=""></div>We will go for the first run with EC2 and S3_PING, but w/o Docker.<div class="">If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist), we will test this.</div><div class=""><br class=""></div><div class="">Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a proper solution, only workarounds. :(</div><div class=""><br class=""></div><div class=""><br class=""><br class=""><div class=""><blockquote class=""><div class="">Am 15.12.2015 um 15:47 schrieb Paul Blair <<a href="mailto:pblair@clearme.com" class="" target="_blank">pblair@clearme.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;"><div class="">I've also been working on setting up clustered Keycloak on Docker containers in EC2 and would be interested in any potential solutions for this configuration. </div><div class=""><br class=""></div><div class="">Alternatively I've set up on EC2 without Docker with S3_PING. I'd be interested in hearing about the issues with this configuration.</div><div class=""><br class=""></div><span id="OLK_SRC_BODY_SECTION" class=""><div class="" style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);"><span class="" style="font-weight: bold;">From:<span class="Apple-converted-space"> </span></span>Scott Rossillo <<a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a>><br class=""><span class="" style="font-weight: bold;">Date:<span class="Apple-converted-space"> </span></span>Mon, 14 Dec 2015 18:31:30 -0500<br class=""><span class="" style="font-weight: bold;">To:<span class="Apple-converted-space"> </span></span>Marek Posolda <<a href="mailto:mposolda@redhat.com" class="" target="_blank">mposolda@redhat.com</a>>, <<a href="mailto:afield@redhat.com" class="" target="_blank">afield@redhat.com</a>><br class=""><span class="" style="font-weight: bold;">Cc:<span class="Apple-converted-space"> </span></span>keycloak-user <<a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a>><br class=""><span class="" style="font-weight: bold;">Subject:<span class="Apple-converted-space"> </span></span>Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?<br class=""></div><div class=""><br class=""></div><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">There are two issues:</div><div class=""><br class=""></div><div class="">1. Infinispan relies on JGroups, which is difficult to configure correctly with the various ping techniques that aren’t UDP multicast. I can elaborate on each one that we tested but it’s just generally complex to get right. That’s not to say it’s impossible or the biggest reason this is complicated on ECS or _insert container service here_, see #2 for that.</div><div class=""><br class=""></div><div class="">2. It is difficult to do discovery correctly with JGroups and Docker. Non-privileged Docker instances - the default and recommend type - do not implicitly know their host’s IP. This causes IP mismatches between what JGroups thinks the machine’s IP is and what it actually is when connecting to hosts on different machines. This is the main issue and it’s not the fault of JGroups per se, but there’s no simple work around.</div><div class=""><br class=""></div><div class="">Take for example a simple 2 node cluster:</div><div class=""><br class=""></div><div class="">Node 1 comes up on the docker0 interface of host A with the IP address 172.16.0.4. The host A IP is 10.10.0.100.</div><div class="">Node 2 comes up on the docker0 interface of host B with the IP address 172.16.0.8. The host B IP is 10.10.0.108.</div><div class=""><br class=""></div><div class="">The 172.16 network is not routable between hosts (by design). Docker does port forwarding for ports we wish to expose to this works fine for HTTP/HTTPS but not the cluster traffic.</div><div class=""><br class=""></div><div class="">So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 advertises 172.16.0.8. The two cannot talk to each other by default. However, using the hard coded IPs and TCP PING, we can set external_addr on Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to discover each other. However, they will not form a cluster. The nodes will reject the handshake thinking they’re not actually 10.10.0.100 or 10.10.0.108 respectively.</div><div class=""><br class=""></div><div class="">I’d like to discuss further and I can share where we’ve gotten so far with workarounds to this but it may be better to get into the weeds on another list.</div><div class=""><br class=""></div><div class="">Let me know what you think.</div><div class=""><br class=""></div><div class="">Best,</div><div class="">Scott</div><div class=""><br class=""></div><div class=""><div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><div id="watermark" class="" style="box-sizing: border-box; color: rgb(169, 169, 169); font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; line-height: 20px; widows: 1; background-color: rgb(255, 255, 255);"><a href="http://www.sigstr.com/" class="" target="_blank" style="box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; background-color: transparent; outline: 0px !important;"><img alt="Powered by Sigstr" border="0" class="" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark"></a></div></div></div><br class=""><div class=""><blockquote class=""><div class="">On Dec 14, 2015, at 5:32 PM, Marek Posolda <<a href="mailto:mposolda@redhat.com" class="" target="_blank">mposolda@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div class="moz-cite-prefix">CCing Alan Field from RH Infinispan team and forwarding his question:<span class="Apple-converted-space"> </span><br class=""><pre class="">I'd like to know which configuration files you are using and why is is
harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
interested in how big a cluster you are using in AWS.
</pre><br class="">On 14/12/15 22:24, Scott Rossillo wrote:<br class=""></div><blockquote cite="mid:622AE9A5-3E81-4CA5-B4B6-CACD84051DB2@smartling.com" class="">AWS was why we didn’t use Infinispan to begin with. That and it’s even more complicated when you deploy using Amazon’s Docker service (ECS) or Beanstalk.<div class=""><br class=""></div><div class="">It’s too bad Infinispan / JGroups are beasts when the out of the box configuration can’t be used. I’m planning to document this as we fix but I’d avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak DB, unless you’re using Mongo and it’s easier to test locally.</div><div class=""><br class=""></div><div class="">TCPPING will bite you on AWS if Amazon decides to replace one of your instances (which it does occasionally w/ECS or Beanstalk).</div><div class=""><br class=""></div><div class="">Best, </div><div class="">Scott</div><div class=""><br class=""><div class=""><div class="">Scott Rossillo</div><div class="">Smartling | Senior Software Engineer</div><div class=""><a href="mailto:srossillo@smartling.com" class="" target="_blank">srossillo@smartling.com</a></div><div class=""><br class=""></div><div class=""><a href="http://www.sigstr.com/" class="" target="_blank" style="font-family: gesta, Arial, Helvetica, sans-serif; font-size: 14px; widows: 1; box-sizing: border-box; color: rgb(0, 124, 194); text-decoration: none; outline: 0px !important;"><img alt="Powered by Sigstr" class="" border="0" style="box-sizing: border-box; border: 0px; vertical-align: top; max-width: 100%; height: auto; width: inherit; color: rgb(99, 99, 99); font-family: Helvetica; font-size: 11px;" src="https://app.sigstr.com/uc/55e5d41c6533390d03580000/watermark"></a></div></div><br class=""><div class=""><blockquote class=""><div class="">On Dec 14, 2015, at 10:59 AM, Marek Posolda <<a href="mailto:mposolda@redhat.com" class="" target="_blank">mposolda@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On 14/12/15 16:55, Marek Posolda wrote:<br class=""><blockquote class="">On 14/12/15 15:58, Bill Burke wrote:<br class=""><blockquote class="">On 12/14/2015 5:01 AM, Niko Köbler wrote:<br class=""><blockquote class="">Hi Marek,<br class=""><br class=""><blockquote class="">Am 14.12.2015 um 08:50 schrieb Marek Posolda <<a class="moz-txt-link-abbreviated" href="mailto:mposolda@redhat.com" target="_blank">mposolda@redhat.com</a><br class=""><<a href="mailto:mposolda@redhat.com" class="" target="_blank">mailto:mposolda@redhat.com</a>>>:<br class=""><br class="">Btv. what's your motivation to not use infinispan? If you afraid of<br class="">cluster communication, you don't need to worry much about it, because<br class="">if you run single keycloak through standalone.xml, the infinispan<br class="">automatically works in LOCAL mode and there is no any cluster<br class="">communication at all.<br class=""></blockquote>My current customer is running his apps in AWS. As known, multicast is<br class="">not available in cloud infrastructures. Wildfly/Infinispan Cluster works<br class="">pretty well with multicast w/o having to know too much about JGroups<br class="">config. S3_PING seams to be a viable way to get a cluster running in AWS.<br class="">But additionally, my customer doesn’t have any (deep) knowledge about<br class="">JBoss infrastructures and so I’m looking for a way to be able to run<br class="">Keycloak in a cluster in AWS without the need to build up deeper<br class="">knowlegde of JGroups config, for example in getting rid of Infinispan.<br class="">But I do understand all the concerns in doing this.<br class="">I still have to test S3_PING, if it works as easy as multicast. If yes,<br class="">we can use it, if no… I don’t know yet. But this gets offtopic for<br class="">Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.<br class=""><br class=""></blockquote>seems to me it would be much easier to get Infinispan working on AWS<br class="">than to write and maintain an entire new caching mechanism and hope we<br class="">don't refactor the cache SPI.<br class=""><br class=""><br class=""></blockquote>+1<br class=""><br class="">I am sure infinispan/JGroups has possibility to run in non-multicast<br class="">environment. You may just need to figure how exactly to configure it. So<br class="">I agree that this issue is more related to Wildfly/Infinispan itself<br class="">than to Keycloak.<br class=""><br class="">You may need to use jgroups protocols like TCP instead of default UDP<br class="">and maybe TCPPING (this requires to manually list all your cluster<br class="">nodes. But still, it's much better option IMO than rewriting UserSession<br class="">SPI)<br class=""></blockquote>Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING<span class="Apple-converted-space"> </span><br class=""><a href="http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100" class="" target="_blank">http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100</a><span class="Apple-converted-space"> </span>, but it's<span class="Apple-converted-space"> </span><br class="">not official part of jgroups.<br class=""><br class="">Marek<br class=""><blockquote class=""><br class="">Marek<br class="">_______________________________________________<br class="">keycloak-user mailing list<br class=""><a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a><br class=""><a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br class=""></blockquote><br class="">_______________________________________________<br class="">keycloak-user mailing list<br class=""><a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a><br class=""><a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><br class=""></div></div></blockquote></div><br class=""></div></blockquote><br class=""></div></div></blockquote></div><br class=""></div></div>_______________________________________________ keycloak-user mailing list<span class="Apple-converted-space"> </span><a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a><span class="Apple-converted-space"> </span><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" class="" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></span></div>_______________________________________________<br class="">keycloak-user mailing list<br class=""><a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a><br class=""><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" class="" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></div></blockquote></div><br class=""></div><br class="">_______________________________________________<br class="">keycloak-user mailing list<br class=""><a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a><br class=""><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" class="" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></blockquote><div class=""><br class=""></div></div></div>_______________________________________________<br class="">keycloak-user mailing list<br class=""><a href="mailto:keycloak-user@lists.jboss.org" class="" target="_blank">keycloak-user@lists.jboss.org</a><br class=""><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user" class="" target="_blank">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></div></blockquote></div></blockquote></div></div></blockquote></div><br class=""></div></blockquote><div class=""><br class=""></div></div></div></div></blockquote></div><br class=""></div></blockquote><div class=""><br class=""></div></div></div></div></blockquote></div><br class=""></div></body></html>