Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?

From: "Scott Rossillo" <srossillo(a)smartling.com&gt; To: "Alan Field" <afield(a)redhat.com&gt; Cc: "Niko Köbler" <niko(a)n-k.de&gt;, "keycloak-user" <keycloak-user(a)lists.jboss.org&gt; Sent: Wednesday, December 16, 2015 2:19:27 PM Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?

Hi Alan,

Thanks for the informative email. The steps you outlined are similar to what I’ve tested with ECS. The gossip router is definitely a no-go for production since it’s a single point of failure.

I am testing this down at the JGroups level right now and got it working with ECS. There were two issues. On TCP you have to specify the external_addr to match the EC2 host otherwise the nodes won’t form a cluster. Secondly, FD_SOCK attempts to connect back on a random port. With Docker instances, this fails. Using a known client_bind_port works well.

Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc

Most interesting are probably:

https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e...

https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s...

With this set up the nodes on different machines communicate without issue. I still have to add in something other than TCP_PING, but that wasn’t the main issue. Will use JDBC_PING most likely. Not a fan of S3 for coordination. Plus I already need an RDBMS for Keycloak.

Scott Rossillo Smartling | Senior Software Engineer srossillo(a)smartling.com

> On Dec 15, 2015, at 2:13 PM, Alan Field < afield(a)redhat.com > wrote:

> Just to be clear, I have successfully tested Infinispan library and server > mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. > None of the cloud providers support multicast. The Docker case is a little > different though, because of the issues with getting access to the IP > address.

> Thanks, > Alan

> ----- Original Message -----

> > From: "Niko Köbler" < niko(a)n-k.de > > > > To: "Paul Blair" < pblair(a)clearme.com > > > > Cc: "keycloak-user" < keycloak-user(a)lists.jboss.org > > > > Sent: Tuesday, December 15, 2015 1:53:18 PM > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions > > SPI > > ? >

> > We will go for the first run with EC2 and S3_PING, but w/o Docker. > > > If we/you/whoever will find a proper solution (possibly on the jgroups > > mailinglist), we will test this. >

> > Seams that everybody is aware of the Docker/Cloud/Multicast issues, but > > no-one has a proper solution, only workarounds. :( >

> > > Am 15.12.2015 um 15:47 schrieb Paul Blair < pblair(a)clearme.com >: > > >

> > > I've also been working on setting up clustered Keycloak on Docker > > > containers > > > in EC2 and would be interested in any potential solutions for this > > > configuration. > > >

> > > Alternatively I've set up on EC2 without Docker with S3_PING. I'd be > > > interested in hearing about the issues with this configuration. > > >

> > > From: Scott Rossillo < srossillo(a)smartling.com > > > > > > > Date: Mon, 14 Dec 2015 18:31:30 -0500 > > > > > > To: Marek Posolda < mposolda(a)redhat.com >, < afield(a)redhat.com > > > > > > > Cc: keycloak-user < keycloak-user(a)lists.jboss.org > > > > > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User > > > Sessions > > > SPI > > > ? > > >

> > > There are two issues: > > >

> > > 1. Infinispan relies on JGroups, which is difficult to configure > > > correctly > > > with the various ping techniques that aren’t UDP multicast. I can > > > elaborate > > > on each one that we tested but it’s just generally complex to get > > > right. > > > That’s not to say it’s impossible or the biggest reason this is > > > complicated > > > on ECS or _insert container service here_, see #2 for that. > > >

> > > 2. It is difficult to do discovery correctly with JGroups and Docker. > > > Non-privileged Docker instances - the default and recommend type - do > > > not > > > implicitly know their host’s IP. This causes IP mismatches between what > > > JGroups thinks the machine’s IP is and what it actually is when > > > connecting > > > to hosts on different machines. This is the main issue and it’s not the > > > fault of JGroups per se, but there’s no simple work around. > > >

> > > Take for example a simple 2 node cluster: > > >

> > > Node 1 comes up on the docker0 interface of host A with the IP address > > > 172.16.0.4. The host A IP is 10.10.0.100. > > > > > > Node 2 comes up on the docker0 interface of host B with the IP address > > > 172.16.0.8. The host B IP is 10.10.0.108. > > >

> > > The 172.16 network is not routable between hosts (by design). Docker > > > does > > > port forwarding for ports we wish to expose to this works fine for > > > HTTP/HTTPS but not the cluster traffic. > > >

> > > So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 > > > advertises 172.16.0.8. The two cannot talk to each other by default. > > > However, using the hard coded IPs and TCP PING, we can set > > > external_addr > > > on > > > Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and > > > set > > > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to > > > discover each other. However, they will not form a cluster. The nodes > > > will > > > reject the handshake thinking they’re not actually 10.10.0.100 or > > > 10.10.0.108 respectively. > > >

> > > I’d like to discuss further and I can share where we’ve gotten so far > > > with > > > workarounds to this but it may be better to get into the weeds on > > > another > > > list. > > >

> > > Let me know what you think. > > >

> > > Best, > > > > > > Scott > > >

> > > Scott Rossillo > > > > > > Smartling | Senior Software Engineer > > > > > > srossillo(a)smartling.com > > >

> > > > On Dec 14, 2015, at 5:32 PM, Marek Posolda < mposolda(a)redhat.com > > > > > wrote: > > > > > >

> > > > CCing Alan Field from RH Infinispan team and forwarding his question: > > > > > > > > > > I'd like to know which configuration files you are using and why is > > > > is > > > > > > > > > > harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd > > > > also > > > > be > > > > > > > > > > interested in how big a cluster you are using in AWS. > > > > > >

> > > > On 14/12/15 22:24, Scott Rossillo wrote: > > > > > >

> > > > > AWS was why we didn’t use Infinispan to begin with. That and it’s > > > > > even > > > > > more > > > > > complicated when you deploy using Amazon’s Docker service (ECS) or > > > > > Beanstalk. > > > > > > > > > >

> > > > > It’s too bad Infinispan / JGroups are beasts when the out of the > > > > > box > > > > > configuration can’t be used. I’m planning to document this as we > > > > > fix > > > > > but > > > > > I’d > > > > > avoid S3_PING and use JDBC_PING. You already need JDBC for the > > > > > Keycloak > > > > > DB, > > > > > unless you’re using Mongo and it’s easier to test locally. > > > > > > > > > >

> > > > > TCPPING will bite you on AWS if Amazon decides to replace one of > > > > > your > > > > > instances (which it does occasionally w/ECS or Beanstalk). > > > > > > > > > >

> > > > > Best, > > > > > > > > > > > > > > > Scott > > > > > > > > > >

> > > > > Scott Rossillo > > > > > > > > > > > > > > > Smartling | Senior Software Engineer > > > > > > > > > > > > > > > srossillo(a)smartling.com > > > > > > > > > >

> > > > > > On Dec 14, 2015, at 10:59 AM, Marek Posolda < mposolda(a)redhat.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >

> > > > > > On 14/12/15 16:55, Marek Posolda wrote: > > > > > > > > > > > > > > >

> > > > > > > On 14/12/15 15:58, Bill Burke wrote: > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > On 12/14/2015 5:01 AM, Niko Köbler wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > Hi Marek, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > Am 14.12.2015 um 08:50 schrieb Marek Posolda < > > > > > > > > > > mposolda(a)redhat.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < mailto:mposolda@redhat.com >>: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > Btv. what's your motivation to not use infinispan? If you > > > > > > > > > > afraid > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cluster communication, you don't need to worry much about > > > > > > > > > > it, > > > > > > > > > > because > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if you run single keycloak through standalone.xml, the > > > > > > > > > > infinispan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > automatically works in LOCAL mode and there is no any > > > > > > > > > > cluster > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > communication at all. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > My current customer is running his apps in AWS. As known, > > > > > > > > > multicast > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not available in cloud infrastructures. Wildfly/Infinispan > > > > > > > > > Cluster > > > > > > > > > works > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > pretty well with multicast w/o having to know too much > > > > > > > > > about > > > > > > > > > JGroups > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > config. S3_PING seams to be a viable way to get a cluster > > > > > > > > > running > > > > > > > > > in > > > > > > > > > AWS. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But additionally, my customer doesn’t have any (deep) > > > > > > > > > knowledge > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JBoss infrastructures and so I’m looking for a way to be > > > > > > > > > able > > > > > > > > > to > > > > > > > > > run > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keycloak in a cluster in AWS without the need to build up > > > > > > > > > deeper > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > knowlegde of JGroups config, for example in getting rid of > > > > > > > > > Infinispan. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I do understand all the concerns in doing this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still have to test S3_PING, if it works as easy as > > > > > > > > > multicast. > > > > > > > > > If > > > > > > > > > yes, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > we can use it, if no… I don’t know yet. But this gets > > > > > > > > > offtopic > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keycloak mailinglist, it’s more related to pure > > > > > > > > > Wildfly/Infinispan. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > seems to me it would be much easier to get Infinispan working > > > > > > > > on > > > > > > > > AWS > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > than to write and maintain an entire new caching mechanism > > > > > > > > and > > > > > > > > hope > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > don't refactor the cache SPI. > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > +1 > > > > > > > > > > > > > > > > > > > > >

> > > > > > > I am sure infinispan/JGroups has possibility to run in > > > > > > > non-multicast > > > > > > > > > > > > > > > > > > > > > > > > > > > > environment. You may just need to figure how exactly to > > > > > > > configure > > > > > > > it. > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that this issue is more related to Wildfly/Infinispan > > > > > > > itself > > > > > > > > > > > > > > > > > > > > > > > > > > > > than to Keycloak. > > > > > > > > > > > > > > > > > > > > >

> > > > > > > You may need to use jgroups protocols like TCP instead of > > > > > > > default > > > > > > > UDP > > > > > > > > > > > > > > > > > > > > > > > > > > > > and maybe TCPPING (this requires to manually list all your > > > > > > > cluster > > > > > > > > > > > > > > > > > > > > > > > > > > > > nodes. But still, it's much better option IMO than rewriting > > > > > > > UserSession > > > > > > > > > > > > > > > > > > > > > > > > > > > > SPI) > > > > > > > > > > > > > > > > > > > > >

> > > > > > Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING > > > > > > > > > > > > > > > > > > > > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , > > > > > > but > > > > > > it's > > > > > > > > > > > > > > > > > > > > > not official part of jgroups. > > > > > > > > > > > > > > >

> > > > > > Marek > > > > > > > > > > > > > > >

> > > > > > > Marek > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > > > > > > > > > > > > >

> > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > > > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > > > > > > >

> > > _______________________________________________ keycloak-user mailing > > > list > > > keycloak-user(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > _______________________________________________ > > > > > > keycloak-user mailing list > > > > > > keycloak-user(a)lists.jboss.org > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > >

> > _______________________________________________ > > > keycloak-user mailing list > > > keycloak-user(a)lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-user