Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?

From: "Niko Köbler" <niko(a)n-k.de&gt; To: "Paul Blair" <pblair(a)clearme.com&gt; Cc: "keycloak-user" <keycloak-user(a)lists.jboss.org&gt; Sent: Tuesday, December 15, 2015 1:53:18 PM Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?

We will go for the first run with EC2 and S3_PING, but w/o Docker. If we/you/whoever will find a proper solution (possibly on the jgroups mailinglist), we will test this.

Seams that everybody is aware of the Docker/Cloud/Multicast issues, but no-one has a proper solution, only workarounds. :(

> Am 15.12.2015 um 15:47 schrieb Paul Blair < pblair(a)clearme.com >:

> I've also been working on setting up clustered Keycloak on Docker > containers > in EC2 and would be interested in any potential solutions for this > configuration.

> Alternatively I've set up on EC2 without Docker with S3_PING. I'd be > interested in hearing about the issues with this configuration.

> From: Scott Rossillo < srossillo(a)smartling.com > > Date: Mon, 14 Dec 2015 18:31:30 -0500 > To: Marek Posolda < mposolda(a)redhat.com >, < afield(a)redhat.com > > Cc: keycloak-user < keycloak-user(a)lists.jboss.org > > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions > SPI > ?

> There are two issues:

> 1. Infinispan relies on JGroups, which is difficult to configure correctly > with the various ping techniques that aren’t UDP multicast. I can elaborate > on each one that we tested but it’s just generally complex to get right. > That’s not to say it’s impossible or the biggest reason this is complicated > on ECS or _insert container service here_, see #2 for that.

> 2. It is difficult to do discovery correctly with JGroups and Docker. > Non-privileged Docker instances - the default and recommend type - do not > implicitly know their host’s IP. This causes IP mismatches between what > JGroups thinks the machine’s IP is and what it actually is when connecting > to hosts on different machines. This is the main issue and it’s not the > fault of JGroups per se, but there’s no simple work around.

> Take for example a simple 2 node cluster:

> Node 1 comes up on the docker0 interface of host A with the IP address > 172.16.0.4. The host A IP is 10.10.0.100. > Node 2 comes up on the docker0 interface of host B with the IP address > 172.16.0.8. The host B IP is 10.10.0.108.

> The 172.16 network is not routable between hosts (by design). Docker does > port forwarding for ports we wish to expose to this works fine for > HTTP/HTTPS but not the cluster traffic.

> So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2 > advertises 172.16.0.8. The two cannot talk to each other by default. > However, using the hard coded IPs and TCP PING, we can set external_addr on > Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to > discover each other. However, they will not form a cluster. The nodes will > reject the handshake thinking they’re not actually 10.10.0.100 or > 10.10.0.108 respectively.

> I’d like to discuss further and I can share where we’ve gotten so far with > workarounds to this but it may be better to get into the weeds on another > list.

> Let me know what you think.

> Best, > Scott

> Scott Rossillo > Smartling | Senior Software Engineer > srossillo(a)smartling.com

> > On Dec 14, 2015, at 5:32 PM, Marek Posolda < mposolda(a)redhat.com > wrote: >

> > CCing Alan Field from RH Infinispan team and forwarding his question: > > > I'd like to know which configuration files you are using and why is is > > > harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also > > be > > > interested in how big a cluster you are using in AWS. >

> > On 14/12/15 22:24, Scott Rossillo wrote: >

> > > AWS was why we didn’t use Infinispan to begin with. That and it’s even > > > more > > > complicated when you deploy using Amazon’s Docker service (ECS) or > > > Beanstalk. > > >

> > > It’s too bad Infinispan / JGroups are beasts when the out of the box > > > configuration can’t be used. I’m planning to document this as we fix > > > but > > > I’d > > > avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak > > > DB, > > > unless you’re using Mongo and it’s easier to test locally. > > >

> > > TCPPING will bite you on AWS if Amazon decides to replace one of your > > > instances (which it does occasionally w/ECS or Beanstalk). > > >

> > > Best, > > > > > > Scott > > >

> > > Scott Rossillo > > > > > > Smartling | Senior Software Engineer > > > > > > srossillo(a)smartling.com > > >

> > > > On Dec 14, 2015, at 10:59 AM, Marek Posolda < mposolda(a)redhat.com > > > > > wrote: > > > > > >

> > > > On 14/12/15 16:55, Marek Posolda wrote: > > > > > >

> > > > > On 14/12/15 15:58, Bill Burke wrote: > > > > > > > > > >

> > > > > > On 12/14/2015 5:01 AM, Niko Köbler wrote: > > > > > > > > > > > > > > >

> > > > > > > Hi Marek, > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > Am 14.12.2015 um 08:50 schrieb Marek Posolda < > > > > > > > > mposolda(a)redhat.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < mailto:mposolda@redhat.com >>: > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > Btv. what's your motivation to not use infinispan? If you > > > > > > > > afraid > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cluster communication, you don't need to worry much about it, > > > > > > > > because > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if you run single keycloak through standalone.xml, the > > > > > > > > infinispan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > automatically works in LOCAL mode and there is no any cluster > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > communication at all. > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > My current customer is running his apps in AWS. As known, > > > > > > > multicast > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > > not available in cloud infrastructures. Wildfly/Infinispan > > > > > > > Cluster > > > > > > > works > > > > > > > > > > > > > > > > > > > > > > > > > > > > pretty well with multicast w/o having to know too much about > > > > > > > JGroups > > > > > > > > > > > > > > > > > > > > > > > > > > > > config. S3_PING seams to be a viable way to get a cluster > > > > > > > running > > > > > > > in > > > > > > > AWS. > > > > > > > > > > > > > > > > > > > > > > > > > > > > But additionally, my customer doesn’t have any (deep) knowledge > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > > > > > > > JBoss infrastructures and so I’m looking for a way to be able > > > > > > > to > > > > > > > run > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keycloak in a cluster in AWS without the need to build up > > > > > > > deeper > > > > > > > > > > > > > > > > > > > > > > > > > > > > knowlegde of JGroups config, for example in getting rid of > > > > > > > Infinispan. > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I do understand all the concerns in doing this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still have to test S3_PING, if it works as easy as multicast. > > > > > > > If > > > > > > > yes, > > > > > > > > > > > > > > > > > > > > > > > > > > > > we can use it, if no… I don’t know yet. But this gets offtopic > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keycloak mailinglist, it’s more related to pure > > > > > > > Wildfly/Infinispan. > > > > > > > > > > > > > > > > > > > > >

> > > > > > seems to me it would be much easier to get Infinispan working on > > > > > > AWS > > > > > > > > > > > > > > > > > > > > > than to write and maintain an entire new caching mechanism and > > > > > > hope > > > > > > we > > > > > > > > > > > > > > > > > > > > > don't refactor the cache SPI. > > > > > > > > > > > > > > >

> > > > > +1 > > > > > > > > > >

> > > > > I am sure infinispan/JGroups has possibility to run in > > > > > non-multicast > > > > > > > > > > > > > > > environment. You may just need to figure how exactly to configure > > > > > it. > > > > > So > > > > > > > > > > > > > > > I agree that this issue is more related to Wildfly/Infinispan > > > > > itself > > > > > > > > > > > > > > > than to Keycloak. > > > > > > > > > >

> > > > > You may need to use jgroups protocols like TCP instead of default > > > > > UDP > > > > > > > > > > > > > > > and maybe TCPPING (this requires to manually list all your cluster > > > > > > > > > > > > > > > nodes. But still, it's much better option IMO than rewriting > > > > > UserSession > > > > > > > > > > > > > > > SPI) > > > > > > > > > >

> > > > Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING > > > > > > > > > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but > > > > it's > > > > > > > > > > not official part of jgroups. > > > > > >

> > > > Marek > > > > > >

> > > > > Marek > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > >

> > > > _______________________________________________ > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > >

> _______________________________________________ keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-user > _______________________________________________ > keycloak-user mailing list > keycloak-user(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-user

_______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user