[keycloak-user] Replace use of Infinispan with User Sessions SPI ?

Alan Field afield at redhat.com
Tue Dec 15 14:13:55 EST 2015 

Just to be clear, I have successfully tested Infinispan library and server mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses. None of the cloud providers support multicast. The Docker case is a little different though, because of the issues with getting access to the IP address. 

Thanks, 
Alan 

----- Original Message -----

> From: "Niko Köbler" <niko at n-k.de>
> To: "Paul Blair" <pblair at clearme.com>
> Cc: "keycloak-user" <keycloak-user at lists.jboss.org>
> Sent: Tuesday, December 15, 2015 1:53:18 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
> ?

> We will go for the first run with EC2 and S3_PING, but w/o Docker.
> If we/you/whoever will find a proper solution (possibly on the jgroups
> mailinglist), we will test this.

> Seams that everybody is aware of the Docker/Cloud/Multicast issues, but
> no-one has a proper solution, only workarounds. :(

> > Am 15.12.2015 um 15:47 schrieb Paul Blair < pblair at clearme.com >:
> 

> > I've also been working on setting up clustered Keycloak on Docker
> > containers
> > in EC2 and would be interested in any potential solutions for this
> > configuration.
> 

> > Alternatively I've set up on EC2 without Docker with S3_PING. I'd be
> > interested in hearing about the issues with this configuration.
> 

> > From: Scott Rossillo < srossillo at smartling.com >
> 
> > Date: Mon, 14 Dec 2015 18:31:30 -0500
> 
> > To: Marek Posolda < mposolda at redhat.com >, < afield at redhat.com >
> 
> > Cc: keycloak-user < keycloak-user at lists.jboss.org >
> 
> > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> > SPI
> > ?
> 

> > There are two issues:
> 

> > 1. Infinispan relies on JGroups, which is difficult to configure correctly
> > with the various ping techniques that aren’t UDP multicast. I can elaborate
> > on each one that we tested but it’s just generally complex to get right.
> > That’s not to say it’s impossible or the biggest reason this is complicated
> > on ECS or _insert container service here_, see #2 for that.
> 

> > 2. It is difficult to do discovery correctly with JGroups and Docker.
> > Non-privileged Docker instances - the default and recommend type - do not
> > implicitly know their host’s IP. This causes IP mismatches between what
> > JGroups thinks the machine’s IP is and what it actually is when connecting
> > to hosts on different machines. This is the main issue and it’s not the
> > fault of JGroups per se, but there’s no simple work around.
> 

> > Take for example a simple 2 node cluster:
> 

> > Node 1 comes up on the docker0 interface of host A with the IP address
> > 172.16.0.4. The host A IP is 10.10.0.100.
> 
> > Node 2 comes up on the docker0 interface of host B with the IP address
> > 172.16.0.8. The host B IP is 10.10.0.108.
> 

> > The 172.16 network is not routable between hosts (by design). Docker does
> > port forwarding for ports we wish to expose to this works fine for
> > HTTP/HTTPS but not the cluster traffic.
> 

> > So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
> > advertises 172.16.0.8. The two cannot talk to each other by default.
> > However, using the hard coded IPs and TCP PING, we can set external_addr on
> > Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set
> > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
> > discover each other. However, they will not form a cluster. The nodes will
> > reject the handshake thinking they’re not actually 10.10.0.100 or
> > 10.10.0.108 respectively.
> 

> > I’d like to discuss further and I can share where we’ve gotten so far with
> > workarounds to this but it may be better to get into the weeds on another
> > list.
> 

> > Let me know what you think.
> 

> > Best,
> 
> > Scott
> 

> > Scott Rossillo
> 
> > Smartling | Senior Software Engineer
> 
> > srossillo at smartling.com
> 

> > > On Dec 14, 2015, at 5:32 PM, Marek Posolda < mposolda at redhat.com > wrote:
> > 
> 

> > > CCing Alan Field from RH Infinispan team and forwarding his question:
> > 
> 
> > > I'd like to know which configuration files you are using and why is is
> > 
> 
> > > harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also
> > > be
> > 
> 
> > > interested in how big a cluster you are using in AWS.
> > 
> 

> > > On 14/12/15 22:24, Scott Rossillo wrote:
> > 
> 

> > > > AWS was why we didn’t use Infinispan to begin with. That and it’s even
> > > > more
> > > > complicated when you deploy using Amazon’s Docker service (ECS) or
> > > > Beanstalk.
> > > 
> > 
> 

> > > > It’s too bad Infinispan / JGroups are beasts when the out of the box
> > > > configuration can’t be used. I’m planning to document this as we fix
> > > > but
> > > > I’d
> > > > avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak
> > > > DB,
> > > > unless you’re using Mongo and it’s easier to test locally.
> > > 
> > 
> 

> > > > TCPPING will bite you on AWS if Amazon decides to replace one of your
> > > > instances (which it does occasionally w/ECS or Beanstalk).
> > > 
> > 
> 

> > > > Best,
> > > 
> > 
> 
> > > > Scott
> > > 
> > 
> 

> > > > Scott Rossillo
> > > 
> > 
> 
> > > > Smartling | Senior Software Engineer
> > > 
> > 
> 
> > > > srossillo at smartling.com
> > > 
> > 
> 

> > > > > On Dec 14, 2015, at 10:59 AM, Marek Posolda < mposolda at redhat.com >
> > > > > wrote:
> > > > 
> > > 
> > 
> 

> > > > > On 14/12/15 16:55, Marek Posolda wrote:
> > > > 
> > > 
> > 
> 

> > > > > > On 14/12/15 15:58, Bill Burke wrote:
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > On 12/14/2015 5:01 AM, Niko Köbler wrote:
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > Hi Marek,
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > Am 14.12.2015 um 08:50 schrieb Marek Posolda <
> > > > > > > > > mposolda at redhat.com
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > < mailto:mposolda at redhat.com >>:
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > Btv. what's your motivation to not use infinispan? If you
> > > > > > > > > afraid
> > > > > > > > > of
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > cluster communication, you don't need to worry much about it,
> > > > > > > > > because
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > if you run single keycloak through standalone.xml, the
> > > > > > > > > infinispan
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > automatically works in LOCAL mode and there is no any cluster
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > communication at all.
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > My current customer is running his apps in AWS. As known,
> > > > > > > > multicast
> > > > > > > > is
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > not available in cloud infrastructures. Wildfly/Infinispan
> > > > > > > > Cluster
> > > > > > > > works
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > pretty well with multicast w/o having to know too much about
> > > > > > > > JGroups
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > config. S3_PING seams to be a viable way to get a cluster
> > > > > > > > running
> > > > > > > > in
> > > > > > > > AWS.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > But additionally, my customer doesn’t have any (deep) knowledge
> > > > > > > > about
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > JBoss infrastructures and so I’m looking for a way to be able
> > > > > > > > to
> > > > > > > > run
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > Keycloak in a cluster in AWS without the need to build up
> > > > > > > > deeper
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > knowlegde of JGroups config, for example in getting rid of
> > > > > > > > Infinispan.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > But I do understand all the concerns in doing this.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > I still have to test S3_PING, if it works as easy as multicast.
> > > > > > > > If
> > > > > > > > yes,
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > we can use it, if no… I don’t know yet. But this gets offtopic
> > > > > > > > for
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > Keycloak mailinglist, it’s more related to pure
> > > > > > > > Wildfly/Infinispan.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > seems to me it would be much easier to get Infinispan working on
> > > > > > > AWS
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > than to write and maintain an entire new caching mechanism and
> > > > > > > hope
> > > > > > > we
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > don't refactor the cache SPI.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > +1
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > I am sure infinispan/JGroups has possibility to run in
> > > > > > non-multicast
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > environment. You may just need to figure how exactly to configure
> > > > > > it.
> > > > > > So
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > I agree that this issue is more related to Wildfly/Infinispan
> > > > > > itself
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > than to Keycloak.
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > You may need to use jgroups protocols like TCP instead of default
> > > > > > UDP
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > and maybe TCPPING (this requires to manually list all your cluster
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > nodes. But still, it's much better option IMO than rewriting
> > > > > > UserSession
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > SPI)
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
> > > > 
> > > 
> > 
> 
> > > > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but
> > > > > it's
> > > > 
> > > 
> > 
> 
> > > > > not official part of jgroups.
> > > > 
> > > 
> > 
> 

> > > > > Marek
> > > > 
> > > 
> > 
> 

> > > > > > Marek
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > _______________________________________________
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > keycloak-user mailing list
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > keycloak-user at lists.jboss.org
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > _______________________________________________
> > > > 
> > > 
> > 
> 
> > > > > keycloak-user mailing list
> > > > 
> > > 
> > 
> 
> > > > > keycloak-user at lists.jboss.org
> > > > 
> > > 
> > 
> 
> > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > 
> > > 
> > 
> 

> > _______________________________________________ keycloak-user mailing list
> > keycloak-user at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> 
> > _______________________________________________
> 
> > keycloak-user mailing list
> 
> > keycloak-user at lists.jboss.org
> 
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> 

> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20151215/70745953/attachment-0001.html

More information about the keycloak-user mailing list