[keycloak-user] Replace use of Infinispan with User Sessions SPI ?

Wed Dec 16 15:33:39 EST 2015

Hey Scott, 

Thanks for following up and showing me your code. I have some questions inline for you: 

----- Original Message -----

> From: "Scott Rossillo" <srossillo at smartling.com>
> To: "Alan Field" <afield at redhat.com>
> Cc: "Niko Köbler" <niko at n-k.de>, "keycloak-user"
> <keycloak-user at lists.jboss.org>
> Sent: Wednesday, December 16, 2015 2:19:27 PM
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI
> ?

> Hi Alan,

> Thanks for the informative email. The steps you outlined are similar to what
> I’ve tested with ECS. The gossip router is definitely a no-go for production
> since it’s a single point of failure.

It is possible to use the TUNNEL with multiple gossip routers to avoid this, but I understand not wanting to have to setup and maintain the extra gossip router processes. 

> I am testing this down at the JGroups level right now and got it working with
> ECS. There were two issues. On TCP you have to specify the external_addr to
> match the EC2 host otherwise the nodes won’t form a cluster. Secondly,
> FD_SOCK attempts to connect back on a random port. With Docker instances,
> this fails. Using a known client_bind_port works well.

Which IP address from your example is retrieved with this command: 

EXTERNAL_HOST_IP= $( curl http://169.254.169.254/latest/meta-data/local-ipv4 ) " 

Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this command in EC2, I get the internal IP address for the instance, but not the public IP address. In your example, that would be the 172.16.0.4 address. Also which address is used for the bind_addr when you use -Djgroups.bind_addr=global? 

> Here’s the code I’m testing with: https://github.com/foo4u/aws-infinispan-poc

> Most interesting are probably:

> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh

How are you setting the JGROUPS_INITIAL_HOSTS environment variable? 

> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/src/main/resources/tcp.xml

> With this set up the nodes on different machines communicate without issue. I
> still have to add in something other than TCP_PING, but that wasn’t the main
> issue. Will use JDBC_PING most likely. Not a fan of S3 for coordination.
> Plus I already need an RDBMS for Keycloak.

For my curiosity, can you tell me more about why you don't want to use S3_PING? Is it the cost or something else? Just wondering and JDBC_PING should work fine. 

Thanks, 
Alan 

> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com

> > On Dec 15, 2015, at 2:13 PM, Alan Field < afield at redhat.com > wrote:
> 

> > Just to be clear, I have successfully tested Infinispan library and server
> > mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses.
> > None of the cloud providers support multicast. The Docker case is a little
> > different though, because of the issues with getting access to the IP
> > address.
> 

> > Thanks,
> 
> > Alan
> 

> > ----- Original Message -----
> 

> > > From: "Niko Köbler" < niko at n-k.de >
> > 
> 
> > > To: "Paul Blair" < pblair at clearme.com >
> > 
> 
> > > Cc: "keycloak-user" < keycloak-user at lists.jboss.org >
> > 
> 
> > > Sent: Tuesday, December 15, 2015 1:53:18 PM
> > 
> 
> > > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> > > SPI
> > > ?
> > 
> 

> > > We will go for the first run with EC2 and S3_PING, but w/o Docker.
> > 
> 
> > > If we/you/whoever will find a proper solution (possibly on the jgroups
> > > mailinglist), we will test this.
> > 
> 

> > > Seams that everybody is aware of the Docker/Cloud/Multicast issues, but
> > > no-one has a proper solution, only workarounds. :(
> > 
> 

> > > > Am 15.12.2015 um 15:47 schrieb Paul Blair < pblair at clearme.com >:
> > > 
> > 
> 

> > > > I've also been working on setting up clustered Keycloak on Docker
> > > > containers
> > > > in EC2 and would be interested in any potential solutions for this
> > > > configuration.
> > > 
> > 
> 

> > > > Alternatively I've set up on EC2 without Docker with S3_PING. I'd be
> > > > interested in hearing about the issues with this configuration.
> > > 
> > 
> 

> > > > From: Scott Rossillo < srossillo at smartling.com >
> > > 
> > 
> 
> > > > Date: Mon, 14 Dec 2015 18:31:30 -0500
> > > 
> > 
> 
> > > > To: Marek Posolda < mposolda at redhat.com >, < afield at redhat.com >
> > > 
> > 
> 
> > > > Cc: keycloak-user < keycloak-user at lists.jboss.org >
> > > 
> > 
> 
> > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User
> > > > Sessions
> > > > SPI
> > > > ?
> > > 
> > 
> 

> > > > There are two issues:
> > > 
> > 
> 

> > > > 1. Infinispan relies on JGroups, which is difficult to configure
> > > > correctly
> > > > with the various ping techniques that aren’t UDP multicast. I can
> > > > elaborate
> > > > on each one that we tested but it’s just generally complex to get
> > > > right.
> > > > That’s not to say it’s impossible or the biggest reason this is
> > > > complicated
> > > > on ECS or _insert container service here_, see #2 for that.
> > > 
> > 
> 

> > > > 2. It is difficult to do discovery correctly with JGroups and Docker.
> > > > Non-privileged Docker instances - the default and recommend type - do
> > > > not
> > > > implicitly know their host’s IP. This causes IP mismatches between what
> > > > JGroups thinks the machine’s IP is and what it actually is when
> > > > connecting
> > > > to hosts on different machines. This is the main issue and it’s not the
> > > > fault of JGroups per se, but there’s no simple work around.
> > > 
> > 
> 

> > > > Take for example a simple 2 node cluster:
> > > 
> > 
> 

> > > > Node 1 comes up on the docker0 interface of host A with the IP address
> > > > 172.16.0.4. The host A IP is 10.10.0.100.
> > > 
> > 
> 
> > > > Node 2 comes up on the docker0 interface of host B with the IP address
> > > > 172.16.0.8. The host B IP is 10.10.0.108.
> > > 
> > 
> 

> > > > The 172.16 network is not routable between hosts (by design). Docker
> > > > does
> > > > port forwarding for ports we wish to expose to this works fine for
> > > > HTTP/HTTPS but not the cluster traffic.
> > > 
> > 
> 

> > > > So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
> > > > advertises 172.16.0.8. The two cannot talk to each other by default.
> > > > However, using the hard coded IPs and TCP PING, we can set
> > > > external_addr
> > > > on
> > > > Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and
> > > > set
> > > > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
> > > > discover each other. However, they will not form a cluster. The nodes
> > > > will
> > > > reject the handshake thinking they’re not actually 10.10.0.100 or
> > > > 10.10.0.108 respectively.
> > > 
> > 
> 

> > > > I’d like to discuss further and I can share where we’ve gotten so far
> > > > with
> > > > workarounds to this but it may be better to get into the weeds on
> > > > another
> > > > list.
> > > 
> > 
> 

> > > > Let me know what you think.
> > > 
> > 
> 

> > > > Best,
> > > 
> > 
> 
> > > > Scott
> > > 
> > 
> 

> > > > Scott Rossillo
> > > 
> > 
> 
> > > > Smartling | Senior Software Engineer
> > > 
> > 
> 
> > > > srossillo at smartling.com
> > > 
> > 
> 

> > > > > On Dec 14, 2015, at 5:32 PM, Marek Posolda < mposolda at redhat.com >
> > > > > wrote:
> > > > 
> > > 
> > 
> 

> > > > > CCing Alan Field from RH Infinispan team and forwarding his question:
> > > > 
> > > 
> > 
> 
> > > > > I'd like to know which configuration files you are using and why is
> > > > > is
> > > > 
> > > 
> > 
> 
> > > > > harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd
> > > > > also
> > > > > be
> > > > 
> > > 
> > 
> 
> > > > > interested in how big a cluster you are using in AWS.
> > > > 
> > > 
> > 
> 

> > > > > On 14/12/15 22:24, Scott Rossillo wrote:
> > > > 
> > > 
> > 
> 

> > > > > > AWS was why we didn’t use Infinispan to begin with. That and it’s
> > > > > > even
> > > > > > more
> > > > > > complicated when you deploy using Amazon’s Docker service (ECS) or
> > > > > > Beanstalk.
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > It’s too bad Infinispan / JGroups are beasts when the out of the
> > > > > > box
> > > > > > configuration can’t be used. I’m planning to document this as we
> > > > > > fix
> > > > > > but
> > > > > > I’d
> > > > > > avoid S3_PING and use JDBC_PING. You already need JDBC for the
> > > > > > Keycloak
> > > > > > DB,
> > > > > > unless you’re using Mongo and it’s easier to test locally.
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > TCPPING will bite you on AWS if Amazon decides to replace one of
> > > > > > your
> > > > > > instances (which it does occasionally w/ECS or Beanstalk).
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Best,
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > Scott
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Scott Rossillo
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > Smartling | Senior Software Engineer
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > srossillo at smartling.com
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > On Dec 14, 2015, at 10:59 AM, Marek Posolda < mposolda at redhat.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > On 14/12/15 16:55, Marek Posolda wrote:
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > On 14/12/15 15:58, Bill Burke wrote:
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > On 12/14/2015 5:01 AM, Niko Köbler wrote:
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > > Hi Marek,
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > > > Am 14.12.2015 um 08:50 schrieb Marek Posolda <
> > > > > > > > > > > mposolda at redhat.com
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > > < mailto:mposolda at redhat.com >>:
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > > > Btv. what's your motivation to not use infinispan? If you
> > > > > > > > > > > afraid
> > > > > > > > > > > of
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > > cluster communication, you don't need to worry much about
> > > > > > > > > > > it,
> > > > > > > > > > > because
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > > if you run single keycloak through standalone.xml, the
> > > > > > > > > > > infinispan
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > > automatically works in LOCAL mode and there is no any
> > > > > > > > > > > cluster
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > > communication at all.
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > > My current customer is running his apps in AWS. As known,
> > > > > > > > > > multicast
> > > > > > > > > > is
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > not available in cloud infrastructures. Wildfly/Infinispan
> > > > > > > > > > Cluster
> > > > > > > > > > works
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > pretty well with multicast w/o having to know too much
> > > > > > > > > > about
> > > > > > > > > > JGroups
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > config. S3_PING seams to be a viable way to get a cluster
> > > > > > > > > > running
> > > > > > > > > > in
> > > > > > > > > > AWS.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > But additionally, my customer doesn’t have any (deep)
> > > > > > > > > > knowledge
> > > > > > > > > > about
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > JBoss infrastructures and so I’m looking for a way to be
> > > > > > > > > > able
> > > > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > Keycloak in a cluster in AWS without the need to build up
> > > > > > > > > > deeper
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > knowlegde of JGroups config, for example in getting rid of
> > > > > > > > > > Infinispan.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > But I do understand all the concerns in doing this.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > I still have to test S3_PING, if it works as easy as
> > > > > > > > > > multicast.
> > > > > > > > > > If
> > > > > > > > > > yes,
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > we can use it, if no… I don’t know yet. But this gets
> > > > > > > > > > offtopic
> > > > > > > > > > for
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > > Keycloak mailinglist, it’s more related to pure
> > > > > > > > > > Wildfly/Infinispan.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > > seems to me it would be much easier to get Infinispan working
> > > > > > > > > on
> > > > > > > > > AWS
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > than to write and maintain an entire new caching mechanism
> > > > > > > > > and
> > > > > > > > > hope
> > > > > > > > > we
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > > don't refactor the cache SPI.
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > +1
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > I am sure infinispan/JGroups has possibility to run in
> > > > > > > > non-multicast
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > environment. You may just need to figure how exactly to
> > > > > > > > configure
> > > > > > > > it.
> > > > > > > > So
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > I agree that this issue is more related to Wildfly/Infinispan
> > > > > > > > itself
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > than to Keycloak.
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > You may need to use jgroups protocols like TCP instead of
> > > > > > > > default
> > > > > > > > UDP
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > and maybe TCPPING (this requires to manually list all your
> > > > > > > > cluster
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > nodes. But still, it's much better option IMO than rewriting
> > > > > > > > UserSession
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > SPI)
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 ,
> > > > > > > but
> > > > > > > it's
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > not official part of jgroups.
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > Marek
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > > Marek
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > _______________________________________________
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > keycloak-user mailing list
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > keycloak-user at lists.jboss.org
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > > _______________________________________________
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > keycloak-user mailing list
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > keycloak-user at lists.jboss.org
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > _______________________________________________ keycloak-user mailing
> > > > list
> > > > keycloak-user at lists.jboss.org
> > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > 
> > 
> 
> > > > _______________________________________________
> > > 
> > 
> 
> > > > keycloak-user mailing list
> > > 
> > 
> 
> > > > keycloak-user at lists.jboss.org
> > > 
> > 
> 
> > > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > > 
> > 
> 

> > > _______________________________________________
> > 
> 
> > > keycloak-user mailing list
> > 
> 
> > > keycloak-user at lists.jboss.org
> > 
> 
> > > https://lists.jboss.org/mailman/listinfo/keycloak-user
> > 
> 
> > _______________________________________________
> 
> > keycloak-user mailing list
> 
> > keycloak-user at lists.jboss.org
> 
> > https://lists.jboss.org/mailman/listinfo/keycloak-user
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20151216/bd611b6a/attachment-0001.html 