Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?

From: "Scott Rossillo" <srossillo(a)smartling.com&gt; To: "Alan Field" <afield(a)redhat.com&gt; Cc: "Niko Köbler" <niko(a)n-k.de&gt;, "keycloak-user" <keycloak-user(a)lists.jboss.org&gt; Sent: Wednesday, December 16, 2015 3:45:40 PM Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions SPI ?

Hi Alan,

> It is possible to use the TUNNEL with multiple gossip routers to avoid > this, but I understand not wanting to have to setup and maintain the extra > gossip router processes.

True, it’s mainly about maintaining extra components.

> Which IP address from your example is retrieved with this command: > EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4 > )”

I get the Amazon EC2 instance’s internal IP. This is what I want. There’s another endpoint for public but I don’t want to use it. What’s good about this is when called from inside a Docker container, I manage to get the actual internal IP for the EC2 instance.

> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?

Since this was a test with just 2 known hosts, I injected them as a Docker environment variable with two fixed IPs. Once we switch to JDBC_PING, this will be removed.

> For my curiosity, can you tell me more about why you don't want to use > S3_PING? Is it the cost or something else? Just wondering and JDBC_PING > should work fine.

S3_PING, like Gossip Router adds an external dependency on another service. S3 has had consistency issues 3 times in 2015 (at least in US East). I don’t want to rely another component when I already need the database to be up. Less components, less chance of failure. Also, there are ton of variables to set with S3 and it requires preliminary work. I want something that scales well from dev to QA to prod. JDBC_PING has a datasource_jndi_name property. I can just reuse the data source I set up for Keycloak.

I hope I got all your questions.

Best, Scott

Scott Rossillo Smartling | Senior Software Engineer srossillo(a)smartling.com

> On Dec 16, 2015, at 3:33 PM, Alan Field < afield(a)redhat.com > wrote:

> Hey Scott,

> Thanks for following up and showing me your code. I have some questions > inline for you:

> ----- Original Message -----

> > From: "Scott Rossillo" < srossillo(a)smartling.com > > > > To: "Alan Field" < afield(a)redhat.com > > > > Cc: "Niko Köbler" < niko(a)n-k.de >, "keycloak-user" < > > keycloak-user(a)lists.jboss.org > > > > Sent: Wednesday, December 16, 2015 2:19:27 PM > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions > > SPI > > ? >

Hi Alan,

> > Thanks for the informative email. The steps you outlined are similar to > > what > > I’ve tested with ECS. The gossip router is definitely a no-go for > > production > > since it’s a single point of failure. >

> It is possible to use the TUNNEL with multiple gossip routers to avoid > this, > but I understand not wanting to have to setup and maintain the extra gossip > router processes.

> > I am testing this down at the JGroups level right now and got it working > > with > > ECS. There were two issues. On TCP you have to specify the external_addr > > to > > match the EC2 host otherwise the nodes won’t form a cluster. Secondly, > > FD_SOCK attempts to connect back on a random port. With Docker instances, > > this fails. Using a known client_bind_port works well. >

> Which IP address from your example is retrieved with this command:

> EXTERNAL_HOST_IP= $( curl > http://169.254.169.254/latest/meta-data/local-ipv4 > ) "

> Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this > command in EC2, I get the internal IP address for the instance, but not the > public IP address. In your example, that would be the 172.16.0.4 address. > Also which address is used for the bind_addr when you use > -Djgroups.bind_addr=global?

> > Here’s the code I’m testing with: > > https://github.com/foo4u/aws-infinispan-poc >

> > Most interesting are probably: >

> > https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/e... >

> How are you setting the JGROUPS_INITIAL_HOSTS environment variable?

> > https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/s... >

> > With this set up the nodes on different machines communicate without > > issue. > > I > > still have to add in something other than TCP_PING, but that wasn’t the > > main > > issue. Will use JDBC_PING most likely. Not a fan of S3 for coordination. > > Plus I already need an RDBMS for Keycloak. >

> For my curiosity, can you tell me more about why you don't want to use > S3_PING? Is it the cost or something else? Just wondering and JDBC_PING > should work fine.

> Thanks, > Alan

> > Scott Rossillo > > > Smartling | Senior Software Engineer > > > srossillo(a)smartling.com >

> > > On Dec 15, 2015, at 2:13 PM, Alan Field < afield(a)redhat.com > wrote: > > >

> > > Just to be clear, I have successfully tested Infinispan library and > > > server > > > mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP > > > addresses. > > > None of the cloud providers support multicast. The Docker case is a > > > little > > > different though, because of the issues with getting access to the IP > > > address. > > >

> > > Thanks, > > > > > > Alan > > >

> > > ----- Original Message ----- > > >

> > > > From: "Niko Köbler" < niko(a)n-k.de > > > > > > > > > > > To: "Paul Blair" < pblair(a)clearme.com > > > > > > > > > > > Cc: "keycloak-user" < keycloak-user(a)lists.jboss.org > > > > > > > > > > > Sent: Tuesday, December 15, 2015 1:53:18 PM > > > > > > > > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User > > > > Sessions > > > > SPI > > > > ? > > > > > >

> > > > We will go for the first run with EC2 and S3_PING, but w/o Docker. > > > > > > > > > > If we/you/whoever will find a proper solution (possibly on the > > > > jgroups > > > > mailinglist), we will test this. > > > > > >

> > > > Seams that everybody is aware of the Docker/Cloud/Multicast issues, > > > > but > > > > no-one has a proper solution, only workarounds. :( > > > > > >

> > > > > Am 15.12.2015 um 15:47 schrieb Paul Blair < pblair(a)clearme.com >: > > > > > > > > > >

> > > > > I've also been working on setting up clustered Keycloak on Docker > > > > > containers > > > > > in EC2 and would be interested in any potential solutions for this > > > > > configuration. > > > > > > > > > >

> > > > > Alternatively I've set up on EC2 without Docker with S3_PING. I'd > > > > > be > > > > > interested in hearing about the issues with this configuration. > > > > > > > > > >

> > > > > From: Scott Rossillo < srossillo(a)smartling.com > > > > > > > > > > > > > > > > Date: Mon, 14 Dec 2015 18:31:30 -0500 > > > > > > > > > > > > > > > To: Marek Posolda < mposolda(a)redhat.com >, < afield(a)redhat.com > > > > > > > > > > > > > > > > Cc: keycloak-user < keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > > Subject: Re: [keycloak-user] Replace use of Infinispan with User > > > > > Sessions > > > > > SPI > > > > > ? > > > > > > > > > >

> > > > > There are two issues: > > > > > > > > > >

> > > > > 1. Infinispan relies on JGroups, which is difficult to configure > > > > > correctly > > > > > with the various ping techniques that aren’t UDP multicast. I can > > > > > elaborate > > > > > on each one that we tested but it’s just generally complex to get > > > > > right. > > > > > That’s not to say it’s impossible or the biggest reason this is > > > > > complicated > > > > > on ECS or _insert container service here_, see #2 for that. > > > > > > > > > >

> > > > > 2. It is difficult to do discovery correctly with JGroups and > > > > > Docker. > > > > > Non-privileged Docker instances - the default and recommend type - > > > > > do > > > > > not > > > > > implicitly know their host’s IP. This causes IP mismatches between > > > > > what > > > > > JGroups thinks the machine’s IP is and what it actually is when > > > > > connecting > > > > > to hosts on different machines. This is the main issue and it’s not > > > > > the > > > > > fault of JGroups per se, but there’s no simple work around. > > > > > > > > > >

> > > > > Take for example a simple 2 node cluster: > > > > > > > > > >

> > > > > Node 1 comes up on the docker0 interface of host A with the IP > > > > > address > > > > > 172.16.0.4. The host A IP is 10.10.0.100. > > > > > > > > > > > > > > > Node 2 comes up on the docker0 interface of host B with the IP > > > > > address > > > > > 172.16.0.8. The host B IP is 10.10.0.108. > > > > > > > > > >

> > > > > The 172.16 network is not routable between hosts (by design). > > > > > Docker > > > > > does > > > > > port forwarding for ports we wish to expose to this works fine for > > > > > HTTP/HTTPS but not the cluster traffic. > > > > > > > > > >

> > > > > So Node 1 will advertise itself as having IP 172.16.0.4 while Node > > > > > 2 > > > > > advertises 172.16.0.8. The two cannot talk to each other by > > > > > default. > > > > > However, using the hard coded IPs and TCP PING, we can set > > > > > external_addr > > > > > on > > > > > Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 > > > > > and > > > > > set > > > > > initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the > > > > > nodes > > > > > to > > > > > discover each other. However, they will not form a cluster. The > > > > > nodes > > > > > will > > > > > reject the handshake thinking they’re not actually 10.10.0.100 or > > > > > 10.10.0.108 respectively. > > > > > > > > > >

> > > > > I’d like to discuss further and I can share where we’ve gotten so > > > > > far > > > > > with > > > > > workarounds to this but it may be better to get into the weeds on > > > > > another > > > > > list. > > > > > > > > > >

> > > > > Let me know what you think. > > > > > > > > > >

> > > > > Best, > > > > > > > > > > > > > > > Scott > > > > > > > > > >

> > > > > Scott Rossillo > > > > > > > > > > > > > > > Smartling | Senior Software Engineer > > > > > > > > > > > > > > > srossillo(a)smartling.com > > > > > > > > > >

> > > > > > On Dec 14, 2015, at 5:32 PM, Marek Posolda < mposolda(a)redhat.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >

> > > > > > CCing Alan Field from RH Infinispan team and forwarding his > > > > > > question: > > > > > > > > > > > > > > > > > > > > > I'd like to know which configuration files you are using and why > > > > > > is > > > > > > is > > > > > > > > > > > > > > > > > > > > > harder to use with Amazon’s Docker service (ECS) or Beanstalk. > > > > > > I'd > > > > > > also > > > > > > be > > > > > > > > > > > > > > > > > > > > > interested in how big a cluster you are using in AWS. > > > > > > > > > > > > > > >

> > > > > > On 14/12/15 22:24, Scott Rossillo wrote: > > > > > > > > > > > > > > >

> > > > > > > AWS was why we didn’t use Infinispan to begin with. That and > > > > > > > it’s > > > > > > > even > > > > > > > more > > > > > > > complicated when you deploy using Amazon’s Docker service (ECS) > > > > > > > or > > > > > > > Beanstalk. > > > > > > > > > > > > > > > > > > > > >

> > > > > > > It’s too bad Infinispan / JGroups are beasts when the out of > > > > > > > the > > > > > > > box > > > > > > > configuration can’t be used. I’m planning to document this as > > > > > > > we > > > > > > > fix > > > > > > > but > > > > > > > I’d > > > > > > > avoid S3_PING and use JDBC_PING. You already need JDBC for the > > > > > > > Keycloak > > > > > > > DB, > > > > > > > unless you’re using Mongo and it’s easier to test locally. > > > > > > > > > > > > > > > > > > > > >

> > > > > > > TCPPING will bite you on AWS if Amazon decides to replace one > > > > > > > of > > > > > > > your > > > > > > > instances (which it does occasionally w/ECS or Beanstalk). > > > > > > > > > > > > > > > > > > > > >

> > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Scott > > > > > > > > > > > > > > > > > > > > >

> > > > > > > Scott Rossillo > > > > > > > > > > > > > > > > > > > > > > > > > > > > Smartling | Senior Software Engineer > > > > > > > > > > > > > > > > > > > > > > > > > > > > srossillo(a)smartling.com > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > On Dec 14, 2015, at 10:59 AM, Marek Posolda < > > > > > > > > mposolda(a)redhat.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > On 14/12/15 16:55, Marek Posolda wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > On 14/12/15 15:58, Bill Burke wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > On 12/14/2015 5:01 AM, Niko Köbler wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > > Hi Marek, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > > > Am 14.12.2015 um 08:50 schrieb Marek Posolda < > > > > > > > > > > > > mposolda(a)redhat.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < mailto:mposolda@redhat.com >>: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > > > Btv. what's your motivation to not use infinispan? If > > > > > > > > > > > > you > > > > > > > > > > > > afraid > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > cluster communication, you don't need to worry much > > > > > > > > > > > > about > > > > > > > > > > > > it, > > > > > > > > > > > > because > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > if you run single keycloak through standalone.xml, > > > > > > > > > > > > the > > > > > > > > > > > > infinispan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > automatically works in LOCAL mode and there is no any > > > > > > > > > > > > cluster > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > communication at all. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > > My current customer is running his apps in AWS. As > > > > > > > > > > > known, > > > > > > > > > > > multicast > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not available in cloud infrastructures. > > > > > > > > > > > Wildfly/Infinispan > > > > > > > > > > > Cluster > > > > > > > > > > > works > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > pretty well with multicast w/o having to know too much > > > > > > > > > > > about > > > > > > > > > > > JGroups > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > config. S3_PING seams to be a viable way to get a > > > > > > > > > > > cluster > > > > > > > > > > > running > > > > > > > > > > > in > > > > > > > > > > > AWS. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But additionally, my customer doesn’t have any (deep) > > > > > > > > > > > knowledge > > > > > > > > > > > about > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JBoss infrastructures and so I’m looking for a way to > > > > > > > > > > > be > > > > > > > > > > > able > > > > > > > > > > > to > > > > > > > > > > > run > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keycloak in a cluster in AWS without the need to build > > > > > > > > > > > up > > > > > > > > > > > deeper > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > knowlegde of JGroups config, for example in getting rid > > > > > > > > > > > of > > > > > > > > > > > Infinispan. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I do understand all the concerns in doing this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I still have to test S3_PING, if it works as easy as > > > > > > > > > > > multicast. > > > > > > > > > > > If > > > > > > > > > > > yes, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > we can use it, if no… I don’t know yet. But this gets > > > > > > > > > > > offtopic > > > > > > > > > > > for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Keycloak mailinglist, it’s more related to pure > > > > > > > > > > > Wildfly/Infinispan. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > > seems to me it would be much easier to get Infinispan > > > > > > > > > > working > > > > > > > > > > on > > > > > > > > > > AWS > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > than to write and maintain an entire new caching > > > > > > > > > > mechanism > > > > > > > > > > and > > > > > > > > > > hope > > > > > > > > > > we > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > don't refactor the cache SPI. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > I am sure infinispan/JGroups has possibility to run in > > > > > > > > > non-multicast > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > environment. You may just need to figure how exactly to > > > > > > > > > configure > > > > > > > > > it. > > > > > > > > > So > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I agree that this issue is more related to > > > > > > > > > Wildfly/Infinispan > > > > > > > > > itself > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > than to Keycloak. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > You may need to use jgroups protocols like TCP instead of > > > > > > > > > default > > > > > > > > > UDP > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > and maybe TCPPING (this requires to manually list all your > > > > > > > > > cluster > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > nodes. But still, it's much better option IMO than > > > > > > > > > rewriting > > > > > > > > > UserSession > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > SPI) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > Btv. if TCPPING or S3_PING is an issue, there is also > > > > > > > > AWS_PING > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 > > > > > > > > , > > > > > > > > but > > > > > > > > it's > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > not official part of jgroups. > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > Marek > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > > Marek > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > > > > > > > > > > > > > > > > > > > >

> > > > > _______________________________________________ keycloak-user > > > > > mailing > > > > > list > > > > > keycloak-user(a)lists.jboss.org > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > >

> > > > _______________________________________________ > > > > > > > > > > keycloak-user mailing list > > > > > > > > > > keycloak-user(a)lists.jboss.org > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > > > > > > > > _______________________________________________ > > > > > > keycloak-user mailing list > > > > > > keycloak-user(a)lists.jboss.org > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-user > > >