[keycloak-user] Replace use of Infinispan with User Sessions SPI ?

[Stian Thorgersen]

If anyone (Scott?) wants to contribute how to configure KC clustering on
EC2 to our documentation that'd be great :)

On 16 December 2015 at 23:11, Scott Rossillo <srossillo at smartling.com>
wrote:

>
> I actually set the jgroups.bind_addr to global. I need the EC2 instance's
> address for jgroups.external.addr, see:
>
>
> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com
>
> [image: Powered by Sigstr] <http://www.sigstr.com/>
>
> On Dec 16, 2015, at 4:21 PM, Alan Field <afield at redhat.com> wrote:
>
> Almost...
>
> I guess if the EC2 instance IP works for the bind address, why do you need
> to set external_addr?
>
> Thanks for bearing with me on this! :-)
>
> Alan
>
> ------------------------------
>
> *From: *"Scott Rossillo" <srossillo at smartling.com>
> *To: *"Alan Field" <afield at redhat.com>
> *Cc: *"Niko Köbler" <niko at n-k.de>, "keycloak-user" <
> keycloak-user at lists.jboss.org>
> *Sent: *Wednesday, December 16, 2015 4:17:29 PM
> *Subject: *Re: [keycloak-user] Replace use of Infinispan with User
> Sessions SPI ?
>
> Ah, sorry, my originally contrived example wasn’t using Amazon but just my
> local Docker machine IP.
>
> In the case of my ECS tests, 172.16.0.0/16 is the Docker network’s IP,
> which is local to the machine / EC2 instance. Using ECS, my VPC has an IP
> range of 172.31.0.0/16, so the bind_addr has to be on this network. On my
> small cluster, that’s either 172.31.44.109 or 172.31.45.191.
>
> Does that clear it up?
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com
>
> [image: Powered by Sigstr] <http://www.sigstr.com/>
>
> On Dec 16, 2015, at 4:11 PM, Alan Field <afield at redhat.com> wrote:
>
> Hey Scott,
>
> Thanks, I think you answered all of my questions, but I'm confused by
> something you said in your first email:
>
> "
> The 172.16 network is not routable between hosts (by design). Docker does
> port forwarding for ports we wish to expose to this works fine for
> HTTP/HTTPS but not the cluster traffic.
>
> So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
> advertises 172.16.0.8. The two cannot talk to each other by default.
> "
>
> My understanding is that the 172.16 addresses are the Amazon EC2
> instance’s internal IP, so I'm confused why this didn't work for you
> before. Is the difference that you were setting jgroups.bind_addr to this
> address and now you are setting it to global and setting external_addr to
> the instance’s internal IP? Just trying to understand what the problem was
> and how you fixed it!
>
> Thanks again,
> Alan
>
>
> ------------------------------
>
> *From: *"Scott Rossillo" <srossillo at smartling.com>
> *To: *"Alan Field" <afield at redhat.com>
> *Cc: *"Niko Köbler" <niko at n-k.de>, "keycloak-user" <
> keycloak-user at lists.jboss.org>
> *Sent: *Wednesday, December 16, 2015 3:45:40 PM
> *Subject: *Re: [keycloak-user] Replace use of Infinispan with User
> Sessions SPI ?
>
> Hi Alan,
>
> > It is possible to use the TUNNEL with multiple gossip routers to avoid
> this, but I understand not wanting to have to setup and maintain the extra
> gossip router processes.
>
> True, it’s mainly about maintaining extra components.
>
> > Which IP address from your example is retrieved with this command:
> > EXTERNAL_HOST_IP=$(curl
> http://169.254.169.254/latest/meta-data/local-ipv4)”
>
> I get the Amazon EC2 instance’s internal IP. This is what I want. There’s
> another endpoint for public but I don’t want to use it. What’s good about
> this is when called from inside a Docker container, I manage to get the
> actual internal IP for the EC2 instance.
>
> > How are you setting the  JGROUPS_INITIAL_HOSTS environment variable?
>
> Since this was a test with just 2 known hosts, I injected them as a Docker
> environment variable with two fixed IPs. Once we switch to JDBC_PING, this
> will be removed.
>
> > For my curiosity, can you tell me more about why you don't want to use
> S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
> should work fine.
>
> S3_PING, like Gossip Router adds an external dependency on another
> service. S3 has had consistency issues 3 times in 2015 (at least in US
> East). I don’t want to rely another component when I already need the
> database to be up. Less components, less chance of failure. Also, there are
> ton of variables to set with S3 and it requires preliminary work. I want
> something that scales well from dev to QA to prod. JDBC_PING has a
> datasource_jndi_name property. I can just reuse the data source I set up
> for Keycloak.
>
> I hope I got all your questions.
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com
>
> [image: Powered by Sigstr] <http://www.sigstr.com/>
>
> On Dec 16, 2015, at 3:33 PM, Alan Field <afield at redhat.com> wrote:
>
> Hey Scott,
>
> Thanks for following up and showing me your code. I have some questions
> inline for you:
>
> ------------------------------
>
> *From: *"Scott Rossillo" <srossillo at smartling.com>
> *To: *"Alan Field" <afield at redhat.com>
> *Cc: *"Niko Köbler" <niko at n-k.de>, "keycloak-user" <
> keycloak-user at lists.jboss.org>
> *Sent: *Wednesday, December 16, 2015 2:19:27 PM
> *Subject: *Re: [keycloak-user] Replace use of Infinispan with User
> Sessions        SPI ?
>
> Hi Alan,
>
> Thanks for the informative email. The steps you outlined are similar to
> what I’ve tested with ECS.  The gossip router is definitely a no-go for
> production since it’s a single point of failure.
>
> It is possible to use the TUNNEL with multiple gossip routers to avoid
> this, but I understand not wanting to have to setup and maintain the extra
> gossip router processes.
>
>
> I am testing this down at the JGroups level right now and got it working
> with ECS. There were two issues. On TCP you have to specify
> the external_addr to match the EC2 host otherwise the nodes won’t form a
> cluster. Secondly, FD_SOCK attempts to connect back on a random port. With
> Docker instances, this fails. Using a known client_bind_port works well.
>
> Which IP address from your example is retrieved with this command:
>
> EXTERNAL_HOST_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4
> )"
>
> Is it the 172.16.0.4 address or the 10.10.0.100 address? When I use this
> command in EC2, I get the internal IP address for the instance, but not the
> public IP address. In your example, that would be the 172.16.0.4 address.
> Also which address is used for the bind_addr when you use
> -Djgroups.bind_addr=global?
>
>
> Here’s the code I’m testing with:
> https://github.com/foo4u/aws-infinispan-poc
>
> Most interesting are probably:
>
>
> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/entrypoint.sh
>
> How are you setting the  JGROUPS_INITIAL_HOSTS environment variable?
>
>
> https://github.com/foo4u/aws-infinispan-poc/blob/master/ecs-jgroups-poc/src/main/resources/tcp.xml
>
> With this set up the nodes on different machines communicate without
> issue. I still have to add in something other than TCP_PING, but that
> wasn’t the main issue. Will use JDBC_PING most likely. Not a fan of S3 for
> coordination. Plus I already need an RDBMS for Keycloak.
>
> For my curiosity, can you tell me more about why you don't want to use
> S3_PING? Is it the cost or something else? Just wondering and JDBC_PING
> should work fine.
>
> Thanks,
> Alan
>
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com
>
> [image: Latest News + Events]
> <https://app.sigstr.com/uc/55e5d41c6533390d03580000>
> [image: Powered by Sigstr] <http://www.sigstr.com/>
>
> On Dec 15, 2015, at 2:13 PM, Alan Field <afield at redhat.com> wrote:
>
> Just to be clear, I have successfully tested Infinispan library and server
> mode clusters on EC2 using S3_PING, TCP, and the internal EC2 IP addresses.
> None of the cloud providers support multicast. The Docker case is a little
> different though, because of the issues with getting access to the IP
> address.
>
> Thanks,
> Alan
>
> ------------------------------
>
> *From: *"Niko Köbler" <niko at n-k.de>
> *To: *"Paul Blair" <pblair at clearme.com>
> *Cc: *"keycloak-user" <keycloak-user at lists.jboss.org>
> *Sent: *Tuesday, December 15, 2015 1:53:18 PM
> *Subject: *Re: [keycloak-user] Replace use of Infinispan with User
> Sessions        SPI ?
>
> We will go for the first run with EC2 and S3_PING, but w/o Docker.
> If we/you/whoever will find a proper solution (possibly on the jgroups
> mailinglist), we will test this.
>
> Seams that everybody is aware of the Docker/Cloud/Multicast issues, but
> no-one has a proper solution, only workarounds. :(
>
>
>
> Am 15.12.2015 um 15:47 schrieb Paul Blair <pblair at clearme.com>:
>
> I've also been working on setting up clustered Keycloak on Docker
> containers in EC2 and would be interested in any potential solutions for
> this configuration.
>
> Alternatively I've set up on EC2 without Docker with S3_PING. I'd be
> interested in hearing about the issues with this configuration.
>
> From: Scott Rossillo <srossillo at smartling.com>
> Date: Mon, 14 Dec 2015 18:31:30 -0500
> To: Marek Posolda <mposolda at redhat.com>, <afield at redhat.com>
> Cc: keycloak-user <keycloak-user at lists.jboss.org>
> Subject: Re: [keycloak-user] Replace use of Infinispan with User Sessions
> SPI ?
>
> There are two issues:
>
> 1. Infinispan relies on JGroups, which is difficult to configure correctly
> with the various ping techniques that aren’t UDP multicast. I can elaborate
> on each one that we tested but it’s just generally complex to get right.
> That’s not to say it’s impossible or the biggest reason this is complicated
> on ECS or _insert container service here_, see #2 for that.
>
> 2. It is difficult to do discovery correctly with JGroups and Docker.
> Non-privileged Docker instances - the default and recommend type - do not
> implicitly know their host’s IP. This causes IP mismatches between what
> JGroups thinks the machine’s IP is and what it actually is when connecting
> to hosts on different machines.  This is the main issue and it’s not the
> fault of JGroups per se, but there’s no simple work around.
>
> Take for example a simple 2 node cluster:
>
> Node 1 comes up on the docker0 interface of host A with the IP address
> 172.16.0.4. The host A IP is 10.10.0.100.
> Node 2 comes up on the docker0 interface of host B with the IP address
> 172.16.0.8. The host B IP is 10.10.0.108.
>
> The 172.16 network is not routable between hosts (by design). Docker does
> port forwarding for ports we wish to expose to this works fine for
> HTTP/HTTPS but not the cluster traffic.
>
> So Node 1 will advertise itself as having IP 172.16.0.4 while Node 2
> advertises 172.16.0.8. The two cannot talk to each other by default.
> However, using the hard coded IPs and TCP PING, we can set external_addr on
> Node 1 to 10.10.0.100 and external_addr on Node 2 to 10.10.0.108 and set
> initial_hosts to 10.10.0.100, 10.10.0.108. This will cause the nodes to
> discover each other. However, they will not form a cluster. The nodes will
> reject the handshake thinking they’re not actually 10.10.0.100 or
> 10.10.0.108 respectively.
>
> I’d like to discuss further and I can share where we’ve gotten so far with
> workarounds to this but it may be better to get into the weeds on another
> list.
>
> Let me know what you think.
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com
>
> [image: Powered by Sigstr] <http://www.sigstr.com/>
>
> On Dec 14, 2015, at 5:32 PM, Marek Posolda <mposolda at redhat.com> wrote:
>
> CCing Alan Field from RH Infinispan team and forwarding his question:
>
> I'd like to know which configuration files you are using and why is is
> harder to use with Amazon’s Docker service (ECS) or Beanstalk. I'd also be
> interested in how big a cluster you are using in AWS.
>
>
>
>
> On 14/12/15 22:24, Scott Rossillo wrote:
>
> AWS was why we didn’t use Infinispan to begin with.  That and it’s even
> more complicated when you deploy using Amazon’s Docker service (ECS) or
> Beanstalk.
>
> It’s too bad Infinispan  / JGroups are beasts when the out of the box
> configuration can’t be used. I’m planning to document this as we fix but
> I’d avoid S3_PING and use JDBC_PING. You already need JDBC for the Keycloak
> DB, unless you’re using Mongo and it’s easier to test locally.
>
> TCPPING will bite you on AWS if Amazon decides to replace one of your
> instances (which it does occasionally w/ECS or Beanstalk).
>
> Best,
> Scott
>
> Scott Rossillo
> Smartling | Senior Software Engineer
> srossillo at smartling.com
>
> [image: Powered by Sigstr] <http://www.sigstr.com/>
>
> On Dec 14, 2015, at 10:59 AM, Marek Posolda <mposolda at redhat.com> wrote:
>
> On 14/12/15 16:55, Marek Posolda wrote:
>
> On 14/12/15 15:58, Bill Burke wrote:
>
> On 12/14/2015 5:01 AM, Niko Köbler wrote:
>
> Hi Marek,
>
> Am 14.12.2015 um 08:50 schrieb Marek Posolda <mposolda at redhat.com
> <mailto:mposolda at redhat.com <mposolda at redhat.com>>>:
>
> Btv. what's your motivation to not use infinispan? If you afraid of
> cluster communication, you don't need to worry much about it, because
> if you run single keycloak through standalone.xml, the infinispan
> automatically works in LOCAL mode and there is no any cluster
> communication at all.
>
> My current customer is running his apps in AWS. As known, multicast is
> not available in cloud infrastructures. Wildfly/Infinispan Cluster works
> pretty well with multicast w/o having to know too much about JGroups
> config. S3_PING seams to be a viable way to get a cluster running in AWS.
> But additionally, my customer doesn’t have any (deep) knowledge about
> JBoss infrastructures and so I’m looking for a way to be able to run
> Keycloak in a cluster in AWS without the need to build up deeper
> knowlegde of JGroups config, for example in getting rid of Infinispan.
> But I do understand all the concerns in doing this.
> I still have to test S3_PING, if it works as easy as multicast. If yes,
> we can use it, if no… I don’t know yet. But this gets offtopic for
> Keycloak mailinglist, it’s more related to pure Wildfly/Infinispan.
>
> seems to me it would be much easier to get Infinispan working on AWS
> than to write and maintain an entire new caching mechanism and hope we
> don't refactor the cache SPI.
>
>
> +1
>
> I am sure infinispan/JGroups has possibility to run in non-multicast
> environment. You may just need to figure how exactly to configure it. So
> I agree that this issue is more related to Wildfly/Infinispan itself
> than to Keycloak.
>
> You may need to use jgroups protocols like TCP instead of default UDP
> and maybe TCPPING (this requires to manually list all your cluster
> nodes. But still, it's much better option IMO than rewriting UserSession
> SPI)
>
> Btv. if TCPPING or S3_PING is an issue, there is also AWS_PING
> http://www.jgroups.org/manual-3.x/html/protlist.html#d0e5100 , but it's
> not official part of jgroups.
>
> Marek
>
>
> Marek
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>
>
> _______________________________________________ keycloak-user mailing list
>  keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>
>
>
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20151218/67be3ffb/attachment-0001.html