[keycloak-dev] Cross-datacenter configuration issues

Fri Mar 2 09:24:30 EST 2018

----- Original Message -----
> From: "Marek Posolda" <mposolda at redhat.com>
> To: "William Burns" <wburns at redhat.com>, "Hynek Mlnarik" <hmlnarik at redhat.com>
> Cc: "Jared Blashka" <jblashka at redhat.com>, "Boleslaw Dawidowicz" <bdawidow at redhat.com>, "keycloak-dev"
> <keycloak-dev at lists.jboss.org>, "iam-list" <iam-list at redhat.com>, "Tristan Tarrant" <ttarrant at redhat.com>, "Pedro
> Zapata Fernandez" <pzapataf at redhat.com>
> Sent: Friday, March 2, 2018 7:05:09 AM
> Subject: Re: [keycloak-dev] Cross-datacenter configuration issues
> 
> Thanks William for the help!
> 
> I've tried your approach (1) and I've updated just
> infinispan-cachestore-remote to 9.1.6 and left the other dependencies
> not updated. It failed for me with some NoClassDefFoundErrors. This is
> not so surprising as the other infinispan modules and also
> wildfly-clustering integration and parser modules were still using the
> old 8.2.8 versions and there seem to be some API incompatibility though,
> which break things. Here is stacktrace if you want to take a look:
> https://pastebin.com/hzDSX2um

Yeah actually I was talking with Jared about this yesterday. Unfortunately it doesn't work. I was thinking of the embedded mode which allows you to use custom parsers. So this isn't an option.

> 
> In the end, I've chosen a bit different solution, which works for me.
> The RemoteStore of 8.2.8 doesn't support "security" but the RemoteCache
> of 8.2.8 supports that. So I've changed some integration code on
> Keycloak side to read RemoteCache configuration, which is retrieved from
> the RemoteStore, and use it as a configuration template. Then I've
> programatically added security into it and retrieve "secured"
> RemoteCacheManager to access '__script_cache' and some parts of our
> code, which need to execute remote scripts. See [1].
> 
> The RemoteStore itself is not secured. So on JDG side, I need to keep 2
> hotrod endpoints (one unsecured accessed by RemoteStores and second
> secured accessed by __script cache). I've added setup instructions to
> the JIRA: [2] . Jared, I hope this can help you to have things working.
> 
> There maybe also another approach, which Hynek pointed, that we will
> somehow bypass security on JDG side at all. Picketbox has some login
> modules like org.jboss.security.auth.spi.AnonLoginModule and
> org.jboss.security.auth.spi.IdentityLoginModule, which might allow to
> configure JAAS realm in a way that "anonymous" user will be
> automatically authenticated and role ___script_manager' granted to him.
> In other words, this is completely bypass security. But possibly it may
> allow to have things working without require changes on Keycloak code.
> Another thing is, that I didn't manage to have it working, but I didn't
> try to deeply and I am not too familiar with the Wildfly security +
> picketbox stuff.
> 
> So to summary, The approaches I see is:
> 
> (1) Upgrade all infinispan modules in Keycloak to use 9.1.6 (or JDG
> 7.2.ER4). Will require to upgrade wildfly infinispan subsystem and bunch
> of other dependencies (maybe whole EAP in the end). I guess it's not an
> option regarding support, testing etc. On the other hand, it will allow
> to configure RemoteStore security OOTB.

You and others are more familiar with how modules work. TBH I am not that familiar, but this sounds like a no go to me as well.

> 
> (2) Create our own subclass of Infinispan RemoteStore on Keycloak side,
> which will allow to configure secure RemoteCache instances. Will require
> changes on Keycloak sources, and also in the Keycloak servers
> configuration, which will need to use new implementation of RemoteStore,
> so the configuration of the RemoteStore will be completely changed.
> 
> (3) Use the approach, which works for me and which I put in JIRA. So
> have 2 hotrod endpoints on JDG side (unsecured and secured). RemoteStore
> access the unsecured endpoint. Just some sources, which deals with
> __script cache, access the secured hotrod endpoint. This requires to
> "patch" module keycloak-model-infinispan on RHSSO side and also require
> some smaller changes in the configuration (some new configuration
> properties are added on RHSSO side to allow configuration of hotrod
> endpoint security, username ,password etc).
> 
> (4) Bypass JDG security entirely with some picketbox "anonymous access"
> JAAS login modules. I didn't manage to have it working and it completely
> bypass security. On the other hand, it seems to be the only solution,
> which won't require changes on Keycloak/RHSSO side. But we don't know
> yet if it works...

Actually there is another way to bypass security that Tristan brought up that we were was discussing. For JDG 7.2.CR1 we can relax access to protected caches, such as scriptcache, so that there is no global authorization check (removing a single if block).  This way the cache just uses the normal authorization checks if it is even enabled for that cache. Unfortunately CR1 is not going to be available for another 3 weeks though. In the mean time Jared and I discussed just testing with community server version 8.2.8 for the standalone install until we can get JDG 7.2.CR1 installed there. He may even want to test with something a little newer.

Unfortunately that means that this version of keycloak would only work with JDG 7.2.0.CR1 or ISPN 9.0.3 [1] and newer. Either way I will be creating a PR for this on the JDG 7.2 branch today.

[1] https://issues.jboss.org/browse/ISPN-7814

> 
> For now, my vote is to go with (3) and it's the only working. Any more
> options?
> 
> [1]
> https://github.com/mposolda/keycloak/blob/jdg-security/model/infinispan/src/main/java/org/keycloak/connections/infinispan/RemoteCacheProvider.java#L110-L149
> [2]
> https://issues.jboss.org/browse/KEYCLOAK-6783?focusedCommentId=13540721&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13540721
> 
> Marek
> 
> 
> On 01/03/18 15:42, William Burns wrote:
> > So I was helping out Jared when found this issue.
> >
> > This is caused by a combination of issues due to varying versions
> > (Infinispaan 8.2.8 and JDG 7.1)
> >
> > 1. JDG 7.1 only allows accessing internal caches remotely over loopback or
> > when authorization is enabled. This came about in Infinispan 9.0 in [a]
> > 2. Infinispan 8.2.6 does not allow for remote cache stores to use
> > authentication. This was added in Infinispan 9.0.(2|3) and 9.1 in [b]
> >
> > Unfortunately with the combination of the 2 there is now way for an ISPN
> > pre 9.0.2 from accessing a remote server 9.0 internal caches not on
> > loopback, such as the script cache.
> >
> > There are 2 ways that I was able to come up with (thanks Tristan for the
> > second)
> >
> > 1. Replace the infinispan-cachestore-remote.jar in the ISPN 8.2.6 install
> > to have a newer version such as 9.1.6.Final at [c]. This should allow the
> > 8.2.8 instance to use authorization on the remote store. I have not tested
> > this, but everything should be contained in the jar (including parser).
> > 2. Use JDG 7.2.ER4 or newer in the keycloak EAP instances as a module. This
> > is the first version that allows for the user to deploy their own version
> > of JDG as a module accessible over JNDI.
> >
> > - Will
> >
> > [a] https://issues.jboss.org/browse/ISPN-6457
> > [b] https://issues.jboss.org/browse/ISPN-7868
> > [c]
> > https://mvnrepository.com/artifact/org.infinispan/infinispan-cachestore-remote
> >
> > ----- Original Message -----
> >> From: "Hynek Mlnarik" <hmlnarik at redhat.com>
> >> To: "Jared Blashka" <jblashka at redhat.com>
> >> Cc: "Marek Posolda" <mposolda at redhat.com>, "Boleslaw Dawidowicz"
> >> <bdawidow at redhat.com>, "keycloak-dev"
> >> <keycloak-dev at lists.jboss.org>, "William Burns" <wburns at redhat.com>,
> >> "iam-list" <iam-list at redhat.com>
> >> Sent: Thursday, March 1, 2018 3:33:17 AM
> >> Subject: Re: [keycloak-dev] Cross-datacenter configuration issues
> >>
> >> I can only agree that it seems to be a difference between Infinispan
> >> server
> >> and JDG, since we did test it in Amazon where each instance of Keycloak
> >> and
> >> Infinispan was installed on separate VM [1]. Whether this difference might
> >> be indeed there should be confirmed by someone from JDG team. William,
> >> could you please comment here?
> >>
> >> [1]
> >> http://blog.keycloak.org/2018/01/keycloak-cross-data-center-setup-in-aws.html
> >>
> >> On Thu, Mar 1, 2018 at 9:25 AM, Marek Posolda <mposolda at redhat.com> wrote:
> >>
> >>> I've just simulated the issue and created
> >>> https://issues.jboss.org/browse/KEYCLOAK-6783 . I am looking at it.
> >>>
> >>> What works and what we tested is:
> >>>
> >>>    * Setup with infinispan-server-8.2.8 on "local" network (infinispan
> >>>      server bind on loopback address like "localhost" . Different
> >>>      infinispan servers running on the same laptop, but on various port
> >>>      offsets)
> >>>
> >>>    * Setup with JDG server 7.1.0 on "local" network (JDG server bound on
> >>>      loopback address like "localhost" . Different JDG servers running on
> >>>      the same laptop, but on various port offsets)
> >>>
> >>>    * Setup with infinispan-server-8.2.8 on "real" network (testing with
> >>>      infinispan hosts bound to real host with IP addresses like
> >>>      192.168.0.1
> >>> )
> >>>
> >>> We didn't test the combination with JDG server bind on "real" addresses
> >>> and this is the only one where the issue happens
> >>>
> >>> It seems JDG 7.1.0 has some additional security when compared with the
> >>> community infinispan-server 8.2.8 .
> >>>
> >>> The easiest workaround for you might be to test with community
> >>> infinispan-server 8.2.8 instead of JDG 7.1.0 . Server can be downloaded
> >>> from this address:
> >>> http://downloads.jboss.org/infinispan/8.2.8.Final/
> >>> infinispan-server-8.2.8.Final-bin.zip
> >>> .
> >>>
> >>> I hope to update you later today once I have some more info. Thanks for
> >>> the report and all the details you mentioned.
> >>>
> >>> Marek
> >>>
> >>>
> >>> On 28/02/18 21:36, Jared Blashka wrote:
> >>>> Hey all,
> >>>>
> >>>> I'm working on testing out the cross-datacenter replication
> >>>> configuration in our development environment and I'm running into some
> >>>> issues.
> >>>>
> >>>> I stood up some JDG 7.1 instances and some RH-SSO 7.2 instances all
> >>>> running on my localhost all with different port offsets, followed the
> >>>> instructions[1], and everything seemed to work well enough.
> >>>>
> >>>> Once I got beyond that and tried running RH-SSO and JDG on separate
> >>>> servers I started running into issues[2] during RH-SSO startup. Looks
> >>>> like RH-SSO is unable to connect to the remote ___script_cache but
> >>>> that cache isn't mentioned anywhere in the RH-SSO documentation. The
> >>>> error message (and online searching) indicates that this cache only
> >>>> allows remote connections if authorization is enabled. I didn't see
> >>>> any mention of configuration related to authentication or security for
> >>>> the remote caches in the documentation either.
> >>>>
> >>>> At this point we roped in a JDG expert (cc'ed here) and found some
> >>>> additional Infinispan documentation[3] on how to add authentication to
> >>>> the *remote* caches within the JDG configuration but nothing much in
> >>>> the way of adding authentication to the client cache configuration
> >>>> inside RH-SSO that didn't involve programmatic changes. After some
> >>>> additional searching we found some info[4] detailing how to add
> >>>> security configurations to a remote-cache configuration in Infinispan
> >>>> *9.1* but EAP 7.1 is only running Infinispan *8.2* which doesn't have
> >>>> these changes.
> >>>>
> >>>> How did you get this working?
> >>>>
> >>>> Jared Blashka - Identity & Access Management
> >>>>
> >>>>
> >>>> [1]
> >>>> https://access.redhat.com/documentation/en-us/red_hat_
> >>> single_sign-on/7.2/pdf/server_installation_and_
> >>> configuration_guide/Red_Hat_Single_Sign-On-7.2-Server_Installation_and_
> >>> Configuration_Guide-en-US.pdf#__WKANCHOR_1e
> >>>> [2] http://pastebin.test.redhat.com/559674
> >>>> [3]
> >>>> http://infinispan.org/docs/stable/server_guide/server_
> >>> guide.html#general_concepts
> >>>> [4]
> >>>> https://docs.jboss.org/infinispan/9.1/configdocs/
> >>> infinispan-cachestore-remote-config-9.1.html
> >>>
> >>>
> >>> _______________________________________________
> >>> keycloak-dev mailing list
> >>> keycloak-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> >>>
> >>
> >>
> >> --
> >>
> >> --Hynek
> >>
> 
> 