[keycloak-dev] Cross-datacenter configuration issues
Marek Posolda
mposolda at redhat.com
Fri Mar 2 07:05:09 EST 2018
Thanks William for the help!
I've tried your approach (1) and I've updated just
infinispan-cachestore-remote to 9.1.6 and left the other dependencies
not updated. It failed for me with some NoClassDefFoundErrors. This is
not so surprising as the other infinispan modules and also
wildfly-clustering integration and parser modules were still using the
old 8.2.8 versions and there seem to be some API incompatibility though,
which break things. Here is stacktrace if you want to take a look:
https://pastebin.com/hzDSX2um
In the end, I've chosen a bit different solution, which works for me.
The RemoteStore of 8.2.8 doesn't support "security" but the RemoteCache
of 8.2.8 supports that. So I've changed some integration code on
Keycloak side to read RemoteCache configuration, which is retrieved from
the RemoteStore, and use it as a configuration template. Then I've
programatically added security into it and retrieve "secured"
RemoteCacheManager to access '__script_cache' and some parts of our
code, which need to execute remote scripts. See [1].
The RemoteStore itself is not secured. So on JDG side, I need to keep 2
hotrod endpoints (one unsecured accessed by RemoteStores and second
secured accessed by __script cache). I've added setup instructions to
the JIRA: [2] . Jared, I hope this can help you to have things working.
There maybe also another approach, which Hynek pointed, that we will
somehow bypass security on JDG side at all. Picketbox has some login
modules like org.jboss.security.auth.spi.AnonLoginModule and
org.jboss.security.auth.spi.IdentityLoginModule, which might allow to
configure JAAS realm in a way that "anonymous" user will be
automatically authenticated and role ___script_manager' granted to him.
In other words, this is completely bypass security. But possibly it may
allow to have things working without require changes on Keycloak code.
Another thing is, that I didn't manage to have it working, but I didn't
try to deeply and I am not too familiar with the Wildfly security +
picketbox stuff.
So to summary, The approaches I see is:
(1) Upgrade all infinispan modules in Keycloak to use 9.1.6 (or JDG
7.2.ER4). Will require to upgrade wildfly infinispan subsystem and bunch
of other dependencies (maybe whole EAP in the end). I guess it's not an
option regarding support, testing etc. On the other hand, it will allow
to configure RemoteStore security OOTB.
(2) Create our own subclass of Infinispan RemoteStore on Keycloak side,
which will allow to configure secure RemoteCache instances. Will require
changes on Keycloak sources, and also in the Keycloak servers
configuration, which will need to use new implementation of RemoteStore,
so the configuration of the RemoteStore will be completely changed.
(3) Use the approach, which works for me and which I put in JIRA. So
have 2 hotrod endpoints on JDG side (unsecured and secured). RemoteStore
access the unsecured endpoint. Just some sources, which deals with
__script cache, access the secured hotrod endpoint. This requires to
"patch" module keycloak-model-infinispan on RHSSO side and also require
some smaller changes in the configuration (some new configuration
properties are added on RHSSO side to allow configuration of hotrod
endpoint security, username ,password etc).
(4) Bypass JDG security entirely with some picketbox "anonymous access"
JAAS login modules. I didn't manage to have it working and it completely
bypass security. On the other hand, it seems to be the only solution,
which won't require changes on Keycloak/RHSSO side. But we don't know
yet if it works...
For now, my vote is to go with (3) and it's the only working. Any more
options?
[1]
https://github.com/mposolda/keycloak/blob/jdg-security/model/infinispan/src/main/java/org/keycloak/connections/infinispan/RemoteCacheProvider.java#L110-L149
[2]
https://issues.jboss.org/browse/KEYCLOAK-6783?focusedCommentId=13540721&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13540721
Marek
On 01/03/18 15:42, William Burns wrote:
> So I was helping out Jared when found this issue.
>
> This is caused by a combination of issues due to varying versions (Infinispaan 8.2.8 and JDG 7.1)
>
> 1. JDG 7.1 only allows accessing internal caches remotely over loopback or when authorization is enabled. This came about in Infinispan 9.0 in [a]
> 2. Infinispan 8.2.6 does not allow for remote cache stores to use authentication. This was added in Infinispan 9.0.(2|3) and 9.1 in [b]
>
> Unfortunately with the combination of the 2 there is now way for an ISPN pre 9.0.2 from accessing a remote server 9.0 internal caches not on loopback, such as the script cache.
>
> There are 2 ways that I was able to come up with (thanks Tristan for the second)
>
> 1. Replace the infinispan-cachestore-remote.jar in the ISPN 8.2.6 install to have a newer version such as 9.1.6.Final at [c]. This should allow the 8.2.8 instance to use authorization on the remote store. I have not tested this, but everything should be contained in the jar (including parser).
> 2. Use JDG 7.2.ER4 or newer in the keycloak EAP instances as a module. This is the first version that allows for the user to deploy their own version of JDG as a module accessible over JNDI.
>
> - Will
>
> [a] https://issues.jboss.org/browse/ISPN-6457
> [b] https://issues.jboss.org/browse/ISPN-7868
> [c] https://mvnrepository.com/artifact/org.infinispan/infinispan-cachestore-remote
>
> ----- Original Message -----
>> From: "Hynek Mlnarik" <hmlnarik at redhat.com>
>> To: "Jared Blashka" <jblashka at redhat.com>
>> Cc: "Marek Posolda" <mposolda at redhat.com>, "Boleslaw Dawidowicz" <bdawidow at redhat.com>, "keycloak-dev"
>> <keycloak-dev at lists.jboss.org>, "William Burns" <wburns at redhat.com>, "iam-list" <iam-list at redhat.com>
>> Sent: Thursday, March 1, 2018 3:33:17 AM
>> Subject: Re: [keycloak-dev] Cross-datacenter configuration issues
>>
>> I can only agree that it seems to be a difference between Infinispan server
>> and JDG, since we did test it in Amazon where each instance of Keycloak and
>> Infinispan was installed on separate VM [1]. Whether this difference might
>> be indeed there should be confirmed by someone from JDG team. William,
>> could you please comment here?
>>
>> [1]
>> http://blog.keycloak.org/2018/01/keycloak-cross-data-center-setup-in-aws.html
>>
>> On Thu, Mar 1, 2018 at 9:25 AM, Marek Posolda <mposolda at redhat.com> wrote:
>>
>>> I've just simulated the issue and created
>>> https://issues.jboss.org/browse/KEYCLOAK-6783 . I am looking at it.
>>>
>>> What works and what we tested is:
>>>
>>> * Setup with infinispan-server-8.2.8 on "local" network (infinispan
>>> server bind on loopback address like "localhost" . Different
>>> infinispan servers running on the same laptop, but on various port
>>> offsets)
>>>
>>> * Setup with JDG server 7.1.0 on "local" network (JDG server bound on
>>> loopback address like "localhost" . Different JDG servers running on
>>> the same laptop, but on various port offsets)
>>>
>>> * Setup with infinispan-server-8.2.8 on "real" network (testing with
>>> infinispan hosts bound to real host with IP addresses like 192.168.0.1
>>> )
>>>
>>> We didn't test the combination with JDG server bind on "real" addresses
>>> and this is the only one where the issue happens
>>>
>>> It seems JDG 7.1.0 has some additional security when compared with the
>>> community infinispan-server 8.2.8 .
>>>
>>> The easiest workaround for you might be to test with community
>>> infinispan-server 8.2.8 instead of JDG 7.1.0 . Server can be downloaded
>>> from this address:
>>> http://downloads.jboss.org/infinispan/8.2.8.Final/
>>> infinispan-server-8.2.8.Final-bin.zip
>>> .
>>>
>>> I hope to update you later today once I have some more info. Thanks for
>>> the report and all the details you mentioned.
>>>
>>> Marek
>>>
>>>
>>> On 28/02/18 21:36, Jared Blashka wrote:
>>>> Hey all,
>>>>
>>>> I'm working on testing out the cross-datacenter replication
>>>> configuration in our development environment and I'm running into some
>>>> issues.
>>>>
>>>> I stood up some JDG 7.1 instances and some RH-SSO 7.2 instances all
>>>> running on my localhost all with different port offsets, followed the
>>>> instructions[1], and everything seemed to work well enough.
>>>>
>>>> Once I got beyond that and tried running RH-SSO and JDG on separate
>>>> servers I started running into issues[2] during RH-SSO startup. Looks
>>>> like RH-SSO is unable to connect to the remote ___script_cache but
>>>> that cache isn't mentioned anywhere in the RH-SSO documentation. The
>>>> error message (and online searching) indicates that this cache only
>>>> allows remote connections if authorization is enabled. I didn't see
>>>> any mention of configuration related to authentication or security for
>>>> the remote caches in the documentation either.
>>>>
>>>> At this point we roped in a JDG expert (cc'ed here) and found some
>>>> additional Infinispan documentation[3] on how to add authentication to
>>>> the *remote* caches within the JDG configuration but nothing much in
>>>> the way of adding authentication to the client cache configuration
>>>> inside RH-SSO that didn't involve programmatic changes. After some
>>>> additional searching we found some info[4] detailing how to add
>>>> security configurations to a remote-cache configuration in Infinispan
>>>> *9.1* but EAP 7.1 is only running Infinispan *8.2* which doesn't have
>>>> these changes.
>>>>
>>>> How did you get this working?
>>>>
>>>> Jared Blashka - Identity & Access Management
>>>>
>>>>
>>>> [1]
>>>> https://access.redhat.com/documentation/en-us/red_hat_
>>> single_sign-on/7.2/pdf/server_installation_and_
>>> configuration_guide/Red_Hat_Single_Sign-On-7.2-Server_Installation_and_
>>> Configuration_Guide-en-US.pdf#__WKANCHOR_1e
>>>> [2] http://pastebin.test.redhat.com/559674
>>>> [3]
>>>> http://infinispan.org/docs/stable/server_guide/server_
>>> guide.html#general_concepts
>>>> [4]
>>>> https://docs.jboss.org/infinispan/9.1/configdocs/
>>> infinispan-cachestore-remote-config-9.1.html
>>>
>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>
>>
>> --
>>
>> --Hynek
>>
More information about the keycloak-dev
mailing list