1:05 p.m.
Thanks William for the help!
I've tried your approach (1) and I've updated just
infinispan-cachestore-remote to 9.1.6 and left the other dependencies
not updated. It failed for me with some NoClassDefFoundErrors. This is
not so surprising as the other infinispan modules and also
wildfly-clustering integration and parser modules were still using the
old 8.2.8 versions and there seem to be some API incompatibility though,
which break things. Here is stacktrace if you want to take a look:
https://pastebin.com/hzDSX2um
In the end, I've chosen a bit different solution, which works for me.
The RemoteStore of 8.2.8 doesn't support "security" but the RemoteCache
of 8.2.8 supports that. So I've changed some integration code on
Keycloak side to read RemoteCache configuration, which is retrieved from
the RemoteStore, and use it as a configuration template. Then I've
programatically added security into it and retrieve "secured"
RemoteCacheManager to access '__script_cache' and some parts of our
code, which need to execute remote scripts. See [1].
The RemoteStore itself is not secured. So on JDG side, I need to keep 2
hotrod endpoints (one unsecured accessed by RemoteStores and second
secured accessed by __script cache). I've added setup instructions to
the JIRA: [2] . Jared, I hope this can help you to have things working.
There maybe also another approach, which Hynek pointed, that we will
somehow bypass security on JDG side at all. Picketbox has some login
modules like org.jboss.security.auth.spi.AnonLoginModule and
org.jboss.security.auth.spi.IdentityLoginModule, which might allow to
configure JAAS realm in a way that "anonymous" user will be
automatically authenticated and role ___script_manager' granted to him.
In other words, this is completely bypass security. But possibly it may
allow to have things working without require changes on Keycloak code.
Another thing is, that I didn't manage to have it working, but I didn't
try to deeply and I am not too familiar with the Wildfly security +
picketbox stuff.
So to summary, The approaches I see is:
(1) Upgrade all infinispan modules in Keycloak to use 9.1.6 (or JDG
7.2.ER4). Will require to upgrade wildfly infinispan subsystem and bunch
of other dependencies (maybe whole EAP in the end). I guess it's not an
option regarding support, testing etc. On the other hand, it will allow
to configure RemoteStore security OOTB.
(2) Create our own subclass of Infinispan RemoteStore on Keycloak side,
which will allow to configure secure RemoteCache instances. Will require
changes on Keycloak sources, and also in the Keycloak servers
configuration, which will need to use new implementation of RemoteStore,
so the configuration of the RemoteStore will be completely changed.
(3) Use the approach, which works for me and which I put in JIRA. So
have 2 hotrod endpoints on JDG side (unsecured and secured). RemoteStore
access the unsecured endpoint. Just some sources, which deals with
__script cache, access the secured hotrod endpoint. This requires to
"patch" module keycloak-model-infinispan on RHSSO side and also require
some smaller changes in the configuration (some new configuration
properties are added on RHSSO side to allow configuration of hotrod
endpoint security, username ,password etc).
(4) Bypass JDG security entirely with some picketbox "anonymous access"
JAAS login modules. I didn't manage to have it working and it completely
bypass security. On the other hand, it seems to be the only solution,
which won't require changes on Keycloak/RHSSO side. But we don't know
yet if it works...
For now, my vote is to go with (3) and it's the only working. Any more
options?
[1]
https://github.com/mposolda/keycloak/blob/jdg-security/model/infinispan/s...
[2]
https://issues.jboss.org/browse/KEYCLOAK-6783?focusedCommentId=13540721&a...
Marek
On 01/03/18 15:42, William Burns wrote:
So I was helping out Jared when found this issue.
This is caused by a combination of issues due to varying versions (Infinispaan 8.2.8 and
JDG 7.1)
1. JDG 7.1 only allows accessing internal caches remotely over loopback or when
authorization is enabled. This came about in Infinispan 9.0 in [a]
2. Infinispan 8.2.6 does not allow for remote cache stores to use authentication. This
was added in Infinispan 9.0.(2|3) and 9.1 in [b]
Unfortunately with the combination of the 2 there is now way for an ISPN pre 9.0.2 from
accessing a remote server 9.0 internal caches not on loopback, such as the script cache.
There are 2 ways that I was able to come up with (thanks Tristan for the second)
1. Replace the infinispan-cachestore-remote.jar in the ISPN 8.2.6 install to have a newer
version such as 9.1.6.Final at [c]. This should allow the 8.2.8 instance to use
authorization on the remote store. I have not tested this, but everything should be
contained in the jar (including parser).
2. Use JDG 7.2.ER4 or newer in the keycloak EAP instances as a module. This is the first
version that allows for the user to deploy their own version of JDG as a module accessible
over JNDI.
- Will
[a] https://issues.jboss.org/browse/ISPN-6457
[b] https://issues.jboss.org/browse/ISPN-7868
[c] https://mvnrepository.com/artifact/org.infinispan/infinispan-cachestore-r...
----- Original Message -----
> From: "Hynek Mlnarik" <hmlnarik(a)redhat.com>
> To: "Jared Blashka" <jblashka(a)redhat.com>
> Cc: "Marek Posolda" <mposolda(a)redhat.com>, "Boleslaw
Dawidowicz" <bdawidow(a)redhat.com>, "keycloak-dev"
> <keycloak-dev(a)lists.jboss.org>, "William Burns"
<wburns(a)redhat.com>, "iam-list" <iam-list(a)redhat.com>
> Sent: Thursday, March 1, 2018 3:33:17 AM
> Subject: Re: [keycloak-dev] Cross-datacenter configuration issues
>
> I can only agree that it seems to be a difference between Infinispan server
> and JDG, since we did test it in Amazon where each instance of Keycloak and
> Infinispan was installed on separate VM [1]. Whether this difference might
> be indeed there should be confirmed by someone from JDG team. William,
> could you please comment here?
>
> [1]
> http://blog.keycloak.org/2018/01/keycloak-cross-data-center-setup-in-aws....
>
> On Thu, Mar 1, 2018 at 9:25 AM, Marek Posolda <mposolda(a)redhat.com> wrote:
>
>> I've just simulated the issue and created
>> https://issues.jboss.org/browse/KEYCLOAK-6783 . I am looking at it.
>>
>> What works and what we tested is:
>>
>> * Setup with infinispan-server-8.2.8 on "local" network (infinispan
>> server bind on loopback address like "localhost" . Different
>> infinispan servers running on the same laptop, but on various port
>> offsets)
>>
>> * Setup with JDG server 7.1.0 on "local" network (JDG server bound
on
>> loopback address like "localhost" . Different JDG servers running
on
>> the same laptop, but on various port offsets)
>>
>> * Setup with infinispan-server-8.2.8 on "real" network (testing
with
>> infinispan hosts bound to real host with IP addresses like 192.168.0.1
>> )
>>
>> We didn't test the combination with JDG server bind on "real"
addresses
>> and this is the only one where the issue happens
>>
>> It seems JDG 7.1.0 has some additional security when compared with the
>> community infinispan-server 8.2.8 .
>>
>> The easiest workaround for you might be to test with community
>> infinispan-server 8.2.8 instead of JDG 7.1.0 . Server can be downloaded
>> from this address:
>> http://downloads.jboss.org/infinispan/8.2.8.Final/
>> infinispan-server-8.2.8.Final-bin.zip
>> .
>>
>> I hope to update you later today once I have some more info. Thanks for
>> the report and all the details you mentioned.
>>
>> Marek
>>
>>
>> On 28/02/18 21:36, Jared Blashka wrote:
>>> Hey all,
>>>
>>> I'm working on testing out the cross-datacenter replication
>>> configuration in our development environment and I'm running into some
>>> issues.
>>>
>>> I stood up some JDG 7.1 instances and some RH-SSO 7.2 instances all
>>> running on my localhost all with different port offsets, followed the
>>> instructions[1], and everything seemed to work well enough.
>>>
>>> Once I got beyond that and tried running RH-SSO and JDG on separate
>>> servers I started running into issues[2] during RH-SSO startup. Looks
>>> like RH-SSO is unable to connect to the remote ___script_cache but
>>> that cache isn't mentioned anywhere in the RH-SSO documentation. The
>>> error message (and online searching) indicates that this cache only
>>> allows remote connections if authorization is enabled. I didn't see
>>> any mention of configuration related to authentication or security for
>>> the remote caches in the documentation either.
>>>
>>> At this point we roped in a JDG expert (cc'ed here) and found some
>>> additional Infinispan documentation[3] on how to add authentication to
>>> the *remote* caches within the JDG configuration but nothing much in
>>> the way of adding authentication to the client cache configuration
>>> inside RH-SSO that didn't involve programmatic changes. After some
>>> additional searching we found some info[4] detailing how to add
>>> security configurations to a remote-cache configuration in Infinispan
>>> *9.1* but EAP 7.1 is only running Infinispan *8.2* which doesn't have
>>> these changes.
>>>
>>> How did you get this working?
>>>
>>> Jared Blashka - Identity & Access Management
>>>
>>>
>>> [1]
>>> https://access.redhat.com/documentation/en-us/red_hat_
>> single_sign-on/7.2/pdf/server_installation_and_
>> configuration_guide/Red_Hat_Single_Sign-On-7.2-Server_Installation_and_
>> Configuration_Guide-en-US.pdf#__WKANCHOR_1e
>>> [2] http://pastebin.test.redhat.com/559674
>>> [3]
>>> http://infinispan.org/docs/stable/server_guide/server_
>> guide.html#general_concepts
>>> [4]
>>> https://docs.jboss.org/infinispan/9.1/configdocs/
>> infinispan-cachestore-remote-config-9.1.html
>>
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>
>
> --
>
> --Hynek
>
3:24 p.m.
New subject: Cross-datacenter configuration issues
----- Original Message -----
From: "Marek Posolda" <mposolda(a)redhat.com>
To: "William Burns" <wburns(a)redhat.com>, "Hynek Mlnarik"
<hmlnarik(a)redhat.com>
Cc: "Jared Blashka" <jblashka(a)redhat.com>, "Boleslaw
Dawidowicz" <bdawidow(a)redhat.com>, "keycloak-dev"
<keycloak-dev(a)lists.jboss.org>, "iam-list" <iam-list(a)redhat.com>,
"Tristan Tarrant" <ttarrant(a)redhat.com>, "Pedro
Zapata Fernandez" <pzapataf(a)redhat.com>
Sent: Friday, March 2, 2018 7:05:09 AM
Subject: Re: [keycloak-dev] Cross-datacenter configuration issues
Thanks William for the help!
I've tried your approach (1) and I've updated just
infinispan-cachestore-remote to 9.1.6 and left the other dependencies
not updated. It failed for me with some NoClassDefFoundErrors. This is
not so surprising as the other infinispan modules and also
wildfly-clustering integration and parser modules were still using the
old 8.2.8 versions and there seem to be some API incompatibility though,
which break things. Here is stacktrace if you want to take a look:
https://pastebin.com/hzDSX2um
Yeah actually I was talking with Jared about this yesterday. Unfortunately it doesn't
work. I was thinking of the embedded mode which allows you to use custom parsers. So this
isn't an option.
In the end, I've chosen a bit different solution, which works for me.
The RemoteStore of 8.2.8 doesn't support "security" but the RemoteCache
of 8.2.8 supports that. So I've changed some integration code on
Keycloak side to read RemoteCache configuration, which is retrieved from
the RemoteStore, and use it as a configuration template. Then I've
programatically added security into it and retrieve "secured"
RemoteCacheManager to access '__script_cache' and some parts of our
code, which need to execute remote scripts. See [1].
The RemoteStore itself is not secured. So on JDG side, I need to keep 2
hotrod endpoints (one unsecured accessed by RemoteStores and second
secured accessed by __script cache). I've added setup instructions to
the JIRA: [2] . Jared, I hope this can help you to have things working.
There maybe also another approach, which Hynek pointed, that we will
somehow bypass security on JDG side at all. Picketbox has some login
modules like org.jboss.security.auth.spi.AnonLoginModule and
org.jboss.security.auth.spi.IdentityLoginModule, which might allow to
configure JAAS realm in a way that "anonymous" user will be
automatically authenticated and role ___script_manager' granted to him.
In other words, this is completely bypass security. But possibly it may
allow to have things working without require changes on Keycloak code.
Another thing is, that I didn't manage to have it working, but I didn't
try to deeply and I am not too familiar with the Wildfly security +
picketbox stuff.
So to summary, The approaches I see is:
(1) Upgrade all infinispan modules in Keycloak to use 9.1.6 (or JDG
7.2.ER4). Will require to upgrade wildfly infinispan subsystem and bunch
of other dependencies (maybe whole EAP in the end). I guess it's not an
option regarding support, testing etc. On the other hand, it will allow
to configure RemoteStore security OOTB.
You and others are more familiar with how modules work. TBH I am not that familiar, but
this sounds like a no go to me as well.
(2) Create our own subclass of Infinispan RemoteStore on Keycloak side,
which will allow to configure secure RemoteCache instances. Will require
changes on Keycloak sources, and also in the Keycloak servers
configuration, which will need to use new implementation of RemoteStore,
so the configuration of the RemoteStore will be completely changed.
(3) Use the approach, which works for me and which I put in JIRA. So
have 2 hotrod endpoints on JDG side (unsecured and secured). RemoteStore
access the unsecured endpoint. Just some sources, which deals with
__script cache, access the secured hotrod endpoint. This requires to
"patch" module keycloak-model-infinispan on RHSSO side and also require
some smaller changes in the configuration (some new configuration
properties are added on RHSSO side to allow configuration of hotrod
endpoint security, username ,password etc).
(4) Bypass JDG security entirely with some picketbox "anonymous access"
JAAS login modules. I didn't manage to have it working and it completely
bypass security. On the other hand, it seems to be the only solution,
which won't require changes on Keycloak/RHSSO side. But we don't know
yet if it works...
Actually there is another way to bypass security that Tristan brought up that we were was
discussing. For JDG 7.2.CR1 we can relax access to protected caches, such as scriptcache,
so that there is no global authorization check (removing a single if block). This way the
cache just uses the normal authorization checks if it is even enabled for that cache.
Unfortunately CR1 is not going to be available for another 3 weeks though. In the mean
time Jared and I discussed just testing with community server version 8.2.8 for the
standalone install until we can get JDG 7.2.CR1 installed there. He may even want to test
with something a little newer.
Unfortunately that means that this version of keycloak would only work with JDG 7.2.0.CR1
or ISPN 9.0.3 [1] and newer. Either way I will be creating a PR for this on the JDG 7.2
branch today.
[1] https://issues.jboss.org/browse/ISPN-7814
For now, my vote is to go with (3) and it's the only working. Any more
options?
[1]
https://github.com/mposolda/keycloak/blob/jdg-security/model/infinispan/s...
[2]
https://issues.jboss.org/browse/KEYCLOAK-6783?focusedCommentId=13540721&a...
Marek
On 01/03/18 15:42, William Burns wrote:
> So I was helping out Jared when found this issue.
>
> This is caused by a combination of issues due to varying versions
> (Infinispaan 8.2.8 and JDG 7.1)
>
> 1. JDG 7.1 only allows accessing internal caches remotely over loopback or
> when authorization is enabled. This came about in Infinispan 9.0 in [a]
> 2. Infinispan 8.2.6 does not allow for remote cache stores to use
> authentication. This was added in Infinispan 9.0.(2|3) and 9.1 in [b]
>
> Unfortunately with the combination of the 2 there is now way for an ISPN
> pre 9.0.2 from accessing a remote server 9.0 internal caches not on
> loopback, such as the script cache.
>
> There are 2 ways that I was able to come up with (thanks Tristan for the
> second)
>
> 1. Replace the infinispan-cachestore-remote.jar in the ISPN 8.2.6 install
> to have a newer version such as 9.1.6.Final at [c]. This should allow the
> 8.2.8 instance to use authorization on the remote store. I have not tested
> this, but everything should be contained in the jar (including parser).
> 2. Use JDG 7.2.ER4 or newer in the keycloak EAP instances as a module. This
> is the first version that allows for the user to deploy their own version
> of JDG as a module accessible over JNDI.
>
> - Will
>
> [a] https://issues.jboss.org/browse/ISPN-6457
> [b] https://issues.jboss.org/browse/ISPN-7868
> [c]
> https://mvnrepository.com/artifact/org.infinispan/infinispan-cachestore-r...
>
> ----- Original Message -----
>> From: "Hynek Mlnarik" <hmlnarik(a)redhat.com>
>> To: "Jared Blashka" <jblashka(a)redhat.com>
>> Cc: "Marek Posolda" <mposolda(a)redhat.com>, "Boleslaw
Dawidowicz"
>> <bdawidow(a)redhat.com>, "keycloak-dev"
>> <keycloak-dev(a)lists.jboss.org>, "William Burns"
<wburns(a)redhat.com>,
>> "iam-list" <iam-list(a)redhat.com>
>> Sent: Thursday, March 1, 2018 3:33:17 AM
>> Subject: Re: [keycloak-dev] Cross-datacenter configuration issues
>>
>> I can only agree that it seems to be a difference between Infinispan
>> server
>> and JDG, since we did test it in Amazon where each instance of Keycloak
>> and
>> Infinispan was installed on separate VM [1]. Whether this difference might
>> be indeed there should be confirmed by someone from JDG team. William,
>> could you please comment here?
>>
>> [1]
>> http://blog.keycloak.org/2018/01/keycloak-cross-data-center-setup-in-aws....
>>
>> On Thu, Mar 1, 2018 at 9:25 AM, Marek Posolda <mposolda(a)redhat.com>
wrote:
>>
>>> I've just simulated the issue and created
>>> https://issues.jboss.org/browse/KEYCLOAK-6783 . I am looking at it.
>>>
>>> What works and what we tested is:
>>>
>>> * Setup with infinispan-server-8.2.8 on "local" network
(infinispan
>>> server bind on loopback address like "localhost" . Different
>>> infinispan servers running on the same laptop, but on various port
>>> offsets)
>>>
>>> * Setup with JDG server 7.1.0 on "local" network (JDG server
bound on
>>> loopback address like "localhost" . Different JDG servers
running on
>>> the same laptop, but on various port offsets)
>>>
>>> * Setup with infinispan-server-8.2.8 on "real" network (testing
with
>>> infinispan hosts bound to real host with IP addresses like
>>> 192.168.0.1
>>> )
>>>
>>> We didn't test the combination with JDG server bind on "real"
addresses
>>> and this is the only one where the issue happens
>>>
>>> It seems JDG 7.1.0 has some additional security when compared with the
>>> community infinispan-server 8.2.8 .
>>>
>>> The easiest workaround for you might be to test with community
>>> infinispan-server 8.2.8 instead of JDG 7.1.0 . Server can be downloaded
>>> from this address:
>>> http://downloads.jboss.org/infinispan/8.2.8.Final/
>>> infinispan-server-8.2.8.Final-bin.zip
>>> .
>>>
>>> I hope to update you later today once I have some more info. Thanks for
>>> the report and all the details you mentioned.
>>>
>>> Marek
>>>
>>>
>>> On 28/02/18 21:36, Jared Blashka wrote:
>>>> Hey all,
>>>>
>>>> I'm working on testing out the cross-datacenter replication
>>>> configuration in our development environment and I'm running into
some
>>>> issues.
>>>>
>>>> I stood up some JDG 7.1 instances and some RH-SSO 7.2 instances all
>>>> running on my localhost all with different port offsets, followed the
>>>> instructions[1], and everything seemed to work well enough.
>>>>
>>>> Once I got beyond that and tried running RH-SSO and JDG on separate
>>>> servers I started running into issues[2] during RH-SSO startup. Looks
>>>> like RH-SSO is unable to connect to the remote ___script_cache but
>>>> that cache isn't mentioned anywhere in the RH-SSO documentation.
The
>>>> error message (and online searching) indicates that this cache only
>>>> allows remote connections if authorization is enabled. I didn't see
>>>> any mention of configuration related to authentication or security for
>>>> the remote caches in the documentation either.
>>>>
>>>> At this point we roped in a JDG expert (cc'ed here) and found some
>>>> additional Infinispan documentation[3] on how to add authentication to
>>>> the *remote* caches within the JDG configuration but nothing much in
>>>> the way of adding authentication to the client cache configuration
>>>> inside RH-SSO that didn't involve programmatic changes. After some
>>>> additional searching we found some info[4] detailing how to add
>>>> security configurations to a remote-cache configuration in Infinispan
>>>> *9.1* but EAP 7.1 is only running Infinispan *8.2* which doesn't
have
>>>> these changes.
>>>>
>>>> How did you get this working?
>>>>
>>>> Jared Blashka - Identity & Access Management
>>>>
>>>>
>>>> [1]
>>>> https://access.redhat.com/documentation/en-us/red_hat_
>>> single_sign-on/7.2/pdf/server_installation_and_
>>> configuration_guide/Red_Hat_Single_Sign-On-7.2-Server_Installation_and_
>>> Configuration_Guide-en-US.pdf#__WKANCHOR_1e
>>>> [2] http://pastebin.test.redhat.com/559674
>>>> [3]
>>>> http://infinispan.org/docs/stable/server_guide/server_
>>> guide.html#general_concepts
>>>> [4]
>>>> https://docs.jboss.org/infinispan/9.1/configdocs/
>>> infinispan-cachestore-remote-config-9.1.html
>>>
>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>
>>
>> --
>>
>> --Hynek
>>
6:20 p.m.
New subject: Cross-datacenter configuration issues
On 02/03/18 15:24, William Burns wrote:
> (3) Use the approach, which works for me and which I put in JIRA.
So
> have 2 hotrod endpoints on JDG side (unsecured and secured). RemoteStore
> access the unsecured endpoint. Just some sources, which deals with
> __script cache, access the secured hotrod endpoint. This requires to
> "patch" module keycloak-model-infinispan on RHSSO side and also require
> some smaller changes in the configuration (some new configuration
> properties are added on RHSSO side to allow configuration of hotrod
> endpoint security, username ,password etc).
>
> (4) Bypass JDG security entirely with some picketbox "anonymous access"
> JAAS login modules. I didn't manage to have it working and it completely
> bypass security. On the other hand, it seems to be the only solution,
> which won't require changes on Keycloak/RHSSO side. But we don't know
> yet if it works...
Actually there is another way to bypass security that Tristan brought up that we were was
discussing. For JDG 7.2.CR1 we can relax access to protected caches, such as scriptcache,
so that there is no global authorization check (removing a single if block). This way the
cache just uses the normal authorization checks if it is even enabled for that cache.
Unfortunately CR1 is not going to be available for another 3 weeks though. In the mean
time Jared and I discussed just testing with community server version 8.2.8 for the
standalone install until we can get JDG 7.2.CR1 installed there. He may even want to test
with something a little newer.
Thanks for the update.
If Jared is fine to temporarily test with Infinispan server 8.2.8, it's
cool. We will try to see if we can bypass security even for JDG 7.1
(Hynek is looking into picketbox "anonymous" modules) and there is this
patch from me to secure access to "_script" cache, which works with 7.1,
but will require to patch Keycloak/RHSSO.
I afraid we can officially update to JDG 7.2 at the moment, when 7.2.GA
is released. However it may be good to test early with 7.2, so we can
find earlier if some fixes are needed on JDG side and notify you before
GA. Hopefully we have time for trying that... :)
BTV. Will JDG 7.2 have better "pagination" support for RemoteCaches, so
I can do something like:
remoteCache.keySet().stream().skip(100).limit(10).collect(Collectors.toList());
to ensure that just 10 items from 100 to 110 are sent over the network?
And will it have Server tasks support? If yes, we will be hopefully able
to get rid of remote scripting entirely :)
But problem is also "client" (Keycloak) side as we rely on the
infinispan version provided by Wildfly/EAP. In other words, we are
always few months (or years?) behind all the cool features and fixes,
which you are currently developing in infinispan master :(
Marek
Unfortunately that means that this version of keycloak would only work with JDG 7.2.0.CR1
or ISPN 9.0.3 [1] and newer. Either way I will be creating a PR for this on the JDG 7.2
branch today.
[1]https://issues.jboss.org/browse/ISPN-7814
2256
days inactive
2256
days old
2 comments
2 participants
participants (2)
-
Marek Posolda -
William Burns