From sthorger at redhat.com Tue Nov 1 03:04:55 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Tue, 1 Nov 2016 08:04:55 +0100 Subject: [keycloak-dev] User SPI cache policies In-Reply-To: <742bd089-d45b-d1f1-2fc4-db44c50dad18@redhat.com> References: <742bd089-d45b-d1f1-2fc4-db44c50dad18@redhat.com> Message-ID: Another question around the user cache policies. As these are eviction policies would they not just be applicable to one node? For example eviction is daily. Node 1 loads the user at 2am. Node 2 loads the user at 2pm. The user is then changed at 3pm. In this case there's 12 hours where node 1 and node 2 will see different data for the user. Sounds like that could cause all sorts of strange behavior. On 31 October 2016 at 19:33, Bill Burke wrote: > You need to know the user before you can evict it. username can be > obtained differently from multiple different authenticators: spnego, > username/password UI, basic auth, etc.. > > On 10/31/16 9:41 AM, Stian Thorgersen wrote: > > Could we not do it as a special first authenticator in the flow? > > On 31 October 2016 at 14:08, Bill Burke wrote: > >> >> >> On 10/31/16 8:51 AM, Stian Thorgersen wrote: >> >> >> >> On 31 October 2016 at 13:49, Bill Burke wrote: >> >>> >>> >>> On 10/31/16 1:48 AM, Stian Thorgersen wrote: >>> >>>> What about evict on authenticate (load from store when user >>>> authenticates)? I think that would be the most useful policy. >>>> >>>> That would need to be implemented at the authenticator level. >> >> >> Implementation details aside, should we not have it? It seems like the >> most likely time you want to fetch the user and especially credentials. >> >> >> Yeah, its a great idea. Implementation details matter though as I'm not >> sure this can be reliably done without coding this in each top-level >> authenticator and requiring an authenticator provider developer to be aware >> of this policy. >> >> Bill >> > > > From sthorger at redhat.com Tue Nov 1 05:04:09 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Tue, 1 Nov 2016 10:04:09 +0100 Subject: [keycloak-dev] Support for key rotation in SAML Redirect binding In-Reply-To: References: <58aa2fa8-3481-2c4a-49ac-b598df2ff560@redhat.com> <6099fbdc-08a3-ec73-7551-0613da4fc74d@redhat.com> <876cd989-74f1-58b9-922a-cadb382562fa@redhat.com> <4c74cea9-88a6-c79f-119f-f66d8f3c9d1b@redhat.com> <91ba1884-c6dc-1b9a-d8d6-51c34a9b1f33@redhat.com> Message-ID: Hynek - your responses seems to be missing the list. Maybe you didn't use reply to all? If SAML is missing a standard way to specify an id for the keys used that's just stupid IMO. Signature validation is expensive and if there are multiple keys available it would be plain stupid to have to iterate over all to check. Granted in most cases we're only talking about a few keys. Sounds like the extensions approach is the best one. The id should be within the assertion and not passed separately. John - Keycloak has SP libraries as well, so this is an extension/optimalization that our adapters can use. Do other SP libs (i.e. mod_auth_mellon) go through the list of keys in order? If so we just need to make sure the list is sorted by most recent (and active key) first so there's a high probability the first key is the correct one. With regards to federation not sure what your comment that the instances needs to share keys is about. One IdP has a list of keys and the other IdP needs to figure out which of the keys are the correct one for an assertion. On 31 October 2016 at 18:14, John Dennis wrote: > On 10/31/2016 11:36 AM, Hynek Mlnarik wrote: > > Surely KC implements both SAML SP and IdP. > > KC is mostly an IdP. The only case I'm aware of where KC operates as an > SP is when KC federates to another IdP (i.e. KC is an identity hub that > is configured to authenticate against other IdP's). For the optimization > you're discussing to be workable the other IdP must also understand the > key id usage (i.e. is another KC instance) *and* share the key id's. > That seems to me to be an uncommon deployment. > > > I am afraid that in a strict sense, there is also no KC-to-SP or > > SP-to-KC communiication. > > Really? SAML is mostly IdP-to-SP and SP-to-Idp communication. > > > But by natural extension of concepts, by "KC-to-KC", an IdP-to-SP > > communication is meant where KC is implementor of both parts. > > See above. > > > SAML 2.0 is designed to be extensible and allows Implementation > > specific extensions that are not interpreted if the receiving party > > does not know how to handle them. This is interoperable as long as > > the meaning of the original SAML message retains the same meaning. > > Hints like key ID are hence valid use of this extension. > > Sure, but I still don't understand when you could take advantage of this > (see above). How often do you think KC is going to federate to another > KC instance that shares the same key ids? > > > Just for the record - SAML IdP is represented by KC server, SAML SP > > part is handled by KC adapters. > > > -- > John > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From mitya at cargosoft.ru Tue Nov 1 05:58:30 2016 From: mitya at cargosoft.ru (Dmitry Telegin) Date: Tue, 01 Nov 2016 12:58:30 +0300 Subject: [keycloak-dev] Making ServerInfoAdminResource::ENUMS mutable In-Reply-To: References: <1476421798.23974.2.camel@cargosoft.ru> <1476731718.4820.0.camel@cargosoft.ru> <1476820622.8706.1.camel@cargosoft.ru> Message-ID: <1477994310.4346.1.camel@cargosoft.ru> Hi Stian, > On 18 October 2016 at 21:57, Dmitry Telegin > wrote: > > Hi Stian, > > > > I completely agree on the first point, this deserves a better > > solution. But knowing that it would require digging deep into the > > guts of KeyCloak, I just didn't dare :) > > > > I would see providers somehow registering their custom enum keys, > > either via API call or via callback, and KeyCloak would maintain > > the list of custom values per provider. ServerInfoAdminResource > > then could return a combined list of enum keys, both built-in and > > defined by providers. But I'm afraid that would require changes to > > API and/or interfaces. What do you think? How would you approach > > this problem yourself? > > We have SPIs to make features pluggable so that's the way to go. We'd > probably want a AdminResourceTypeSpi and at the same time refactor > the current enum to use this. I was thinking about SPIs, but to me the ResourceType use case seemed too minor to deserve its own dedicated SPI. OTOH, it would look quite natural as a part of more generic Admin Resource SPI. I remember it's been on the list for a long time, then why not finally getting hands dirty? Please take a look at the BeerCloak example I've announced earlier in the list. There, I've tried to collect all the techniques I've been using up to the moment to develop ad-hoc admin resources. I think I've got enough experience and understanding to do the whole Admin Resource SPI thing (with hypothetical AdminResourceTypeSpi becoming a part of it, along with roles handling, authorization, event logging etc.) That's it, I'd like to volunteer to implement Admin Resource SPI, and to do it right way ("well designed, generically usable and documented"). Please let me know what do you think. If you greenlight this, I think we'd start a new thread to discuss the design.? > ? > > And BTW, what's wrong with the second PR? The github comment seems > > a bit unclear, so could you please elaborate? It has completely > > different scope, though deals with the same feature. It's not about > > exposing anything via static methods, it's about making > > ResourceType extensible, following a well-defined pattern. I was > > sure that Marek supported it, and I've implemented it in accordance > > with his recommendations. > > Seemed rather hacky to me, but maybe I'm mistaking something. Can you > show an example on how it would actually be used? I don't see how > you'd add additional types and how you'd actually use the resource > type not knowing what type it is. Seems all operations valid on an > enum becomes invalid in this case. See beercloak.BeerResource::create and other methods. There are commented-out lines that show how it could look like. But I don't think it's much relevant now, since we may come up with completely different design with Admin Resource SPI. Cheers, Dmitry From bburke at redhat.com Tue Nov 1 10:14:55 2016 From: bburke at redhat.com (Bill Burke) Date: Tue, 1 Nov 2016 10:14:55 -0400 Subject: [keycloak-dev] User SPI cache policies In-Reply-To: References: <742bd089-d45b-d1f1-2fc4-db44c50dad18@redhat.com> Message-ID: Your analysis is incorrect for EVICT_DAILY and EVICT_WEEKLY. With these policies, when a user is cached, time until next expiration is calculated and the cache timeout for that entry is set in infinispan... ALSO When there is a cache hit, the timestamp on the cached user is checked to see if it should have expired. If it has its invalidated (cluster wide) and the user is loaded from the DB. If the clocks on cluster nodes are out of sync, then yes, there will possibly be issues. Its done this way because Infinispan does not guarantee that eviction will happen at the indicated timeout. Also its impossible to test if I just relied on Infinispan to evict. So, in your example, if Node 1 loads user at 2am, Node 2 loads the user at 2pm, the eviction time is 3pm, then after 3pm, both node 1 and node 2 cache for that user are not valid anymore. Your analysis is correct for MAX_LIFESPAN, but only if a third-party has modified the user outside the scope of Keycloak. This inconsistent cache could happen irregardless of the cache policy in that external-party scenario. On 11/1/16 3:04 AM, Stian Thorgersen wrote: > Another question around the user cache policies. As these are eviction > policies would they not just be applicable to one node? For example > eviction is daily. Node 1 loads the user at 2am. Node 2 loads the user > at 2pm. The user is then changed at 3pm. In this case there's 12 hours > where node 1 and node 2 will see different data for the user. Sounds > like that could cause all sorts of strange behavior. > > On 31 October 2016 at 19:33, Bill Burke > wrote: > > You need to know the user before you can evict it. username can be > obtained differently from multiple different authenticators: > spnego, username/password UI, basic auth, etc.. > > > On 10/31/16 9:41 AM, Stian Thorgersen wrote: >> Could we not do it as a special first authenticator in the flow? >> >> On 31 October 2016 at 14:08, Bill Burke > > wrote: >> >> >> >> On 10/31/16 8:51 AM, Stian Thorgersen wrote: >>> >>> >>> On 31 October 2016 at 13:49, Bill Burke >> > wrote: >>> >>> >>> >>> On 10/31/16 1:48 AM, Stian Thorgersen wrote: >>> >>> What about evict on authenticate (load from store >>> when user authenticates)? I think that would be the >>> most useful policy. >>> >>> That would need to be implemented at the authenticator >>> level. >>> >>> >>> Implementation details aside, should we not have it? It >>> seems like the most likely time you want to fetch the user >>> and especially credentials. >> Yeah, its a great idea. Implementation details matter though >> as I'm not sure this can be reliably done without coding this >> in each top-level authenticator and requiring an >> authenticator provider developer to be aware of this policy. >> >> Bill >> >> > > From jdennis at redhat.com Tue Nov 1 11:23:05 2016 From: jdennis at redhat.com (John Dennis) Date: Tue, 1 Nov 2016 11:23:05 -0400 Subject: [keycloak-dev] Support for key rotation in SAML Redirect binding In-Reply-To: References: <58aa2fa8-3481-2c4a-49ac-b598df2ff560@redhat.com> <6099fbdc-08a3-ec73-7551-0613da4fc74d@redhat.com> <876cd989-74f1-58b9-922a-cadb382562fa@redhat.com> <4c74cea9-88a6-c79f-119f-f66d8f3c9d1b@redhat.com> <91ba1884-c6dc-1b9a-d8d6-51c34a9b1f33@redhat.com> Message-ID: <75b51641-6d7f-abab-2802-39ae512bf09a@redhat.com> On 11/01/2016 05:04 AM, Stian Thorgersen wrote: > Hynek - your responses seems to be missing the list. Maybe you didn't > use reply to all? > > If SAML is missing a standard way to specify an id for the keys used > that's just stupid IMO. Signature validation is expensive and if there > are multiple keys available it would be plain stupid to have to iterate > over all to check. Granted in most cases we're only talking about a few > keys. > > Sounds like the extensions approach is the best one. The id should be > within the assertion and not passed separately. > > John - Keycloak has SP libraries as well, so this is an > extension/optimalization that our adapters can use. Do other SP libs > (i.e. mod_auth_mellon) go through the list of keys in order? If so we > just need to make sure the list is sorted by most recent (and active > key) first so there's a high probability the first key is the correct > one. With regards to federation not sure what your comment that the > instances needs to share keys is about. One IdP has a list of keys and > the other IdP needs to figure out which of the keys are the correct one > for an assertion. 1) SAML does have a mechanism to transmit a keyID, just not in the singular case when the HTTP-Redirect binding is used. The reason being the SAML message is transmitted as a URL query parameter which has practical length limitations. I believe OAuth and OIDC, in conjunction with JOSE, JWT etc. have similar restrictions to accommodate URL length constraints (sorry, can't give you pointers right off the top of my head for these other protocols). 2) In the singular case of the HTTP-Redirect binding you can still sign the AuthnRequest but unlike all other SAML protocol messages there is no official way to include a keyID. This was Hynek's issue/question. 3) The HTTP-Redirect binding is typically used to send a AuthnRequest from a SP to an IdP. It's not the only SAML binding you can use to transmit an AuthnRequest from a SP to an IdP, you can also use the HTTP-POST or SOAP bindings which do not have the length constraint imposed by the HTTP-Redirect binding. 4) Using another binding will not work for standard Web SSO because the flow is different. But that's OK because the proposed mechanism will only work with cooperating entities, e.g. both the SP and the IdP are Keycloak implementations. But since this is proposed only for KC to KC you're free to use whatever binding you want. 5) The only SAML message of concern here is AuthnRequest because only the HTTP-Redirect binding is affected by the keyID limitation. Although I believe it's legal to pass other SAML messages in a HTTP-Redirect binding I'm not aware of any such usage, so this is a very specific case. 6) My comment about instances sharing keys (actually keyID's) was in reference to the fact *both* parties have to know the keyID so you need a mechanism to share that information, the entity metadata is the best mechanism, but you could also do it via some out of band mechanism. 7) The period of time during in which multiple keys are in effect is expected to be relatively short. The chance of selecting the correct key on first attempt is high. The keyID issue affects only one protocol message. The lack of a keyID affects only KC to KC messages. Add all these together and you've got a very narrow case you're trying to optimize for. Is this where you need to be spending your optimization efforts? However this is an easy solution and goes directly to Hynek's question/issue, see next ... 8) I believe adding the keyID as an additional query parameter for HTTP-Redirect is the easiest and cleanest solution. SAML does not prohibit additional query parameters. Query parameters are consistent with the rest of the HTTP-Redirect binding. Of course one has to be careful concerning the total URL length. Of course you could also use a different binding as per item 3 because this is a KC to KC case. 9) Mellon does not currently support key rotation, It would be an RFE. but one we would be interested in implementing. Actually I don't think it would be implemented in mellon, rather it would need to be implemented in the lasso library that mellon uses for SAML. While doing that work we could also make sure keyID's are utilized. 10) I believe Shibboleth v3 supports key rotation, if so it would be valuable to see how they approached the problem. -- John From hmlnarik at redhat.com Tue Nov 1 11:49:59 2016 From: hmlnarik at redhat.com (Hynek Mlnarik) Date: Tue, 1 Nov 2016 16:49:59 +0100 Subject: [keycloak-dev] Support for key rotation in SAML Redirect binding In-Reply-To: <75b51641-6d7f-abab-2802-39ae512bf09a@redhat.com> References: <58aa2fa8-3481-2c4a-49ac-b598df2ff560@redhat.com> <6099fbdc-08a3-ec73-7551-0613da4fc74d@redhat.com> <876cd989-74f1-58b9-922a-cadb382562fa@redhat.com> <4c74cea9-88a6-c79f-119f-f66d8f3c9d1b@redhat.com> <91ba1884-c6dc-1b9a-d8d6-51c34a9b1f33@redhat.com> <75b51641-6d7f-abab-2802-39ae512bf09a@redhat.com> Message-ID: On Tue, Nov 1, 2016 at 4:23 PM, John Dennis wrote: > On 11/01/2016 05:04 AM, Stian Thorgersen wrote: > >> Hynek - your responses seems to be missing the list. Maybe you didn't >> use reply to all? >> >> If SAML is missing a standard way to specify an id for the keys used >> that's just stupid IMO. Signature validation is expensive and if there >> are multiple keys available it would be plain stupid to have to iterate >> over all to check. Granted in most cases we're only talking about a few >> keys. >> >> Sounds like the extensions approach is the best one. The id should be >> within the assertion and not passed separately. >> >> John - Keycloak has SP libraries as well, so this is an >> extension/optimalization that our adapters can use. Do other SP libs >> (i.e. mod_auth_mellon) go through the list of keys in order? If so we >> just need to make sure the list is sorted by most recent (and active >> key) first so there's a high probability the first key is the correct >> one. With regards to federation not sure what your comment that the >> instances needs to share keys is about. One IdP has a list of keys and >> the other IdP needs to figure out which of the keys are the correct one >> for an assertion. >> > > 1) SAML does have a mechanism to transmit a keyID, just not in the > singular case when the HTTP-Redirect binding is used. The reason being the > SAML message is transmitted as a URL query parameter which has practical > length limitations. I believe OAuth and OIDC, in conjunction with JOSE, JWT > etc. have similar restrictions to accommodate URL length constraints > (sorry, can't give you pointers right off the top of my head for these > other protocols). > > 2) In the singular case of the HTTP-Redirect binding you can still sign > the AuthnRequest but unlike all other SAML protocol messages there is no > official way to include a keyID. This was Hynek's issue/question. > > 3) The HTTP-Redirect binding is typically used to send a AuthnRequest from > a SP to an IdP. It's not the only SAML binding you can use to transmit an > AuthnRequest from a SP to an IdP, you can also use the HTTP-POST or SOAP > bindings which do not have the length constraint imposed by the > HTTP-Redirect binding. > > 4) Using another binding will not work for standard Web SSO because the > flow is different. But that's OK because the proposed mechanism will only > work with cooperating entities, e.g. both the SP and the IdP are Keycloak > implementations. But since this is proposed only for KC to KC you're free > to use whatever binding you want. > > 5) The only SAML message of concern here is AuthnRequest because only the > HTTP-Redirect binding is affected by the keyID limitation. Although I > believe it's legal to pass other SAML messages in a HTTP-Redirect binding > I'm not aware of any such usage, so this is a very specific case. > > 6) My comment about instances sharing keys (actually keyID's) was in > reference to the fact *both* parties have to know the keyID so you need a > mechanism to share that information, the entity metadata is the best > mechanism, but you could also do it via some out of band mechanism. > Which out of band mechanism you mean? I am not aware of any standard one. > 7) The period of time during in which multiple keys are in effect is > expected to be relatively short. The chance of selecting the correct key on > first attempt is high. The keyID issue affects only one protocol message. > The lack of a keyID affects only KC to KC messages. Add all these together > and you've got a very narrow case you're trying to optimize for. Is this > where you need to be spending your optimization efforts? However this is an > easy solution and goes directly to Hynek's question/issue, see next ... > > 8) I believe adding the keyID as an additional query parameter for > HTTP-Redirect is the easiest and cleanest solution. SAML does not prohibit > additional query parameters. Query parameters are consistent with the rest > of the HTTP-Redirect binding. Of course one has to be careful concerning > the total URL length. Of course you could also use a different binding as > per item 3 because this is a KC to KC case. > Key ID in query parameter (or any additional query parameter) in the parameter would not protected by the signature, only SAMLRequest/SAMLResponse, RelayState, and SigAlg are. Including it in SAML protocol message has hence benefit in Key ID protection. > 9) Mellon does not currently support key rotation, It would be an RFE. but > one we would be interested in implementing. Actually I don't think it would > be implemented in mellon, rather it would need to be implemented in the > lasso library that mellon uses for SAML. While doing that work we could > also make sure keyID's are utilized. > That would be valuable. 10) I believe Shibboleth v3 supports key rotation, if so it would be > valuable to see how they approached the problem. I agree, thanks for the pointer. From jdennis at redhat.com Tue Nov 1 12:46:48 2016 From: jdennis at redhat.com (John Dennis) Date: Tue, 1 Nov 2016 12:46:48 -0400 Subject: [keycloak-dev] Support for key rotation in SAML Redirect binding In-Reply-To: References: <58aa2fa8-3481-2c4a-49ac-b598df2ff560@redhat.com> <6099fbdc-08a3-ec73-7551-0613da4fc74d@redhat.com> <876cd989-74f1-58b9-922a-cadb382562fa@redhat.com> <4c74cea9-88a6-c79f-119f-f66d8f3c9d1b@redhat.com> <91ba1884-c6dc-1b9a-d8d6-51c34a9b1f33@redhat.com> <75b51641-6d7f-abab-2802-39ae512bf09a@redhat.com> Message-ID: <2cae1c16-ab94-59cd-38bd-9d7cffe1602c@redhat.com> On 11/01/2016 11:49 AM, Hynek Mlnarik wrote: > 6) My comment about instances sharing keys (actually keyID's) was in > reference to the fact *both* parties have to know the keyID so you > need a mechanism to share that information, the entity metadata is > the best mechanism, but you could also do it via some out of band > mechanism. > > > Which out of band mechanism you mean? I am not aware of any standard one. By out of band I mean a non-standard exchange of information, not back channel. So yes, the only standard way I know is via metadata. > 8) I believe adding the keyID as an additional query parameter for > HTTP-Redirect is the easiest and cleanest solution. SAML does not > prohibit additional query parameters. Query parameters are > consistent with the rest of the HTTP-Redirect binding. Of course one > has to be careful concerning the total URL length. Of course you > could also use a different binding as per item 3 because this is a > KC to KC case. > > > Key ID in query parameter (or any additional query parameter) in the > parameter would not protected by the signature, only > SAMLRequest/SAMLResponse, RelayState, and SigAlg are. Including it in > SAML protocol message has hence benefit in Key ID protection. The recommendation is such messages be exchanged over a secure transport such as TLS, since this is an HTTP-Redirect binding that means HTTPS. That means all the query parameters are pretty safe. Even if the keyID were modified by a nefarious actor what would the practical consequences be? I suppose that might depend on how strict your implementation is, here is the basic choices as I see them: 1) keyID lookup fails, you fail the protocol exchange. 2) keyID lookup succeeds but fails to validate signature. You fail the protocol exchange. However if either of the two above cases occur you could be more forgiving and treat it as if no keyID had been passed and fall back to iterating over the keys bound to the entity, something you have to implement anyway because the keyID will often be absent. The xmlSig spec is silent on what happens if the keyID (actually the KeyName) does not produce a known key. It does not specify if one may fallback to application specific context or not. I personally feel it's OK to use application specific context, if one accepts that then the keyID if present becomes a hint not a mandate. In summary, I don't see tampering with the keyID a significant risk over HTTPS or if it's treated only as a hint a serious problem that could be used to compromise the communication. -- John From sthorger at redhat.com Wed Nov 2 03:08:40 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Wed, 2 Nov 2016 08:08:40 +0100 Subject: [keycloak-dev] User SPI cache policies In-Reply-To: References: <742bd089-d45b-d1f1-2fc4-db44c50dad18@redhat.com> Message-ID: How is the cache timeout propagated to the cluster is there a separate and new replicated cache for that? On 1 November 2016 at 15:14, Bill Burke wrote: > Your analysis is incorrect for EVICT_DAILY and EVICT_WEEKLY. With these > policies, when a user is cached, time until next expiration is calculated > and the cache timeout for that entry is set in infinispan... > > ALSO > > When there is a cache hit, the timestamp on the cached user is checked to > see if it should have expired. If it has its invalidated (cluster wide) > and the user is loaded from the DB. If the clocks on cluster nodes are out > of sync, then yes, there will possibly be issues. > > Its done this way because Infinispan does not guarantee that eviction will > happen at the indicated timeout. Also its impossible to test if I just > relied on Infinispan to evict. > > So, in your example, if Node 1 loads user at 2am, Node 2 loads the user at > 2pm, the eviction time is 3pm, then after 3pm, both node 1 and node 2 cache > for that user are not valid anymore. > > Your analysis is correct for MAX_LIFESPAN, but only if a third-party has > modified the user outside the scope of Keycloak. This inconsistent cache > could happen irregardless of the cache policy in that external-party > scenario. > > > > On 11/1/16 3:04 AM, Stian Thorgersen wrote: > > Another question around the user cache policies. As these are eviction > policies would they not just be applicable to one node? For example > eviction is daily. Node 1 loads the user at 2am. Node 2 loads the user at > 2pm. The user is then changed at 3pm. In this case there's 12 hours where > node 1 and node 2 will see different data for the user. Sounds like that > could cause all sorts of strange behavior. > > On 31 October 2016 at 19:33, Bill Burke wrote: > >> You need to know the user before you can evict it. username can be >> obtained differently from multiple different authenticators: spnego, >> username/password UI, basic auth, etc.. >> >> On 10/31/16 9:41 AM, Stian Thorgersen wrote: >> >> Could we not do it as a special first authenticator in the flow? >> >> On 31 October 2016 at 14:08, Bill Burke wrote: >> >>> >>> >>> On 10/31/16 8:51 AM, Stian Thorgersen wrote: >>> >>> >>> >>> On 31 October 2016 at 13:49, Bill Burke wrote: >>> >>>> >>>> >>>> On 10/31/16 1:48 AM, Stian Thorgersen wrote: >>>> >>>>> What about evict on authenticate (load from store when user >>>>> authenticates)? I think that would be the most useful policy. >>>>> >>>>> That would need to be implemented at the authenticator level. >>> >>> >>> Implementation details aside, should we not have it? It seems like the >>> most likely time you want to fetch the user and especially credentials. >>> >>> >>> Yeah, its a great idea. Implementation details matter though as I'm not >>> sure this can be reliably done without coding this in each top-level >>> authenticator and requiring an authenticator provider developer to be aware >>> of this policy. >>> >>> Bill >>> >> >> >> > > From bburke at redhat.com Wed Nov 2 09:46:47 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 2 Nov 2016 09:46:47 -0400 Subject: [keycloak-dev] User SPI cache policies In-Reply-To: References: <742bd089-d45b-d1f1-2fc4-db44c50dad18@redhat.com> Message-ID: <1622726a-c6af-f1c1-e257-cb8bb514cff4@redhat.com> The cache timeout does not need to be propagated as the user is only cached on the machine that requested access to the user. Its an invalidation cache. i.e. For the ldap provider, it has a cache policy that every day at 3pm any users cached before that time are invalidated. * Node 1 accesses a user at 12pm. User is not in cache. Node 1 caches it within invalidation cache with a timeout of 3 hours. 3pm - 12pm = 3 hours * Node 2 access same user at 1pm. User is not in its cache. Node 2 caches it with a timeout of 2 hours. * Node 1 access user at 4pm. If Infinispan has not evicted the user already and there is a cache hit, Keycloak checks the cache timestamp of the user. It will see that the cache timestamp is older than 3pm and invalidate the user across the cluster. On 11/2/16 3:08 AM, Stian Thorgersen wrote: > How is the cache timeout propagated to the cluster is there a separate > and new replicated cache for that? > > On 1 November 2016 at 15:14, Bill Burke > wrote: > > Your analysis is incorrect for EVICT_DAILY and EVICT_WEEKLY. With > these policies, when a user is cached, time until next expiration > is calculated and the cache timeout for that entry is set in > infinispan... > > ALSO > > When there is a cache hit, the timestamp on the cached user is > checked to see if it should have expired. If it has its > invalidated (cluster wide) and the user is loaded from the DB. If > the clocks on cluster nodes are out of sync, then yes, there will > possibly be issues. > > Its done this way because Infinispan does not guarantee that > eviction will happen at the indicated timeout. Also its > impossible to test if I just relied on Infinispan to evict. > > So, in your example, if Node 1 loads user at 2am, Node 2 loads the > user at 2pm, the eviction time is 3pm, then after 3pm, both node 1 > and node 2 cache for that user are not valid anymore. > > Your analysis is correct for MAX_LIFESPAN, but only if a > third-party has modified the user outside the scope of Keycloak. > This inconsistent cache could happen irregardless of the cache > policy in that external-party scenario. > > > > On 11/1/16 3:04 AM, Stian Thorgersen wrote: >> Another question around the user cache policies. As these are >> eviction policies would they not just be applicable to one node? >> For example eviction is daily. Node 1 loads the user at 2am. Node >> 2 loads the user at 2pm. The user is then changed at 3pm. In this >> case there's 12 hours where node 1 and node 2 will see different >> data for the user. Sounds like that could cause all sorts of >> strange behavior. >> >> On 31 October 2016 at 19:33, Bill Burke > > wrote: >> >> You need to know the user before you can evict it. username >> can be obtained differently from multiple different >> authenticators: spnego, username/password UI, basic auth, etc.. >> >> >> On 10/31/16 9:41 AM, Stian Thorgersen wrote: >>> Could we not do it as a special first authenticator in the flow? >>> >>> On 31 October 2016 at 14:08, Bill Burke >> > wrote: >>> >>> >>> >>> On 10/31/16 8:51 AM, Stian Thorgersen wrote: >>>> >>>> >>>> On 31 October 2016 at 13:49, Bill Burke >>>> > wrote: >>>> >>>> >>>> >>>> On 10/31/16 1:48 AM, Stian Thorgersen wrote: >>>> >>>> What about evict on authenticate (load from >>>> store when user authenticates)? I think that >>>> would be the most useful policy. >>>> >>>> That would need to be implemented at the >>>> authenticator level. >>>> >>>> >>>> Implementation details aside, should we not have it? It >>>> seems like the most likely time you want to fetch the >>>> user and especially credentials. >>> Yeah, its a great idea. Implementation details matter >>> though as I'm not sure this can be reliably done without >>> coding this in each top-level authenticator and >>> requiring an authenticator provider developer to be >>> aware of this policy. >>> >>> Bill >>> >>> >> >> > > From sthorger at redhat.com Wed Nov 2 13:42:01 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Wed, 2 Nov 2016 18:42:01 +0100 Subject: [keycloak-dev] User SPI cache policies In-Reply-To: <1622726a-c6af-f1c1-e257-cb8bb514cff4@redhat.com> References: <742bd089-d45b-d1f1-2fc4-db44c50dad18@redhat.com> <1622726a-c6af-f1c1-e257-cb8bb514cff4@redhat.com> Message-ID: Of course. Being a bit slow. It's at a specific time, not after an amount of time. For the MAX_LIFESPAN though is that not a potentially big issue? Especially if sessions are not sticky? On 2 November 2016 at 14:46, Bill Burke wrote: > The cache timeout does not need to be propagated as the user is only > cached on the machine that requested access to the user. Its an > invalidation cache. > > i.e. > > For the ldap provider, it has a cache policy that every day at 3pm any > users cached before that time are invalidated. > > * Node 1 accesses a user at 12pm. User is not in cache. Node 1 caches it > within invalidation cache with a timeout of 3 hours. 3pm - 12pm = 3 hours > > * Node 2 access same user at 1pm. User is not in its cache. Node 2 > caches it with a timeout of 2 hours. > > * Node 1 access user at 4pm. If Infinispan has not evicted the user > already and there is a cache hit, Keycloak checks the cache timestamp of > the user. It will see that the cache timestamp is older than 3pm and > invalidate the user across the cluster. > > On 11/2/16 3:08 AM, Stian Thorgersen wrote: > > How is the cache timeout propagated to the cluster is there a separate and > new replicated cache for that? > > On 1 November 2016 at 15:14, Bill Burke wrote: > >> Your analysis is incorrect for EVICT_DAILY and EVICT_WEEKLY. With these >> policies, when a user is cached, time until next expiration is calculated >> and the cache timeout for that entry is set in infinispan... >> >> ALSO >> >> When there is a cache hit, the timestamp on the cached user is checked to >> see if it should have expired. If it has its invalidated (cluster wide) >> and the user is loaded from the DB. If the clocks on cluster nodes are out >> of sync, then yes, there will possibly be issues. >> >> Its done this way because Infinispan does not guarantee that eviction >> will happen at the indicated timeout. Also its impossible to test if I >> just relied on Infinispan to evict. >> >> So, in your example, if Node 1 loads user at 2am, Node 2 loads the user >> at 2pm, the eviction time is 3pm, then after 3pm, both node 1 and node 2 >> cache for that user are not valid anymore. >> >> Your analysis is correct for MAX_LIFESPAN, but only if a third-party has >> modified the user outside the scope of Keycloak. This inconsistent cache >> could happen irregardless of the cache policy in that external-party >> scenario. >> >> >> >> On 11/1/16 3:04 AM, Stian Thorgersen wrote: >> >> Another question around the user cache policies. As these are eviction >> policies would they not just be applicable to one node? For example >> eviction is daily. Node 1 loads the user at 2am. Node 2 loads the user at >> 2pm. The user is then changed at 3pm. In this case there's 12 hours where >> node 1 and node 2 will see different data for the user. Sounds like that >> could cause all sorts of strange behavior. >> >> On 31 October 2016 at 19:33, Bill Burke wrote: >> >>> You need to know the user before you can evict it. username can be >>> obtained differently from multiple different authenticators: spnego, >>> username/password UI, basic auth, etc.. >>> >>> On 10/31/16 9:41 AM, Stian Thorgersen wrote: >>> >>> Could we not do it as a special first authenticator in the flow? >>> >>> On 31 October 2016 at 14:08, Bill Burke wrote: >>> >>>> >>>> >>>> On 10/31/16 8:51 AM, Stian Thorgersen wrote: >>>> >>>> >>>> >>>> On 31 October 2016 at 13:49, Bill Burke wrote: >>>> >>>>> >>>>> >>>>> On 10/31/16 1:48 AM, Stian Thorgersen wrote: >>>>> >>>>>> What about evict on authenticate (load from store when user >>>>>> authenticates)? I think that would be the most useful policy. >>>>>> >>>>>> That would need to be implemented at the authenticator level. >>>> >>>> >>>> Implementation details aside, should we not have it? It seems like the >>>> most likely time you want to fetch the user and especially credentials. >>>> >>>> >>>> Yeah, its a great idea. Implementation details matter though as I'm >>>> not sure this can be reliably done without coding this in each top-level >>>> authenticator and requiring an authenticator provider developer to be aware >>>> of this policy. >>>> >>>> Bill >>>> >>> >>> >>> >> >> > > From sthorger at redhat.com Thu Nov 3 05:46:45 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Thu, 3 Nov 2016 10:46:45 +0100 Subject: [keycloak-dev] Making ServerInfoAdminResource::ENUMS mutable In-Reply-To: <1477994310.4346.1.camel@cargosoft.ru> References: <1476421798.23974.2.camel@cargosoft.ru> <1476731718.4820.0.camel@cargosoft.ru> <1476820622.8706.1.camel@cargosoft.ru> <1477994310.4346.1.camel@cargosoft.ru> Message-ID: Introducing this as part of a admin resource spi would make perfect sense. One issue with introducing admin resource spi now is that it would have to be reworked once we rework how permissions are done. We will most likely go away from a simple role based permissions to utilizing the authorization services. Further, we simply don't have time to review or discuss a the admin resource spi at the moment as we're focusing on wrapping up 2.x series. At this point there is no chance that it would be included in 2.x in any case. If you'd like we can revisit this early next year, when we'd be able to discuss it for 3.x. On 1 November 2016 at 10:58, Dmitry Telegin wrote: > Hi Stian, > > > > On 18 October 2016 at 21:57, Dmitry Telegin wrote: > > Hi Stian, > > I completely agree on the first point, this deserves a better solution. > But knowing that it would require digging deep into the guts of KeyCloak, I > just didn't dare :) > > I would see providers somehow registering their custom enum keys, either > via API call or via callback, and KeyCloak would maintain the list of > custom values per provider. ServerInfoAdminResource then could return a > combined list of enum keys, both built-in and defined by providers. But I'm > afraid that would require changes to API and/or interfaces. What do you > think? How would you approach this problem yourself? > > > We have SPIs to make features pluggable so that's the way to go. We'd > probably want a AdminResourceTypeSpi and at the same time refactor the > current enum to use this. > > > I was thinking about SPIs, but to me the ResourceType use case seemed too > minor to deserve its own dedicated SPI. OTOH, it would look quite natural > as a part of more generic Admin Resource SPI. I remember it's been on the > list for a long time, then why not finally getting hands dirty? > > Please take a look at the BeerCloak example I've announced earlier in the > list. There, I've tried to collect all the techniques I've been using up to > the moment to develop ad-hoc admin resources. I think I've got enough > experience and understanding to do the whole Admin Resource SPI thing (with > hypothetical AdminResourceTypeSpi becoming a part of it, along with roles > handling, authorization, event logging etc.) That's it, I'd like to > volunteer to implement Admin Resource SPI, and to do it right way ("well > designed, generically usable and documented"). > > Please let me know what do you think. If you greenlight this, I think we'd > start a new thread to discuss the design. > > > > > > And BTW, what's wrong with the second PR? The github comment seems a bit > unclear, so could you please elaborate? It has completely different scope, > though deals with the same feature. It's not about exposing anything via > static methods, it's about making ResourceType extensible, following a > well-defined pattern. I was sure that Marek supported it, and I've > implemented it in accordance with his recommendations. > > > Seemed rather hacky to me, but maybe I'm mistaking something. Can you show > an example on how it would actually be used? I don't see how you'd add > additional types and how you'd actually use the resource type not knowing > what type it is. Seems all operations valid on an enum becomes invalid in > this case. > > > See beercloak.BeerResource::create and other methods. There are > commented-out lines that show how it could look like. But I don't think > it's much relevant now, since we may come up with completely different > design with Admin Resource SPI. > > Cheers, > Dmitry > From mposolda at redhat.com Thu Nov 3 07:06:09 2016 From: mposolda at redhat.com (Marek Posolda) Date: Thu, 3 Nov 2016 12:06:09 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches Message-ID: I was looking at the cache issue reported by customer. I found the cause of it and couple of other related issues: KEYCLOAK-3857 - Bad performance with clustered invalidation cache when updating object KEYCLOAK-3858 - Removing model object send lots of invalidation messages across cluster KEYCLOAK-3859 - Lots of userCache invalidation messages when invalidating realm KEYCLOAK-3860 - All realm users are invalidated from cache when removing some realm object In shortcut, our cache works fine in local mode. But in cluster, there are issues with the invalidation caches . We don't have issues with stale entries, but this is purchased but lots of various performance issues like: - There are lots of invalidation messages sent across the cluster - Eviction on the node, which received invalidation event, is also very uneffective. For example evicting realm with 1000 roles needs to call 1000 predicates, which iterates the cache 1000 times. - Invalidation cache doesn't allow to differ between the context why the object was invalidated. For example when I update realm settings on node1, I need to invalidate just the CachedRealm object, but not all the other objects dependent on the realm. However the invalidation event received on node2 doesn't know, if I invalidated CachedRealm because of realm update or because of realm removal. So for more safety, it assumes removal, which evicts all realm objects! See https://issues.jboss.org/browse/KEYCLOAK-3857 for details. - Finally we have the workaround with the "invalidation.key" objects in our invalidation caches. This is currently needed because when invalidating object on node1, the invalidation event is NOT received on node2 unless the object is there. Hence the workaround with the "invalidation.key" records just to avoid this limitation of invalidation cache. For solve all these issues, I propose: - Instead of relying on invalidation caches, we will send notification across cluster what happened (eg. message "realm XY was updated"). All the nodes will receive this notification and will evict all their locally cached objects accordingly and bump their revisions locally. This would be much more stable, performant and will allow us to remove some workarounds. Some details: - The caches "realms" and "users" won't be "invalidation" caches, but they will be "local" caches. - When any object needs to be removed from cache because of some reason (eg. updating realm), the notification message will be sent from node1 to all other cluster nodes. We will use the replicated cache for that. Node1 will send the notification message like "realm XY was updated" . - Other cluster nodes will receive this message and they will locally trigger evictions of all the objects dependent on particular realm. In case of realm update, it's just the CachedRealm object itself. In case of realm removal, it is all realm objects etc. - Note message will contain also context "realm XY was updated" or "realm XY was removed" . Not just "invalidate realm XY". This allows much more flexibility and in particular avoid the issues like https://issues.jboss.org/browse/KEYCLOAK-3857 . - We already have replicated cache called "work", which we are using to notify other cluster nodes about various events. So we will just use this one. No need to add another replicated cache, we will probably just need to configure LRU eviction for the existing one. - Also note that messages will be always received. We won't need workaround with "invalidation.key" objects anymore. - Also we don't need recursive evictions (which has very poor performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), because receiving node will know exactly what happened. It will remove objects just the same way like the "sender" node. - Finally the amount of traffic sent across the cluster will be much lower. This sounds like the big step, but IMO it's not that bad :) Note that we already have all the predicates in place for individual objects. The only change will be about sending/receiving notifications across cluster. I think I am able to prototype something by tomorrow to doublecheck this approach working and then finish it somewhen middle next week. WDYT? Marek From sthorger at redhat.com Thu Nov 3 09:09:47 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Thu, 3 Nov 2016 14:09:47 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: Sounds like a plan to me and I can't see us fixing it in a more trivial matter either. This also aligns better with what we've recently discussed with regards to cross DC support. Cross DC support would use JDG and have a replicate cache between DCs to send invalidation messages, which is exactly what you are proposing so this shoulod be a step towards that. It may make sense that you work that in straight away. Basically we should support propagating invalidation messages by using an external Infinispan/JDG cluster as the store for the cache. On 3 November 2016 at 12:06, Marek Posolda wrote: > I was looking at the cache issue reported by customer. I found the cause > of it and couple of other related issues: > > KEYCLOAK-3857 - Bad performance with clustered invalidation cache when > updating object > KEYCLOAK-3858 - Removing model object send lots of invalidation messages > across cluster > KEYCLOAK-3859 - Lots of userCache invalidation messages when > invalidating realm > KEYCLOAK-3860 - All realm users are invalidated from cache when removing > some realm object > > > In shortcut, our cache works fine in local mode. But in cluster, there > are issues with the invalidation caches . We don't have issues with > stale entries, but this is purchased but lots of various performance > issues like: > > - There are lots of invalidation messages sent across the cluster > > - Eviction on the node, which received invalidation event, is also very > uneffective. For example evicting realm with 1000 roles needs to call > 1000 predicates, which iterates the cache 1000 times. > > - Invalidation cache doesn't allow to differ between the context why the > object was invalidated. For example when I update realm settings on > node1, I need to invalidate just the CachedRealm object, but not all the > other objects dependent on the realm. However the invalidation event > received on node2 doesn't know, if I invalidated CachedRealm because of > realm update or because of realm removal. So for more safety, it assumes > removal, which evicts all realm objects! See > https://issues.jboss.org/browse/KEYCLOAK-3857 for details. > > - Finally we have the workaround with the "invalidation.key" objects in > our invalidation caches. This is currently needed because when > invalidating object on node1, the invalidation event is NOT received on > node2 unless the object is there. Hence the workaround with the > "invalidation.key" records just to avoid this limitation of invalidation > cache. > > > For solve all these issues, I propose: > - Instead of relying on invalidation caches, we will send notification > across cluster what happened (eg. message "realm XY was updated"). All > the nodes will receive this notification and will evict all their > locally cached objects accordingly and bump their revisions locally. > This would be much more stable, performant and will allow us to remove > some workarounds. > > Some details: > > - The caches "realms" and "users" won't be "invalidation" caches, but > they will be "local" caches. > > - When any object needs to be removed from cache because of some reason > (eg. updating realm), the notification message will be sent from node1 > to all other cluster nodes. We will use the replicated cache for that. > Node1 will send the notification message like "realm XY was updated" . > > - Other cluster nodes will receive this message and they will locally > trigger evictions of all the objects dependent on particular realm. In > case of realm update, it's just the CachedRealm object itself. In case > of realm removal, it is all realm objects etc. > > - Note message will contain also context "realm XY was updated" or > "realm XY was removed" . Not just "invalidate realm XY". This allows > much more flexibility and in particular avoid the issues like > https://issues.jboss.org/browse/KEYCLOAK-3857 . > > - We already have replicated cache called "work", which we are using to > notify other cluster nodes about various events. So we will just use > this one. No need to add another replicated cache, we will probably just > need to configure LRU eviction for the existing one. > > - Also note that messages will be always received. We won't need > workaround with "invalidation.key" objects anymore. > > - Also we don't need recursive evictions (which has very poor > performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), > because receiving node will know exactly what happened. It will remove > objects just the same way like the "sender" node. > > - Finally the amount of traffic sent across the cluster will be much lower. > > This sounds like the big step, but IMO it's not that bad :) Note that we > already have all the predicates in place for individual objects. The > only change will be about sending/receiving notifications across > cluster. I think I am able to prototype something by tomorrow to > doublecheck this approach working and then finish it somewhen middle > next week. WDYT? > > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From bburke at redhat.com Thu Nov 3 13:10:04 2016 From: bburke at redhat.com (Bill Burke) Date: Thu, 3 Nov 2016 13:10:04 -0400 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: +1 to this. We are already hacking Infinispan horribly and doing nasty work arounds to get the behavior we want. Some notes: * You have to be sure to invalidate indexes (email->user, username->user) correctly * We cache relationships in some scenarios, i.e. the list of clients in a realm. Your proposal makes things so much easier for caching relationships as we know the difference between a realm delete and a realm update and a client create, update, and removal * Cache is currently not interacted with at all for queries that return multiple users (getUser(), searchForUser, etc.). We invalidate the realm on these queries as we don't know if the caller is going to perform any updates on those users. * Role and group removal is a tricky thing as role/group mappings are cached with the user. You still have the iteration problem discusssed earlier and you have to decide which is more efficient, evicting all users in the realm, or iterating over all users that might have this role/group mapping. There's also client scope mapping, client mappers, and broker mappers which may also reference the role/group. * If a user storage provider is removed, this is also a tricky scenario and you may be better off evicting all users instead of iterating over all cached users to find all that were loaded by this provider. * Client removal is tricky because you also have to remove all user role mappings, scope mappings, that are cached. There may be other cases I'm missing. On 11/3/16 7:06 AM, Marek Posolda wrote: > I was looking at the cache issue reported by customer. I found the cause > of it and couple of other related issues: > > KEYCLOAK-3857 - Bad performance with clustered invalidation cache when > updating object > KEYCLOAK-3858 - Removing model object send lots of invalidation messages > across cluster > KEYCLOAK-3859 - Lots of userCache invalidation messages when > invalidating realm > KEYCLOAK-3860 - All realm users are invalidated from cache when removing > some realm object > > > In shortcut, our cache works fine in local mode. But in cluster, there > are issues with the invalidation caches . We don't have issues with > stale entries, but this is purchased but lots of various performance > issues like: > > - There are lots of invalidation messages sent across the cluster > > - Eviction on the node, which received invalidation event, is also very > uneffective. For example evicting realm with 1000 roles needs to call > 1000 predicates, which iterates the cache 1000 times. > > - Invalidation cache doesn't allow to differ between the context why the > object was invalidated. For example when I update realm settings on > node1, I need to invalidate just the CachedRealm object, but not all the > other objects dependent on the realm. However the invalidation event > received on node2 doesn't know, if I invalidated CachedRealm because of > realm update or because of realm removal. So for more safety, it assumes > removal, which evicts all realm objects! See > https://issues.jboss.org/browse/KEYCLOAK-3857 for details. > > - Finally we have the workaround with the "invalidation.key" objects in > our invalidation caches. This is currently needed because when > invalidating object on node1, the invalidation event is NOT received on > node2 unless the object is there. Hence the workaround with the > "invalidation.key" records just to avoid this limitation of invalidation > cache. > > > For solve all these issues, I propose: > - Instead of relying on invalidation caches, we will send notification > across cluster what happened (eg. message "realm XY was updated"). All > the nodes will receive this notification and will evict all their > locally cached objects accordingly and bump their revisions locally. > This would be much more stable, performant and will allow us to remove > some workarounds. > > Some details: > > - The caches "realms" and "users" won't be "invalidation" caches, but > they will be "local" caches. > > - When any object needs to be removed from cache because of some reason > (eg. updating realm), the notification message will be sent from node1 > to all other cluster nodes. We will use the replicated cache for that. > Node1 will send the notification message like "realm XY was updated" . > > - Other cluster nodes will receive this message and they will locally > trigger evictions of all the objects dependent on particular realm. In > case of realm update, it's just the CachedRealm object itself. In case > of realm removal, it is all realm objects etc. > > - Note message will contain also context "realm XY was updated" or > "realm XY was removed" . Not just "invalidate realm XY". This allows > much more flexibility and in particular avoid the issues like > https://issues.jboss.org/browse/KEYCLOAK-3857 . > > - We already have replicated cache called "work", which we are using to > notify other cluster nodes about various events. So we will just use > this one. No need to add another replicated cache, we will probably just > need to configure LRU eviction for the existing one. > > - Also note that messages will be always received. We won't need > workaround with "invalidation.key" objects anymore. > > - Also we don't need recursive evictions (which has very poor > performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), > because receiving node will know exactly what happened. It will remove > objects just the same way like the "sender" node. > > - Finally the amount of traffic sent across the cluster will be much lower. > > This sounds like the big step, but IMO it's not that bad :) Note that we > already have all the predicates in place for individual objects. The > only change will be about sending/receiving notifications across > cluster. I think I am able to prototype something by tomorrow to > doublecheck this approach working and then finish it somewhen middle > next week. WDYT? > > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Fri Nov 4 03:55:28 2016 From: mposolda at redhat.com (Marek Posolda) Date: Fri, 4 Nov 2016 08:55:28 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: Thanks for the tips. More inline On 03/11/16 18:10, Bill Burke wrote: > +1 to this. We are already hacking Infinispan horribly and doing nasty > work arounds to get the behavior we want. > > Some notes: > > * You have to be sure to invalidate indexes (email->user, > username->user) correctly > > * We cache relationships in some scenarios, i.e. the list of clients in > a realm. Your proposal makes things so much easier for caching > relationships as we know the difference between a realm delete and a > realm update and a client create, update, and removal > > * Cache is currently not interacted with at all for queries that return > multiple users (getUser(), searchForUser, etc.). We invalidate the > realm on these queries as we don't know if the caller is going to > perform any updates on those users. > > * Role and group removal is a tricky thing as role/group mappings are > cached with the user. You still have the iteration problem discusssed > earlier and you have to decide which is more efficient, evicting all > users in the realm, or iterating over all users that might have this > role/group mapping. There's also client scope mapping, client mappers, > and broker mappers which may also reference the role/group. Yep, right now we are just evicting all users when any role/client/group is removed. JIRA created already to improve it https://issues.jboss.org/browse/KEYCLOAK-3860 for better handling of that. Instead of evicting all users (which we have now) or iterating over all users, I was thinking about 3rd solution: Not evict any users straight away, but keep them in cache. So if role 123 was removed, users with this roleID will be still in cache. However when operation on user is called (eg. user.getRoleMappings ), then handle it in the UserAdapter itself and just ignore the removed role. For example user.getRoleMappings will be like: for (String roleId : roleIds) { RoleModel role = session.realms().getRole(roleId); if (role == null) { // Assume role 123 was deleted, so just ignore it continue; } else { ... } } Same for clients, where we need to handle consents, client role mappings and maybe more... Thing is, that there are some other issues with this solution like: - Potential risk of regressions (NPE exceptions etc) - Fact that we don't cache unsuccessful DB lookups right now. For example when I query DB for role "123" and DB returns null, we don't store anything in the cache ATM. So the information that "role 123 doesn't exist" is not in the cache. That's not very good for this solution as if we keep users with the outdated role mappings in the cache, there is chance that DB will be queried thousand of times for non-existing role "123" . So IMO we will need to somehow cache failed DB lookups too, if we want to go with this solution... Anyway, it's probably ok to postpone https://issues.jboss.org/browse/KEYCLOAK-3860 and do it later. It looks like fair amount of work with potential regressions and it's not directly related to clustering and invalidation issues seen by customer, so fine to be postponed if we don't have time to do it now. > > * If a user storage provider is removed, this is also a tricky scenario > and you may be better off evicting all users instead of iterating over > all cached users to find all that were loaded by this provider. +1, It seems that removing UserStorage provider will be very rare scenario? So evicting all users is probably safe. That's what we are doing already. > > * Client removal is tricky because you also have to remove all user role > mappings, scope mappings, that are cached. AFAIK we already have most of predicates done by you for invalidate all the "dependent" objects. However I will doublecheck... In particular, I am not sure if removing role also evicts broker mappers and protocolMappers right now... Marek > > There may be other cases I'm missing. > > > On 11/3/16 7:06 AM, Marek Posolda wrote: >> I was looking at the cache issue reported by customer. I found the cause >> of it and couple of other related issues: >> >> KEYCLOAK-3857 - Bad performance with clustered invalidation cache when >> updating object >> KEYCLOAK-3858 - Removing model object send lots of invalidation messages >> across cluster >> KEYCLOAK-3859 - Lots of userCache invalidation messages when >> invalidating realm >> KEYCLOAK-3860 - All realm users are invalidated from cache when removing >> some realm object >> >> >> In shortcut, our cache works fine in local mode. But in cluster, there >> are issues with the invalidation caches . We don't have issues with >> stale entries, but this is purchased but lots of various performance >> issues like: >> >> - There are lots of invalidation messages sent across the cluster >> >> - Eviction on the node, which received invalidation event, is also very >> uneffective. For example evicting realm with 1000 roles needs to call >> 1000 predicates, which iterates the cache 1000 times. >> >> - Invalidation cache doesn't allow to differ between the context why the >> object was invalidated. For example when I update realm settings on >> node1, I need to invalidate just the CachedRealm object, but not all the >> other objects dependent on the realm. However the invalidation event >> received on node2 doesn't know, if I invalidated CachedRealm because of >> realm update or because of realm removal. So for more safety, it assumes >> removal, which evicts all realm objects! See >> https://issues.jboss.org/browse/KEYCLOAK-3857 for details. >> >> - Finally we have the workaround with the "invalidation.key" objects in >> our invalidation caches. This is currently needed because when >> invalidating object on node1, the invalidation event is NOT received on >> node2 unless the object is there. Hence the workaround with the >> "invalidation.key" records just to avoid this limitation of invalidation >> cache. >> >> >> For solve all these issues, I propose: >> - Instead of relying on invalidation caches, we will send notification >> across cluster what happened (eg. message "realm XY was updated"). All >> the nodes will receive this notification and will evict all their >> locally cached objects accordingly and bump their revisions locally. >> This would be much more stable, performant and will allow us to remove >> some workarounds. >> >> Some details: >> >> - The caches "realms" and "users" won't be "invalidation" caches, but >> they will be "local" caches. >> >> - When any object needs to be removed from cache because of some reason >> (eg. updating realm), the notification message will be sent from node1 >> to all other cluster nodes. We will use the replicated cache for that. >> Node1 will send the notification message like "realm XY was updated" . >> >> - Other cluster nodes will receive this message and they will locally >> trigger evictions of all the objects dependent on particular realm. In >> case of realm update, it's just the CachedRealm object itself. In case >> of realm removal, it is all realm objects etc. >> >> - Note message will contain also context "realm XY was updated" or >> "realm XY was removed" . Not just "invalidate realm XY". This allows >> much more flexibility and in particular avoid the issues like >> https://issues.jboss.org/browse/KEYCLOAK-3857 . >> >> - We already have replicated cache called "work", which we are using to >> notify other cluster nodes about various events. So we will just use >> this one. No need to add another replicated cache, we will probably just >> need to configure LRU eviction for the existing one. >> >> - Also note that messages will be always received. We won't need >> workaround with "invalidation.key" objects anymore. >> >> - Also we don't need recursive evictions (which has very poor >> performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), >> because receiving node will know exactly what happened. It will remove >> objects just the same way like the "sender" node. >> >> - Finally the amount of traffic sent across the cluster will be much lower. >> >> This sounds like the big step, but IMO it's not that bad :) Note that we >> already have all the predicates in place for individual objects. The >> only change will be about sending/receiving notifications across >> cluster. I think I am able to prototype something by tomorrow to >> doublecheck this approach working and then finish it somewhen middle >> next week. WDYT? >> >> Marek >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Fri Nov 4 04:02:24 2016 From: mposolda at redhat.com (Marek Posolda) Date: Fri, 4 Nov 2016 09:02:24 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: On 03/11/16 14:09, Stian Thorgersen wrote: > Sounds like a plan to me and I can't see us fixing it in a more > trivial matter either. > > This also aligns better with what we've recently discussed with > regards to cross DC support. Cross DC support would use JDG and have a > replicate cache between DCs to send invalidation messages, which is > exactly what you are proposing so this shoulod be a step towards that. > It may make sense that you work that in straight away. Basically we > should support propagating invalidation messages by using an external > Infinispan/JDG cluster as the store for the cache. So it seems, we may need something like InvalidationManager SPI, as an abstraction just for the actual transport (sending/receiving invalidation messages) ? By default, it will just be replicated infinispan cache, but it will be possible to use something completely different for actual message transport (JMS messaging, external JDG for cross DC support etc). WDYT? Marek > > On 3 November 2016 at 12:06, Marek Posolda > wrote: > > I was looking at the cache issue reported by customer. I found the > cause > of it and couple of other related issues: > > KEYCLOAK-3857 - Bad performance with clustered invalidation cache when > updating object > KEYCLOAK-3858 - Removing model object send lots of invalidation > messages > across cluster > KEYCLOAK-3859 - Lots of userCache invalidation messages when > invalidating realm > KEYCLOAK-3860 - All realm users are invalidated from cache when > removing > some realm object > > > In shortcut, our cache works fine in local mode. But in cluster, there > are issues with the invalidation caches . We don't have issues with > stale entries, but this is purchased but lots of various performance > issues like: > > - There are lots of invalidation messages sent across the cluster > > - Eviction on the node, which received invalidation event, is also > very > uneffective. For example evicting realm with 1000 roles needs to call > 1000 predicates, which iterates the cache 1000 times. > > - Invalidation cache doesn't allow to differ between the context > why the > object was invalidated. For example when I update realm settings on > node1, I need to invalidate just the CachedRealm object, but not > all the > other objects dependent on the realm. However the invalidation event > received on node2 doesn't know, if I invalidated CachedRealm > because of > realm update or because of realm removal. So for more safety, it > assumes > removal, which evicts all realm objects! See > https://issues.jboss.org/browse/KEYCLOAK-3857 > for details. > > - Finally we have the workaround with the "invalidation.key" > objects in > our invalidation caches. This is currently needed because when > invalidating object on node1, the invalidation event is NOT > received on > node2 unless the object is there. Hence the workaround with the > "invalidation.key" records just to avoid this limitation of > invalidation > cache. > > > For solve all these issues, I propose: > - Instead of relying on invalidation caches, we will send notification > across cluster what happened (eg. message "realm XY was updated"). All > the nodes will receive this notification and will evict all their > locally cached objects accordingly and bump their revisions locally. > This would be much more stable, performant and will allow us to remove > some workarounds. > > Some details: > > - The caches "realms" and "users" won't be "invalidation" caches, but > they will be "local" caches. > > - When any object needs to be removed from cache because of some > reason > (eg. updating realm), the notification message will be sent from node1 > to all other cluster nodes. We will use the replicated cache for that. > Node1 will send the notification message like "realm XY was updated" . > > - Other cluster nodes will receive this message and they will locally > trigger evictions of all the objects dependent on particular realm. In > case of realm update, it's just the CachedRealm object itself. In case > of realm removal, it is all realm objects etc. > > - Note message will contain also context "realm XY was updated" or > "realm XY was removed" . Not just "invalidate realm XY". This allows > much more flexibility and in particular avoid the issues like > https://issues.jboss.org/browse/KEYCLOAK-3857 > . > > - We already have replicated cache called "work", which we are > using to > notify other cluster nodes about various events. So we will just use > this one. No need to add another replicated cache, we will > probably just > need to configure LRU eviction for the existing one. > > - Also note that messages will be always received. We won't need > workaround with "invalidation.key" objects anymore. > > - Also we don't need recursive evictions (which has very poor > performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 > ), > because receiving node will know exactly what happened. It will remove > objects just the same way like the "sender" node. > > - Finally the amount of traffic sent across the cluster will be > much lower. > > This sounds like the big step, but IMO it's not that bad :) Note > that we > already have all the predicates in place for individual objects. The > only change will be about sending/receiving notifications across > cluster. I think I am able to prototype something by tomorrow to > doublecheck this approach working and then finish it somewhen middle > next week. WDYT? > > Marek > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > From thomas.darimont at googlemail.com Fri Nov 4 11:06:07 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Fri, 4 Nov 2016 16:06:07 +0100 Subject: [keycloak-dev] Custom UserStorageProvider and TOTP Credentials Message-ID: Hello, I have a custom UserStorageProvider as a replacement for the deprecated UserFederationProvider mechanism which connects users from a legacy user store with Keycloak. Since the legacy user store will remain the leading system for a while. I want to prohibit users from changing passwords and OTP secrets via the account page. For password changes this can be easily done by throwing an UnsupportedOperationException in the implementation of the org.keycloak.credential.CredentialInputUpdater. updateCredential(RealmModel, UserModel, CredentialInput) method for the UserCredentialModel.PASSWORD. If a user tries to change a password he/she sees an error message that this is not supported for their accounts. However prohibiting users from configuring a TOTP token cannot be done this way, since thowing an exception results in a stacktrace presented to the user. (stacktrace below) I think the reason for this is that the AccountService.processTotpUpdate method: org.keycloak.services.resources.AccountService.processTotpUpdate( MultivaluedMap) is missing some error handling: See: https://github.com/keycloak/keycloak /blob/ccaac408630d83b6d66b041a2ef4c23102f0ed60/services/src/main/java/org/ keycloak/services/resources/AccountService.java#L563 In AccountService.processPasswordUpdate the error handling seems to be fine. org.keycloak.services.resources.AccountService.processPasswordUpdate( MultivaluedMap) see: https://github.com/keycloak/keycloak /blob/ccaac408630d83b6d66b041a2ef4c23102f0ed60/services/src/main/java/org/ keycloak/services/resources/AccountService.java#L641 Could the error handling (try / catch) of AccountService. processPasswordUpdate also be applied to AccountService.processTotpUpdate? This would ease the control of what credentials a "federated" user can change. Bottom line it should be possible for a UserStorageProvider to only validate credentials (Password, OTP, etc.) ( org.keycloak.credential. CredentialInputUpdater) but prohibit updating them. Currently the login in org.keycloak.credential.UserCredentialStoreManager. updateCredential(RealmModel, UserModel, CredentialInput) falls back to updating the credentials anyway if the CredentialInputUpdater. updateCredential(..) Method implemented by the custom UserStorageProvider does NOT handle the credential update by returning false in CredentialInputUpdater.updateCredential(..) Cheers, Thomas Error processing request Context Path:/auth Servlet Path: Path Info:/realms/user-storage-provider-dev/account/totp Query String:null Stack Trace org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: updateCredential(realm,user,credentialInput) not supported org.jboss.resteasy.core.ExceptionHandler.handleApplicationException( ExceptionHandler.java:76) org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler .java:212) org.jboss.resteasy.core.SynchronousDispatcher.writeException( SynchronousDispatcher.java:168) org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher .java:411) org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher .java:202) org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher .service(ServletContainerDispatcher.java:221) org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service( HttpServletDispatcher.java:56) org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service( HttpServletDispatcher.java:51) javax.servlet.http.HttpServlet.service(HttpServlet.java:847) io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler .java:85) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter( FilterHandler.java:129) org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter( KeycloakSessionServletFilter.java:90) io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter( FilterHandler.java:131) io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler .java:84) io.undertow.servlet.handlers.security.ServletSecurityRoleHandler. handleRequest(ServletSecurityRoleHandler.java:62) io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest( ServletDispatchingHandler.java:36) io.undertow.servlet.handlers.security.SSLInformationAssociationHandler. handleRequest(SSLInformationAssociationHandler.java:131) io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler. handleRequest(ServletAuthenticationCallHandler.java:57) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler .java:43) io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest( AbstractConfidentialityHandler.java:46) io.undertow.servlet.handlers.security. ServletConfidentialityConstraintHandler.handleRequest( ServletConfidentialityConstraintHandler.java:64) io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest( AuthenticationMechanismsHandler.java:60) io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler. handleRequest(CachedAuthenticatedSessionHandler.java:77) io.undertow.security.handlers.AbstractSecurityContextAssociationHandler. handleRequest(AbstractSecurityContextAssociationHandler.java:43) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler .java:43) io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler .java:43) io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest( ServletInitialHandler.java:284) io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( ServletInitialHandler.java:263) io.undertow.servlet.handlers.ServletInitialHandler.access$000( ServletInitialHandler.java:81) io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest( ServletInitialHandler.java:174) io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor .java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:617) java.lang.Thread.run(Thread.java:745) From bburke at redhat.com Fri Nov 4 20:16:26 2016 From: bburke at redhat.com (Bill Burke) Date: Fri, 4 Nov 2016 20:16:26 -0400 Subject: [keycloak-dev] evict on update/remove, make providers cache aware Message-ID: <9005fa16-4d9f-e2b4-c48e-511ecd03f005@redhat.com> I was thinking that we change all Mongo and JPA adapters, and RealmProvider and UserProvider methods that can update or remove models from the database and have them evict the cache of that type. This will make sure that if somebody gets a non-cached model or provider and performs an update, the cache is still invalidated. Sound ok? Thanks, Bill From bburke at redhat.com Sat Nov 5 10:11:05 2016 From: bburke at redhat.com (Bill Burke) Date: Sat, 5 Nov 2016 10:11:05 -0400 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: <5a779b44-6759-e6e8-871b-ec10141922ef@redhat.com> We don't currently evict the user cache if a realm has changed. This is an issue if somebody has modified a UserStorageProvider. Should we evict user cache for any realm changes? Or should I modify the cache RealmAdapter and detect whether a UserStorageProvider is being modified (or any of its subcomponents(mappers)).? On 11/3/16 1:10 PM, Bill Burke wrote: > +1 to this. We are already hacking Infinispan horribly and doing nasty > work arounds to get the behavior we want. > > Some notes: > > * You have to be sure to invalidate indexes (email->user, > username->user) correctly > > * We cache relationships in some scenarios, i.e. the list of clients in > a realm. Your proposal makes things so much easier for caching > relationships as we know the difference between a realm delete and a > realm update and a client create, update, and removal > > * Cache is currently not interacted with at all for queries that return > multiple users (getUser(), searchForUser, etc.). We invalidate the > realm on these queries as we don't know if the caller is going to > perform any updates on those users. > > * Role and group removal is a tricky thing as role/group mappings are > cached with the user. You still have the iteration problem discusssed > earlier and you have to decide which is more efficient, evicting all > users in the realm, or iterating over all users that might have this > role/group mapping. There's also client scope mapping, client mappers, > and broker mappers which may also reference the role/group. > > * If a user storage provider is removed, this is also a tricky scenario > and you may be better off evicting all users instead of iterating over > all cached users to find all that were loaded by this provider. > > * Client removal is tricky because you also have to remove all user role > mappings, scope mappings, that are cached. > > There may be other cases I'm missing. > > > On 11/3/16 7:06 AM, Marek Posolda wrote: >> I was looking at the cache issue reported by customer. I found the cause >> of it and couple of other related issues: >> >> KEYCLOAK-3857 - Bad performance with clustered invalidation cache when >> updating object >> KEYCLOAK-3858 - Removing model object send lots of invalidation messages >> across cluster >> KEYCLOAK-3859 - Lots of userCache invalidation messages when >> invalidating realm >> KEYCLOAK-3860 - All realm users are invalidated from cache when removing >> some realm object >> >> >> In shortcut, our cache works fine in local mode. But in cluster, there >> are issues with the invalidation caches . We don't have issues with >> stale entries, but this is purchased but lots of various performance >> issues like: >> >> - There are lots of invalidation messages sent across the cluster >> >> - Eviction on the node, which received invalidation event, is also very >> uneffective. For example evicting realm with 1000 roles needs to call >> 1000 predicates, which iterates the cache 1000 times. >> >> - Invalidation cache doesn't allow to differ between the context why the >> object was invalidated. For example when I update realm settings on >> node1, I need to invalidate just the CachedRealm object, but not all the >> other objects dependent on the realm. However the invalidation event >> received on node2 doesn't know, if I invalidated CachedRealm because of >> realm update or because of realm removal. So for more safety, it assumes >> removal, which evicts all realm objects! See >> https://issues.jboss.org/browse/KEYCLOAK-3857 for details. >> >> - Finally we have the workaround with the "invalidation.key" objects in >> our invalidation caches. This is currently needed because when >> invalidating object on node1, the invalidation event is NOT received on >> node2 unless the object is there. Hence the workaround with the >> "invalidation.key" records just to avoid this limitation of invalidation >> cache. >> >> >> For solve all these issues, I propose: >> - Instead of relying on invalidation caches, we will send notification >> across cluster what happened (eg. message "realm XY was updated"). All >> the nodes will receive this notification and will evict all their >> locally cached objects accordingly and bump their revisions locally. >> This would be much more stable, performant and will allow us to remove >> some workarounds. >> >> Some details: >> >> - The caches "realms" and "users" won't be "invalidation" caches, but >> they will be "local" caches. >> >> - When any object needs to be removed from cache because of some reason >> (eg. updating realm), the notification message will be sent from node1 >> to all other cluster nodes. We will use the replicated cache for that. >> Node1 will send the notification message like "realm XY was updated" . >> >> - Other cluster nodes will receive this message and they will locally >> trigger evictions of all the objects dependent on particular realm. In >> case of realm update, it's just the CachedRealm object itself. In case >> of realm removal, it is all realm objects etc. >> >> - Note message will contain also context "realm XY was updated" or >> "realm XY was removed" . Not just "invalidate realm XY". This allows >> much more flexibility and in particular avoid the issues like >> https://issues.jboss.org/browse/KEYCLOAK-3857 . >> >> - We already have replicated cache called "work", which we are using to >> notify other cluster nodes about various events. So we will just use >> this one. No need to add another replicated cache, we will probably just >> need to configure LRU eviction for the existing one. >> >> - Also note that messages will be always received. We won't need >> workaround with "invalidation.key" objects anymore. >> >> - Also we don't need recursive evictions (which has very poor >> performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), >> because receiving node will know exactly what happened. It will remove >> objects just the same way like the "sender" node. >> >> - Finally the amount of traffic sent across the cluster will be much lower. >> >> This sounds like the big step, but IMO it's not that bad :) Note that we >> already have all the predicates in place for individual objects. The >> only change will be about sending/receiving notifications across >> cluster. I think I am able to prototype something by tomorrow to >> doublecheck this approach working and then finish it somewhen middle >> next week. WDYT? >> >> Marek >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Sun Nov 6 13:25:21 2016 From: mposolda at redhat.com (Marek Posolda) Date: Sun, 6 Nov 2016 19:25:21 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: <5a779b44-6759-e6e8-871b-ec10141922ef@redhat.com> References: <5a779b44-6759-e6e8-871b-ec10141922ef@redhat.com> Message-ID: +1 to evict the user cache just for the UserStorageProvider updates (or it's subcomponents). Not evict for every realm update as that sounds like the unecesary performance penalty. Marek On 05/11/16 15:11, Bill Burke wrote: > We don't currently evict the user cache if a realm has changed. This is > an issue if somebody has modified a UserStorageProvider. Should we evict > user cache for any realm changes? Or should I modify the cache > RealmAdapter and detect whether a UserStorageProvider is being modified > (or any of its subcomponents(mappers)).? > > > On 11/3/16 1:10 PM, Bill Burke wrote: >> +1 to this. We are already hacking Infinispan horribly and doing nasty >> work arounds to get the behavior we want. >> >> Some notes: >> >> * You have to be sure to invalidate indexes (email->user, >> username->user) correctly >> >> * We cache relationships in some scenarios, i.e. the list of clients in >> a realm. Your proposal makes things so much easier for caching >> relationships as we know the difference between a realm delete and a >> realm update and a client create, update, and removal >> >> * Cache is currently not interacted with at all for queries that return >> multiple users (getUser(), searchForUser, etc.). We invalidate the >> realm on these queries as we don't know if the caller is going to >> perform any updates on those users. >> >> * Role and group removal is a tricky thing as role/group mappings are >> cached with the user. You still have the iteration problem discusssed >> earlier and you have to decide which is more efficient, evicting all >> users in the realm, or iterating over all users that might have this >> role/group mapping. There's also client scope mapping, client mappers, >> and broker mappers which may also reference the role/group. >> >> * If a user storage provider is removed, this is also a tricky scenario >> and you may be better off evicting all users instead of iterating over >> all cached users to find all that were loaded by this provider. >> >> * Client removal is tricky because you also have to remove all user role >> mappings, scope mappings, that are cached. >> >> There may be other cases I'm missing. >> >> >> On 11/3/16 7:06 AM, Marek Posolda wrote: >>> I was looking at the cache issue reported by customer. I found the cause >>> of it and couple of other related issues: >>> >>> KEYCLOAK-3857 - Bad performance with clustered invalidation cache when >>> updating object >>> KEYCLOAK-3858 - Removing model object send lots of invalidation messages >>> across cluster >>> KEYCLOAK-3859 - Lots of userCache invalidation messages when >>> invalidating realm >>> KEYCLOAK-3860 - All realm users are invalidated from cache when removing >>> some realm object >>> >>> >>> In shortcut, our cache works fine in local mode. But in cluster, there >>> are issues with the invalidation caches . We don't have issues with >>> stale entries, but this is purchased but lots of various performance >>> issues like: >>> >>> - There are lots of invalidation messages sent across the cluster >>> >>> - Eviction on the node, which received invalidation event, is also very >>> uneffective. For example evicting realm with 1000 roles needs to call >>> 1000 predicates, which iterates the cache 1000 times. >>> >>> - Invalidation cache doesn't allow to differ between the context why the >>> object was invalidated. For example when I update realm settings on >>> node1, I need to invalidate just the CachedRealm object, but not all the >>> other objects dependent on the realm. However the invalidation event >>> received on node2 doesn't know, if I invalidated CachedRealm because of >>> realm update or because of realm removal. So for more safety, it assumes >>> removal, which evicts all realm objects! See >>> https://issues.jboss.org/browse/KEYCLOAK-3857 for details. >>> >>> - Finally we have the workaround with the "invalidation.key" objects in >>> our invalidation caches. This is currently needed because when >>> invalidating object on node1, the invalidation event is NOT received on >>> node2 unless the object is there. Hence the workaround with the >>> "invalidation.key" records just to avoid this limitation of invalidation >>> cache. >>> >>> >>> For solve all these issues, I propose: >>> - Instead of relying on invalidation caches, we will send notification >>> across cluster what happened (eg. message "realm XY was updated"). All >>> the nodes will receive this notification and will evict all their >>> locally cached objects accordingly and bump their revisions locally. >>> This would be much more stable, performant and will allow us to remove >>> some workarounds. >>> >>> Some details: >>> >>> - The caches "realms" and "users" won't be "invalidation" caches, but >>> they will be "local" caches. >>> >>> - When any object needs to be removed from cache because of some reason >>> (eg. updating realm), the notification message will be sent from node1 >>> to all other cluster nodes. We will use the replicated cache for that. >>> Node1 will send the notification message like "realm XY was updated" . >>> >>> - Other cluster nodes will receive this message and they will locally >>> trigger evictions of all the objects dependent on particular realm. In >>> case of realm update, it's just the CachedRealm object itself. In case >>> of realm removal, it is all realm objects etc. >>> >>> - Note message will contain also context "realm XY was updated" or >>> "realm XY was removed" . Not just "invalidate realm XY". This allows >>> much more flexibility and in particular avoid the issues like >>> https://issues.jboss.org/browse/KEYCLOAK-3857 . >>> >>> - We already have replicated cache called "work", which we are using to >>> notify other cluster nodes about various events. So we will just use >>> this one. No need to add another replicated cache, we will probably just >>> need to configure LRU eviction for the existing one. >>> >>> - Also note that messages will be always received. We won't need >>> workaround with "invalidation.key" objects anymore. >>> >>> - Also we don't need recursive evictions (which has very poor >>> performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), >>> because receiving node will know exactly what happened. It will remove >>> objects just the same way like the "sender" node. >>> >>> - Finally the amount of traffic sent across the cluster will be much lower. >>> >>> This sounds like the big step, but IMO it's not that bad :) Note that we >>> already have all the predicates in place for individual objects. The >>> only change will be about sending/receiving notifications across >>> cluster. I think I am able to prototype something by tomorrow to >>> doublecheck this approach working and then finish it somewhen middle >>> next week. WDYT? >>> >>> Marek >>> >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Sun Nov 6 13:47:29 2016 From: mposolda at redhat.com (Marek Posolda) Date: Sun, 6 Nov 2016 19:47:29 +0100 Subject: [keycloak-dev] evict on update/remove, make providers cache aware In-Reply-To: <9005fa16-4d9f-e2b4-c48e-511ecd03f005@redhat.com> References: <9005fa16-4d9f-e2b4-c48e-511ecd03f005@redhat.com> Message-ID: <0f7d7aa0-8075-0266-cdbc-d3485979ca2f@redhat.com> On 05/11/16 01:16, Bill Burke wrote: > I was thinking that we change all Mongo and JPA adapters, and > RealmProvider and UserProvider methods that can update or remove models > from the database and have them evict the cache of that type. This will > make sure that if somebody gets a non-cached model or provider and > performs an update, the cache is still invalidated. Sound ok? Not sure if that's good option to add this logic directly to Mongo or JPA providers? Mongo and JPA will contain some cache related logic, which is not the responsibility of these providers IMO. IMO we can rather: - Ensure that JPA/Mongo adapter is never returned to the caller, if the particular object is in cache. This may be an issue for some search operations (as searches are sometimes not supported by cache implementation), but then we can ensure that cache provider will return cached adapters by manually iterating over search results and make sure that cached adapter is returned. For example like we do for RealmCacheSession.getRealms() : https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/RealmCacheSession.java#L442-L452 - Ensure that JPA/Mongo provider will return some decorated proxy object, which will decorate over JPA/Mongo adapter object and it will evict if set/add/update/remove method is called on this proxy? Hope we can somehow handle it dynamically with dynamic proxies / InvocationHandler like stuff. Marek > Thanks, > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From sthorger at redhat.com Mon Nov 7 00:14:25 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Mon, 7 Nov 2016 06:14:25 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: <5a779b44-6759-e6e8-871b-ec10141922ef@redhat.com> Message-ID: +1 To not evict all users on all realm updates. Realm updates will be a fairly frequent thing. Roles, clients, etc.. So would be good to not evict when those things are changed. On 6 November 2016 at 19:25, Marek Posolda wrote: > +1 to evict the user cache just for the UserStorageProvider updates (or > it's subcomponents). Not evict for every realm update as that sounds > like the unecesary performance penalty. > > Marek > > > On 05/11/16 15:11, Bill Burke wrote: > > We don't currently evict the user cache if a realm has changed. This is > > an issue if somebody has modified a UserStorageProvider. Should we evict > > user cache for any realm changes? Or should I modify the cache > > RealmAdapter and detect whether a UserStorageProvider is being modified > > (or any of its subcomponents(mappers)).? > > > > > > On 11/3/16 1:10 PM, Bill Burke wrote: > >> +1 to this. We are already hacking Infinispan horribly and doing nasty > >> work arounds to get the behavior we want. > >> > >> Some notes: > >> > >> * You have to be sure to invalidate indexes (email->user, > >> username->user) correctly > >> > >> * We cache relationships in some scenarios, i.e. the list of clients in > >> a realm. Your proposal makes things so much easier for caching > >> relationships as we know the difference between a realm delete and a > >> realm update and a client create, update, and removal > >> > >> * Cache is currently not interacted with at all for queries that return > >> multiple users (getUser(), searchForUser, etc.). We invalidate the > >> realm on these queries as we don't know if the caller is going to > >> perform any updates on those users. > >> > >> * Role and group removal is a tricky thing as role/group mappings are > >> cached with the user. You still have the iteration problem discusssed > >> earlier and you have to decide which is more efficient, evicting all > >> users in the realm, or iterating over all users that might have this > >> role/group mapping. There's also client scope mapping, client mappers, > >> and broker mappers which may also reference the role/group. > >> > >> * If a user storage provider is removed, this is also a tricky scenario > >> and you may be better off evicting all users instead of iterating over > >> all cached users to find all that were loaded by this provider. > >> > >> * Client removal is tricky because you also have to remove all user role > >> mappings, scope mappings, that are cached. > >> > >> There may be other cases I'm missing. > >> > >> > >> On 11/3/16 7:06 AM, Marek Posolda wrote: > >>> I was looking at the cache issue reported by customer. I found the > cause > >>> of it and couple of other related issues: > >>> > >>> KEYCLOAK-3857 - Bad performance with clustered invalidation cache when > >>> updating object > >>> KEYCLOAK-3858 - Removing model object send lots of invalidation > messages > >>> across cluster > >>> KEYCLOAK-3859 - Lots of userCache invalidation messages when > >>> invalidating realm > >>> KEYCLOAK-3860 - All realm users are invalidated from cache when > removing > >>> some realm object > >>> > >>> > >>> In shortcut, our cache works fine in local mode. But in cluster, there > >>> are issues with the invalidation caches . We don't have issues with > >>> stale entries, but this is purchased but lots of various performance > >>> issues like: > >>> > >>> - There are lots of invalidation messages sent across the cluster > >>> > >>> - Eviction on the node, which received invalidation event, is also very > >>> uneffective. For example evicting realm with 1000 roles needs to call > >>> 1000 predicates, which iterates the cache 1000 times. > >>> > >>> - Invalidation cache doesn't allow to differ between the context why > the > >>> object was invalidated. For example when I update realm settings on > >>> node1, I need to invalidate just the CachedRealm object, but not all > the > >>> other objects dependent on the realm. However the invalidation event > >>> received on node2 doesn't know, if I invalidated CachedRealm because of > >>> realm update or because of realm removal. So for more safety, it > assumes > >>> removal, which evicts all realm objects! See > >>> https://issues.jboss.org/browse/KEYCLOAK-3857 for details. > >>> > >>> - Finally we have the workaround with the "invalidation.key" objects in > >>> our invalidation caches. This is currently needed because when > >>> invalidating object on node1, the invalidation event is NOT received on > >>> node2 unless the object is there. Hence the workaround with the > >>> "invalidation.key" records just to avoid this limitation of > invalidation > >>> cache. > >>> > >>> > >>> For solve all these issues, I propose: > >>> - Instead of relying on invalidation caches, we will send notification > >>> across cluster what happened (eg. message "realm XY was updated"). All > >>> the nodes will receive this notification and will evict all their > >>> locally cached objects accordingly and bump their revisions locally. > >>> This would be much more stable, performant and will allow us to remove > >>> some workarounds. > >>> > >>> Some details: > >>> > >>> - The caches "realms" and "users" won't be "invalidation" caches, but > >>> they will be "local" caches. > >>> > >>> - When any object needs to be removed from cache because of some reason > >>> (eg. updating realm), the notification message will be sent from node1 > >>> to all other cluster nodes. We will use the replicated cache for that. > >>> Node1 will send the notification message like "realm XY was updated" . > >>> > >>> - Other cluster nodes will receive this message and they will locally > >>> trigger evictions of all the objects dependent on particular realm. In > >>> case of realm update, it's just the CachedRealm object itself. In case > >>> of realm removal, it is all realm objects etc. > >>> > >>> - Note message will contain also context "realm XY was updated" or > >>> "realm XY was removed" . Not just "invalidate realm XY". This allows > >>> much more flexibility and in particular avoid the issues like > >>> https://issues.jboss.org/browse/KEYCLOAK-3857 . > >>> > >>> - We already have replicated cache called "work", which we are using to > >>> notify other cluster nodes about various events. So we will just use > >>> this one. No need to add another replicated cache, we will probably > just > >>> need to configure LRU eviction for the existing one. > >>> > >>> - Also note that messages will be always received. We won't need > >>> workaround with "invalidation.key" objects anymore. > >>> > >>> - Also we don't need recursive evictions (which has very poor > >>> performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), > >>> because receiving node will know exactly what happened. It will remove > >>> objects just the same way like the "sender" node. > >>> > >>> - Finally the amount of traffic sent across the cluster will be much > lower. > >>> > >>> This sounds like the big step, but IMO it's not that bad :) Note that > we > >>> already have all the predicates in place for individual objects. The > >>> only change will be about sending/receiving notifications across > >>> cluster. I think I am able to prototype something by tomorrow to > >>> doublecheck this approach working and then finish it somewhen middle > >>> next week. WDYT? > >>> > >>> Marek > >>> > >>> _______________________________________________ > >>> keycloak-dev mailing list > >>> keycloak-dev at lists.jboss.org > >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev > >> _______________________________________________ > >> keycloak-dev mailing list > >> keycloak-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From sthorger at redhat.com Mon Nov 7 00:19:29 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Mon, 7 Nov 2016 06:19:29 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: The idea was to have an eviction cache like you are proposing that has a JDG instances as the store to propagate evictions cross-DCs. From my understanding the only difference is that we would have to register a listener to the JDG cache. Not sure if it needs an SPI or if we can just have a single config option. JMS is not a good option with regards to usability in either case. In a single cluster it wouldn't give much benefit as we'd need Infinispan in either case. In multi DC it would be complex to setup and we'd still need JDG for sessions. So I think an SPI might be overkill and we should just rather do it with a config switch. Or better yet it detects when a JDG session store is used and automatically sets the listener. On 4 November 2016 at 09:02, Marek Posolda wrote: > On 03/11/16 14:09, Stian Thorgersen wrote: > > Sounds like a plan to me and I can't see us fixing it in a more trivial > matter either. > > This also aligns better with what we've recently discussed with regards to > cross DC support. Cross DC support would use JDG and have a replicate cache > between DCs to send invalidation messages, which is exactly what you are > proposing so this shoulod be a step towards that. It may make sense that > you work that in straight away. Basically we should support propagating > invalidation messages by using an external Infinispan/JDG cluster as the > store for the cache. > > So it seems, we may need something like InvalidationManager SPI, as an > abstraction just for the actual transport (sending/receiving invalidation > messages) ? > > By default, it will just be replicated infinispan cache, but it will be > possible to use something completely different for actual message transport > (JMS messaging, external JDG for cross DC support etc). WDYT? > > Marek > > > On 3 November 2016 at 12:06, Marek Posolda wrote: > >> I was looking at the cache issue reported by customer. I found the cause >> of it and couple of other related issues: >> >> KEYCLOAK-3857 - Bad performance with clustered invalidation cache when >> updating object >> KEYCLOAK-3858 - Removing model object send lots of invalidation messages >> across cluster >> KEYCLOAK-3859 - Lots of userCache invalidation messages when >> invalidating realm >> KEYCLOAK-3860 - All realm users are invalidated from cache when removing >> some realm object >> >> >> In shortcut, our cache works fine in local mode. But in cluster, there >> are issues with the invalidation caches . We don't have issues with >> stale entries, but this is purchased but lots of various performance >> issues like: >> >> - There are lots of invalidation messages sent across the cluster >> >> - Eviction on the node, which received invalidation event, is also very >> uneffective. For example evicting realm with 1000 roles needs to call >> 1000 predicates, which iterates the cache 1000 times. >> >> - Invalidation cache doesn't allow to differ between the context why the >> object was invalidated. For example when I update realm settings on >> node1, I need to invalidate just the CachedRealm object, but not all the >> other objects dependent on the realm. However the invalidation event >> received on node2 doesn't know, if I invalidated CachedRealm because of >> realm update or because of realm removal. So for more safety, it assumes >> removal, which evicts all realm objects! See >> https://issues.jboss.org/browse/KEYCLOAK-3857 for details. >> >> - Finally we have the workaround with the "invalidation.key" objects in >> our invalidation caches. This is currently needed because when >> invalidating object on node1, the invalidation event is NOT received on >> node2 unless the object is there. Hence the workaround with the >> "invalidation.key" records just to avoid this limitation of invalidation >> cache. >> >> >> For solve all these issues, I propose: >> - Instead of relying on invalidation caches, we will send notification >> across cluster what happened (eg. message "realm XY was updated"). All >> the nodes will receive this notification and will evict all their >> locally cached objects accordingly and bump their revisions locally. >> This would be much more stable, performant and will allow us to remove >> some workarounds. >> >> Some details: >> >> - The caches "realms" and "users" won't be "invalidation" caches, but >> they will be "local" caches. >> >> - When any object needs to be removed from cache because of some reason >> (eg. updating realm), the notification message will be sent from node1 >> to all other cluster nodes. We will use the replicated cache for that. >> Node1 will send the notification message like "realm XY was updated" . >> >> - Other cluster nodes will receive this message and they will locally >> trigger evictions of all the objects dependent on particular realm. In >> case of realm update, it's just the CachedRealm object itself. In case >> of realm removal, it is all realm objects etc. >> >> - Note message will contain also context "realm XY was updated" or >> "realm XY was removed" . Not just "invalidate realm XY". This allows >> much more flexibility and in particular avoid the issues like >> https://issues.jboss.org/browse/KEYCLOAK-3857 . >> >> - We already have replicated cache called "work", which we are using to >> notify other cluster nodes about various events. So we will just use >> this one. No need to add another replicated cache, we will probably just >> need to configure LRU eviction for the existing one. >> >> - Also note that messages will be always received. We won't need >> workaround with "invalidation.key" objects anymore. >> >> - Also we don't need recursive evictions (which has very poor >> performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 ), >> because receiving node will know exactly what happened. It will remove >> objects just the same way like the "sender" node. >> >> - Finally the amount of traffic sent across the cluster will be much >> lower. >> >> This sounds like the big step, but IMO it's not that bad :) Note that we >> already have all the predicates in place for individual objects. The >> only change will be about sending/receiving notifications across >> cluster. I think I am able to prototype something by tomorrow to >> doublecheck this approach working and then finish it somewhen middle >> next week. WDYT? >> >> Marek >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > > From sthorger at redhat.com Mon Nov 7 00:22:09 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Mon, 7 Nov 2016 06:22:09 +0100 Subject: [keycloak-dev] evict on update/remove, make providers cache aware In-Reply-To: <9005fa16-4d9f-e2b4-c48e-511ecd03f005@redhat.com> References: <9005fa16-4d9f-e2b4-c48e-511ecd03f005@redhat.com> Message-ID: Why would someone get a non-cached model/provider? Wouldn't it be better to prevent that instead. On 5 November 2016 at 01:16, Bill Burke wrote: > I was thinking that we change all Mongo and JPA adapters, and > RealmProvider and UserProvider methods that can update or remove models > from the database and have them evict the cache of that type. This will > make sure that if somebody gets a non-cached model or provider and > performs an update, the cache is still invalidated. Sound ok? > > Thanks, > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From nafiux at gmail.com Mon Nov 7 01:17:03 2016 From: nafiux at gmail.com (Ignacio Ocampo) Date: Sun, 6 Nov 2016 22:17:03 -0800 Subject: [keycloak-dev] Microservice to Microservice calls, authorization from a queue message Message-ID: Hello Team, *Context:* I'm creating a cloud platform to support multiple applications with SSO. I'm using *Keycloak for authentication* and *Netflix Zuul for authorization* (API Gateway) thru *Keycloak Spring Security Adapter*. Each microservice expect an Authorization header, which contains a valid JWT, from which it will take the username (sub) to process the request. Each microservice-to-microservice call should go thru Netflix Zuul first, passing the Authorization header to maintain a stateless validation. That strategy allow to every microservice to know who is the user (sub) who is invoking the microservice indirectly. *Problem/Question 1:* What happens if a microservice is invoked from a queue message? One idea that I had is to storage in the queue the information related to the message + userInfo, and, create a dedicated microservice to process that kind of messages, with that approach this special microservice should read the userInfo from the queue and process the message. *Problem/Question 2:* But, what happens if the previous special microservice wants to call another normal microservice which expect to receive a JWT in a header? Should this special microservice create by himself a JWT to impersonate the user and be able to call the regular microservices? Another solution that I thought was to storage the original JWT in the queue, but, what happens if the queue calls to the special microservice later? Just after the JWT is not valid anymore (it expired) and the microservice called will reject the request? I will appreciate very much your help. Thanks. -- Ignacio Ocampo Mill?n From bburke at redhat.com Mon Nov 7 08:54:57 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 7 Nov 2016 08:54:57 -0500 Subject: [keycloak-dev] evict on update/remove, make providers cache aware In-Reply-To: References: <9005fa16-4d9f-e2b4-c48e-511ecd03f005@redhat.com> Message-ID: <877987e6-9ab2-b397-9de5-68061612e142@redhat.com> Can't prevent that because of user storage import support. I found this issue in the LDAP adapter when syncing. You have to interact with the UserProvider directly, but still make sure the cache is evicted. Rather than force the user to know about this scenario, it would be best to just handle it internally. On 11/7/16 12:22 AM, Stian Thorgersen wrote: > Why would someone get a non-cached model/provider? Wouldn't it be > better to prevent that instead. > > On 5 November 2016 at 01:16, Bill Burke > wrote: > > I was thinking that we change all Mongo and JPA adapters, and > RealmProvider and UserProvider methods that can update or remove > models > from the database and have them evict the cache of that type. > This will > make sure that if somebody gets a non-cached model or provider and > performs an update, the cache is still invalidated. Sound ok? > > Thanks, > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > From ronyjoy at gmail.com Tue Nov 8 16:35:52 2016 From: ronyjoy at gmail.com (rony joy) Date: Tue, 08 Nov 2016 21:35:52 +0000 Subject: [keycloak-dev] Custom Authenticator Spring bean support Message-ID: We have using keycloak and added some custom authenticators based on keycloak documentation( https://keycloak.gitbooks.io/server-developer-guide/content/topics/auth-spi.html). My question is how can I add spring support(ie autowiring) to my custom authenticators. From gambol99 at gmail.com Thu Nov 10 06:11:10 2016 From: gambol99 at gmail.com (gambol) Date: Thu, 10 Nov 2016 11:11:10 +0000 Subject: [keycloak-dev] OTP API Message-ID: Hiya Does the latest version of Keycloak provide any means of verifying a user's TOTP?. Our use-case at the moment, we have an application which once the user is authenticated we issue a token of sorts ... however, we wish to provide a popup that requests a user's TOPT every few hours which we "could" verify via service account ... I can't see any access at the moment via the rest api Rohith From sthorger at redhat.com Thu Nov 10 07:30:47 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Thu, 10 Nov 2016 13:30:47 +0100 Subject: [keycloak-dev] Splitting server-spi into server-spi and server-spi-private Message-ID: I've sent a PR for splitting server-spi into server-spi and sever-spi-private. server-spi should now aim to include only what is needed to create a user storage provider. There are more in there than I'd like, but it would be to much work to refactor things at this point. The PR is: https://github.com/keycloak/keycloak/pull/3489 I'll give everyone a chance to take a look and object to this before merging. So please review it asap as I'll be merging it tomorrow. From thomas.darimont at googlemail.com Thu Nov 10 07:52:41 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Thu, 10 Nov 2016 13:52:41 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: References: Message-ID: Hello Rohith, not that I know of - we'd also like to have this functionality. What would be the best place to add that? Perhaps this could be added to the UsersResource with a new endpoint like "/users/{userId}/otp-validation" or a (new) dedicated resource. A client could then do a POST to that endpoint with the current user's access token and the entered OTP code. Keycloak could then lookup and check the provided otp code. If the code is corret, response could indicate that via status HTTP 200 or HTTP 400 otherwise. Cheers, Thomas 2016-11-10 12:11 GMT+01:00 gambol : > Hiya > > Does the latest version of Keycloak provide any means of verifying a user's > TOTP?. Our use-case at the moment, we have an application which once the > user is authenticated we issue a token of sorts ... however, we wish to > provide a popup that requests a user's TOPT every few hours which we > "could" verify via service account ... I can't see any access at the moment > via the rest api > > Rohith > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From bburke at redhat.com Thu Nov 10 08:27:23 2016 From: bburke at redhat.com (Bill Burke) Date: Thu, 10 Nov 2016 08:27:23 -0500 Subject: [keycloak-dev] OTP API In-Reply-To: References: Message-ID: Should be generic and not specific to a credential type. Should also hook into brute force detection. IMO though, one of the reasons for SSO and keycloak is that the application does not gather credentials. This is the job of the auth server. IMO, we'd be better off with expiring the login at the client side, redirecting to auth server, auth server sees that the user session is 3 hours old, and requests OTP. On 11/10/16 7:52 AM, Thomas Darimont wrote: > Hello Rohith, > > not that I know of - we'd also like to have this functionality. > > > What would be the best place to add that? Perhaps this could be added to > the UsersResource with a new > endpoint like "/users/{userId}/otp-validation" or a (new) dedicated > resource. > > A client could then do a POST to that endpoint with the current user's > access token and the entered OTP code. > Keycloak could then lookup and check the provided otp code. > If the code is corret, response could indicate that via status HTTP 200 or > HTTP 400 otherwise. > > Cheers, > Thomas > > 2016-11-10 12:11 GMT+01:00 gambol : > >> Hiya >> >> Does the latest version of Keycloak provide any means of verifying a user's >> TOTP?. Our use-case at the moment, we have an application which once the >> user is authenticated we issue a token of sorts ... however, we wish to >> provide a popup that requests a user's TOPT every few hours which we >> "could" verify via service account ... I can't see any access at the moment >> via the rest api >> >> Rohith >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From thomas.darimont at googlemail.com Thu Nov 10 15:55:49 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Thu, 10 Nov 2016 21:55:49 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: References: Message-ID: Hello guys, I agree with you Bill that requiring a user to re-auth by redirecting to the auth-server is often the best thing to do. However in business applications one often sees the requirement to periodically reauthenticate the user without requiring the user to login again (e.g. without leaving the application). A common example for this is that users need to reenter a OTP credential before performing a certain critical operation... I gave this a quick spin... https://github.com/thomasdarimont/keycloak/commit/e891ca604acf5fd494525092c7a88628d9fb20bd as you can see I took some inspiration from the userinfo endpoint. Works quite well so far, but bruteforce protection is still missing. A demo session via curl looks like this: $ KC_REALM=otp-validation-test KC_USERNAME=tester KC_PASSWORD=test KC_CLIENT=test-client KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 KC_URL="http://localhost:8081/auth" ## Request Tokens for credentials KC_RESPONSE=$( \ curl -v \ -d "username=$KC_USERNAME" \ -d "password=$KC_PASSWORD" \ -d 'grant_type=password' \ -d "client_id=$KC_CLIENT" \ -d "client_secret=$KC_CLIENT_SECRET" \ "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ | jq . ) KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) * Trying 127.0.0.1... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8081 (#0) > POST /auth/realms/otp-validation-test/protocol/openid-connect/token HTTP/1.1 > Host: localhost:8081 > User-Agent: curl/7.45.0 > Accept: */* > Content-Length: 122 > Content-Type: application/x-www-form-urlencoded > } [122 bytes data] * upload completely sent off: 122 out of 122 bytes < HTTP/1.1 200 OK < Connection: keep-alive < Content-Type: application/json < Content-Length: 3713 < Date: Thu, 10 Nov 2016 20:32:16 GMT < { [3713 bytes data] # We got an acces token ... now lets try to validate a wrong OTP code curl -v \ -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '[{"type":"totp", "value":"24861"}]' \ $KC_URL/realms/$KC_REALM/credential-validation * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8081 (#0) > POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 > Host: localhost:8081 > User-Agent: curl/7.45.0 > Accept: */* > Authorization: Bearer eyJhb...EW3Q > Content-Type: application/json > Content-Length: 34 > * upload completely sent off: 34 out of 34 bytes < HTTP/1.1 400 Bad Request < Connection: keep-alive < Content-Length: 0 < Date: Thu, 10 Nov 2016 20:30:42 GMT < * Connection #0 to host localhost left intact # Let's try a currently valid OTP code curl -v \ -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '[{"type":"totp", "value":"949784"}]' \ $KC_URL/realms/$KC_REALM/credential-validation * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8081 (#0) > POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 > Host: localhost:8081 > User-Agent: curl/7.45.0 > Accept: */* > Authorization: Bearer eyJhb...EW3Q > Content-Type: application/json > Content-Length: 35 > * upload completely sent off: 35 out of 35 bytes < HTTP/1.1 200 OK < Connection: keep-alive < Content-Length: 0 < Date: Thu, 10 Nov 2016 20:34:11 GMT < * Connection #0 to host localhost left intact Cheers, Thomas 2016-11-10 14:27 GMT+01:00 Bill Burke : > Should be generic and not specific to a credential type. Should also > hook into brute force detection. IMO though, one of the reasons for SSO > and keycloak is that the application does not gather credentials. This > is the job of the auth server. IMO, we'd be better off with expiring > the login at the client side, redirecting to auth server, auth server > sees that the user session is 3 hours old, and requests OTP. > > > On 11/10/16 7:52 AM, Thomas Darimont wrote: > > Hello Rohith, > > > > not that I know of - we'd also like to have this functionality. > > > > > > What would be the best place to add that? Perhaps this could be added to > > the UsersResource with a new > > endpoint like "/users/{userId}/otp-validation" or a (new) dedicated > > resource. > > > > A client could then do a POST to that endpoint with the current user's > > access token and the entered OTP code. > > Keycloak could then lookup and check the provided otp code. > > If the code is corret, response could indicate that via status HTTP 200 > or > > HTTP 400 otherwise. > > > > Cheers, > > Thomas > > > > 2016-11-10 12:11 GMT+01:00 gambol : > > > >> Hiya > >> > >> Does the latest version of Keycloak provide any means of verifying a > user's > >> TOTP?. Our use-case at the moment, we have an application which once the > >> user is authenticated we issue a token of sorts ... however, we wish to > >> provide a popup that requests a user's TOPT every few hours which we > >> "could" verify via service account ... I can't see any access at the > moment > >> via the rest api > >> > >> Rohith > >> _______________________________________________ > >> keycloak-dev mailing list > >> keycloak-dev at lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > >> > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From sthorger at redhat.com Fri Nov 11 02:48:54 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Fri, 11 Nov 2016 08:48:54 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: References: Message-ID: This is not something we should add IMO. As Bill points out the whole idea is to not expose credentials to application. I'd rather see something that works without exposing credentials to the users. Could do a redirect with re-auth required. Could require just adding otp on re-auth. Could possibly do a popup with embedded login from KC. I'm sure there's cleaner ways than the applicaiton collection OTP. However, it's perfectly possible to implement this using a custom rest resource so if you really want it to do this then do it that way. On 10 November 2016 at 21:55, Thomas Darimont < thomas.darimont at googlemail.com> wrote: > Hello guys, > > I agree with you Bill that requiring a user to re-auth by redirecting to > the > auth-server is often the best thing to do. > However in business applications one often sees the requirement to > periodically > reauthenticate the user without requiring the user to login again (e.g. > without leaving the application). > A common example for this is that users need to reenter a OTP credential > before performing a certain critical operation... > > I gave this a quick spin... > https://github.com/thomasdarimont/keycloak/commit/ > e891ca604acf5fd494525092c7a88628d9fb20bd > as you can see I took some inspiration from the userinfo endpoint. > > Works quite well so far, but bruteforce protection is still missing. > > A demo session via curl looks like this: > > $ KC_REALM=otp-validation-test > KC_USERNAME=tester > KC_PASSWORD=test > KC_CLIENT=test-client > KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 > KC_URL="http://localhost:8081/auth" > > ## Request Tokens for credentials > KC_RESPONSE=$( \ > curl -v \ > -d "username=$KC_USERNAME" \ > -d "password=$KC_PASSWORD" \ > -d 'grant_type=password' \ > -d "client_id=$KC_CLIENT" \ > -d "client_secret=$KC_CLIENT_SECRET" \ > "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ > | jq . > ) > > KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) > KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) > KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) > * Trying 127.0.0.1... > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- > 0* Connected to localhost (127.0.0.1) port 8081 (#0) > > POST /auth/realms/otp-validation-test/protocol/openid-connect/token > HTTP/1.1 > > Host: localhost:8081 > > User-Agent: curl/7.45.0 > > Accept: */* > > Content-Length: 122 > > Content-Type: application/x-www-form-urlencoded > > > } [122 bytes data] > * upload completely sent off: 122 out of 122 bytes > < HTTP/1.1 200 OK > < Connection: keep-alive > < Content-Type: application/json > < Content-Length: 3713 > < Date: Thu, 10 Nov 2016 20:32:16 GMT > < > { [3713 bytes data] > > # We got an acces token ... now lets try to validate a wrong OTP code > > curl -v \ > -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ > -H "Content-Type: application/json" \ > -d '[{"type":"totp", "value":"24861"}]' \ > $KC_URL/realms/$KC_REALM/credential-validation > * Trying 127.0.0.1... > * Connected to localhost (127.0.0.1) port 8081 (#0) > > POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 > > Host: localhost:8081 > > User-Agent: curl/7.45.0 > > Accept: */* > > Authorization: Bearer eyJhb...EW3Q > > Content-Type: application/json > > Content-Length: 34 > > > * upload completely sent off: 34 out of 34 bytes > < HTTP/1.1 400 Bad Request > < Connection: keep-alive > < Content-Length: 0 > < Date: Thu, 10 Nov 2016 20:30:42 GMT > < > * Connection #0 to host localhost left intact > > # Let's try a currently valid OTP code > > curl -v \ > -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ > -H "Content-Type: application/json" \ > -d '[{"type":"totp", "value":"949784"}]' \ > $KC_URL/realms/$KC_REALM/credential-validation > * Trying 127.0.0.1... > * Connected to localhost (127.0.0.1) port 8081 (#0) > > POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 > > Host: localhost:8081 > > User-Agent: curl/7.45.0 > > Accept: */* > > Authorization: Bearer eyJhb...EW3Q > > Content-Type: application/json > > Content-Length: 35 > > > * upload completely sent off: 35 out of 35 bytes > < HTTP/1.1 200 OK > < Connection: keep-alive > < Content-Length: 0 > < Date: Thu, 10 Nov 2016 20:34:11 GMT > < > * Connection #0 to host localhost left intact > > Cheers, > Thomas > > 2016-11-10 14:27 GMT+01:00 Bill Burke : > > > Should be generic and not specific to a credential type. Should also > > hook into brute force detection. IMO though, one of the reasons for SSO > > and keycloak is that the application does not gather credentials. This > > is the job of the auth server. IMO, we'd be better off with expiring > > the login at the client side, redirecting to auth server, auth server > > sees that the user session is 3 hours old, and requests OTP. > > > > > > On 11/10/16 7:52 AM, Thomas Darimont wrote: > > > Hello Rohith, > > > > > > not that I know of - we'd also like to have this functionality. > > > > > > > > > What would be the best place to add that? Perhaps this could be added > to > > > the UsersResource with a new > > > endpoint like "/users/{userId}/otp-validation" or a (new) dedicated > > > resource. > > > > > > A client could then do a POST to that endpoint with the current user's > > > access token and the entered OTP code. > > > Keycloak could then lookup and check the provided otp code. > > > If the code is corret, response could indicate that via status HTTP 200 > > or > > > HTTP 400 otherwise. > > > > > > Cheers, > > > Thomas > > > > > > 2016-11-10 12:11 GMT+01:00 gambol : > > > > > >> Hiya > > >> > > >> Does the latest version of Keycloak provide any means of verifying a > > user's > > >> TOTP?. Our use-case at the moment, we have an application which once > the > > >> user is authenticated we issue a token of sorts ... however, we wish > to > > >> provide a popup that requests a user's TOPT every few hours which we > > >> "could" verify via service account ... I can't see any access at the > > moment > > >> via the rest api > > >> > > >> Rohith > > >> _______________________________________________ > > >> keycloak-dev mailing list > > >> keycloak-dev at lists.jboss.org > > >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > > >> > > > _______________________________________________ > > > keycloak-dev mailing list > > > keycloak-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From thomas.darimont at googlemail.com Fri Nov 11 05:00:38 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Fri, 11 Nov 2016 11:00:38 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: References: Message-ID: Hi Stian, thanks for the feedback.I understand the philosophy behind not exposing credentials to an application and agree that one way to solve this is by using a custom endpoint. Question: Is there a general recommendation or guideline about resource names / paths? E.g. something like custom resources should be located beneath a special base path (e.g. /custom) to avoid potential clashes with Keycloak resources in newer versions? I also like the idea for (partial) re-auth via some sort of popup / iframe. Are there any JIRAs logged for this already? Another thing is that this might not work in every case, e.g. when integrating Desktop applications where a the "client" sends the user credentials to a server which then needs to check those credentials against keycloak, e.g. by trying to acquire an access token. Cheers, Thomas 2016-11-11 8:48 GMT+01:00 Stian Thorgersen : > This is not something we should add IMO. As Bill points out the whole idea > is to not expose credentials to application. I'd rather see something that > works without exposing credentials to the users. Could do a redirect with > re-auth required. Could require just adding otp on re-auth. Could possibly > do a popup with embedded login from KC. I'm sure there's cleaner ways than > the applicaiton collection OTP. > > However, it's perfectly possible to implement this using a custom rest > resource so if you really want it to do this then do it that way. > > On 10 November 2016 at 21:55, Thomas Darimont com> wrote: > >> Hello guys, >> >> I agree with you Bill that requiring a user to re-auth by redirecting to >> the >> auth-server is often the best thing to do. >> However in business applications one often sees the requirement to >> periodically >> reauthenticate the user without requiring the user to login again (e.g. >> without leaving the application). >> A common example for this is that users need to reenter a OTP credential >> before performing a certain critical operation... >> >> I gave this a quick spin... >> https://github.com/thomasdarimont/keycloak/commit/e891ca604a >> cf5fd494525092c7a88628d9fb20bd >> as you can see I took some inspiration from the userinfo endpoint. >> >> Works quite well so far, but bruteforce protection is still missing. >> >> A demo session via curl looks like this: >> >> $ KC_REALM=otp-validation-test >> KC_USERNAME=tester >> KC_PASSWORD=test >> KC_CLIENT=test-client >> KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 >> KC_URL="http://localhost:8081/auth" >> >> ## Request Tokens for credentials >> KC_RESPONSE=$( \ >> curl -v \ >> -d "username=$KC_USERNAME" \ >> -d "password=$KC_PASSWORD" \ >> -d 'grant_type=password' \ >> -d "client_id=$KC_CLIENT" \ >> -d "client_secret=$KC_CLIENT_SECRET" \ >> "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ >> | jq . >> ) >> >> KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) >> KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) >> KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) >> * Trying 127.0.0.1... >> % Total % Received % Xferd Average Speed Time Time Time >> Current >> Dload Upload Total Spent Left >> Speed >> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- >> 0* Connected to localhost (127.0.0.1) port 8081 (#0) >> > POST /auth/realms/otp-validation-test/protocol/openid-connect/token >> HTTP/1.1 >> > Host: localhost:8081 >> > User-Agent: curl/7.45.0 >> > Accept: */* >> > Content-Length: 122 >> > Content-Type: application/x-www-form-urlencoded >> > >> } [122 bytes data] >> * upload completely sent off: 122 out of 122 bytes >> < HTTP/1.1 200 OK >> < Connection: keep-alive >> < Content-Type: application/json >> < Content-Length: 3713 >> < Date: Thu, 10 Nov 2016 20:32:16 GMT >> < >> { [3713 bytes data] >> >> # We got an acces token ... now lets try to validate a wrong OTP code >> >> curl -v \ >> -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ >> -H "Content-Type: application/json" \ >> -d '[{"type":"totp", "value":"24861"}]' \ >> $KC_URL/realms/$KC_REALM/credential-validation >> * Trying 127.0.0.1... >> * Connected to localhost (127.0.0.1) port 8081 (#0) >> > POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 >> > Host: localhost:8081 >> > User-Agent: curl/7.45.0 >> > Accept: */* >> > Authorization: Bearer eyJhb...EW3Q >> > Content-Type: application/json >> > Content-Length: 34 >> > >> * upload completely sent off: 34 out of 34 bytes >> < HTTP/1.1 400 Bad Request >> < Connection: keep-alive >> < Content-Length: 0 >> < Date: Thu, 10 Nov 2016 20:30:42 GMT >> < >> * Connection #0 to host localhost left intact >> >> # Let's try a currently valid OTP code >> >> curl -v \ >> -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ >> -H "Content-Type: application/json" \ >> -d '[{"type":"totp", "value":"949784"}]' \ >> $KC_URL/realms/$KC_REALM/credential-validation >> * Trying 127.0.0.1... >> * Connected to localhost (127.0.0.1) port 8081 (#0) >> > POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 >> > Host: localhost:8081 >> > User-Agent: curl/7.45.0 >> > Accept: */* >> > Authorization: Bearer eyJhb...EW3Q >> > Content-Type: application/json >> > Content-Length: 35 >> > >> * upload completely sent off: 35 out of 35 bytes >> < HTTP/1.1 200 OK >> < Connection: keep-alive >> < Content-Length: 0 >> < Date: Thu, 10 Nov 2016 20:34:11 GMT >> < >> * Connection #0 to host localhost left intact >> >> Cheers, >> Thomas >> >> 2016-11-10 14:27 GMT+01:00 Bill Burke : >> >> > Should be generic and not specific to a credential type. Should also >> > hook into brute force detection. IMO though, one of the reasons for SSO >> > and keycloak is that the application does not gather credentials. This >> > is the job of the auth server. IMO, we'd be better off with expiring >> > the login at the client side, redirecting to auth server, auth server >> > sees that the user session is 3 hours old, and requests OTP. >> > >> > >> > On 11/10/16 7:52 AM, Thomas Darimont wrote: >> > > Hello Rohith, >> > > >> > > not that I know of - we'd also like to have this functionality. >> > > >> > > >> > > What would be the best place to add that? Perhaps this could be added >> to >> > > the UsersResource with a new >> > > endpoint like "/users/{userId}/otp-validation" or a (new) dedicated >> > > resource. >> > > >> > > A client could then do a POST to that endpoint with the current >> user's >> > > access token and the entered OTP code. >> > > Keycloak could then lookup and check the provided otp code. >> > > If the code is corret, response could indicate that via status HTTP >> 200 >> > or >> > > HTTP 400 otherwise. >> > > >> > > Cheers, >> > > Thomas >> > > >> > > 2016-11-10 12:11 GMT+01:00 gambol : >> > > >> > >> Hiya >> > >> >> > >> Does the latest version of Keycloak provide any means of verifying a >> > user's >> > >> TOTP?. Our use-case at the moment, we have an application which once >> the >> > >> user is authenticated we issue a token of sorts ... however, we wish >> to >> > >> provide a popup that requests a user's TOPT every few hours which we >> > >> "could" verify via service account ... I can't see any access at the >> > moment >> > >> via the rest api >> > >> >> > >> Rohith >> > >> _______________________________________________ >> > >> keycloak-dev mailing list >> > >> keycloak-dev at lists.jboss.org >> > >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > >> >> > > _______________________________________________ >> > > keycloak-dev mailing list >> > > keycloak-dev at lists.jboss.org >> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > >> > _______________________________________________ >> > keycloak-dev mailing list >> > keycloak-dev at lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > From mposolda at redhat.com Fri Nov 11 05:26:18 2016 From: mposolda at redhat.com (Marek Posolda) Date: Fri, 11 Nov 2016 11:26:18 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: References: Message-ID: <81b6eaa5-aeae-a031-1f1f-2ff294d7ee5a@redhat.com> On 11/11/16 11:00, Thomas Darimont wrote: > Hi Stian, > > thanks for the feedback.I understand the philosophy behind not exposing > credentials to an application and agree that > one way to solve this is by using a custom endpoint. > > Question: Is there a general recommendation or guideline about resource > names / paths? E.g. something like custom resources should be located > beneath a special base path (e.g. /custom) to avoid potential clashes with > Keycloak resources in newer versions? > > I also like the idea for (partial) re-auth via some sort of popup / iframe. > Are there any JIRAs logged for this already? > Another thing is that this might not work in every case, e.g. when > integrating Desktop applications where a > the "client" sends the user credentials to a server which then needs to > check those credentials against keycloak, e.g. by trying to acquire an > access token. One related JIRA is the support for OIDC "display" parameter. This allows to open login page in the popup (among other things). See https://issues.jboss.org/browse/KEYCLOAK-3322 . Regarding desktop client, we already have "Direct Grant" authentication flow, which has some support for OTP verification. You just need to add "totp" parameter to the request though. See the ValidateOTP class for details. Marek > > Cheers, > Thomas > > 2016-11-11 8:48 GMT+01:00 Stian Thorgersen : > >> This is not something we should add IMO. As Bill points out the whole idea >> is to not expose credentials to application. I'd rather see something that >> works without exposing credentials to the users. Could do a redirect with >> re-auth required. Could require just adding otp on re-auth. Could possibly >> do a popup with embedded login from KC. I'm sure there's cleaner ways than >> the applicaiton collection OTP. >> >> However, it's perfectly possible to implement this using a custom rest >> resource so if you really want it to do this then do it that way. >> >> On 10 November 2016 at 21:55, Thomas Darimont > com> wrote: >> >>> Hello guys, >>> >>> I agree with you Bill that requiring a user to re-auth by redirecting to >>> the >>> auth-server is often the best thing to do. >>> However in business applications one often sees the requirement to >>> periodically >>> reauthenticate the user without requiring the user to login again (e.g. >>> without leaving the application). >>> A common example for this is that users need to reenter a OTP credential >>> before performing a certain critical operation... >>> >>> I gave this a quick spin... >>> https://github.com/thomasdarimont/keycloak/commit/e891ca604a >>> cf5fd494525092c7a88628d9fb20bd >>> as you can see I took some inspiration from the userinfo endpoint. >>> >>> Works quite well so far, but bruteforce protection is still missing. >>> >>> A demo session via curl looks like this: >>> >>> $ KC_REALM=otp-validation-test >>> KC_USERNAME=tester >>> KC_PASSWORD=test >>> KC_CLIENT=test-client >>> KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 >>> KC_URL="http://localhost:8081/auth" >>> >>> ## Request Tokens for credentials >>> KC_RESPONSE=$( \ >>> curl -v \ >>> -d "username=$KC_USERNAME" \ >>> -d "password=$KC_PASSWORD" \ >>> -d 'grant_type=password' \ >>> -d "client_id=$KC_CLIENT" \ >>> -d "client_secret=$KC_CLIENT_SECRET" \ >>> "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ >>> | jq . >>> ) >>> >>> KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) >>> KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) >>> KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) >>> * Trying 127.0.0.1... >>> % Total % Received % Xferd Average Speed Time Time Time >>> Current >>> Dload Upload Total Spent Left >>> Speed >>> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- >>> 0* Connected to localhost (127.0.0.1) port 8081 (#0) >>>> POST /auth/realms/otp-validation-test/protocol/openid-connect/token >>> HTTP/1.1 >>>> Host: localhost:8081 >>>> User-Agent: curl/7.45.0 >>>> Accept: */* >>>> Content-Length: 122 >>>> Content-Type: application/x-www-form-urlencoded >>>> >>> } [122 bytes data] >>> * upload completely sent off: 122 out of 122 bytes >>> < HTTP/1.1 200 OK >>> < Connection: keep-alive >>> < Content-Type: application/json >>> < Content-Length: 3713 >>> < Date: Thu, 10 Nov 2016 20:32:16 GMT >>> < >>> { [3713 bytes data] >>> >>> # We got an acces token ... now lets try to validate a wrong OTP code >>> >>> curl -v \ >>> -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ >>> -H "Content-Type: application/json" \ >>> -d '[{"type":"totp", "value":"24861"}]' \ >>> $KC_URL/realms/$KC_REALM/credential-validation >>> * Trying 127.0.0.1... >>> * Connected to localhost (127.0.0.1) port 8081 (#0) >>>> POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 >>>> Host: localhost:8081 >>>> User-Agent: curl/7.45.0 >>>> Accept: */* >>>> Authorization: Bearer eyJhb...EW3Q >>>> Content-Type: application/json >>>> Content-Length: 34 >>>> >>> * upload completely sent off: 34 out of 34 bytes >>> < HTTP/1.1 400 Bad Request >>> < Connection: keep-alive >>> < Content-Length: 0 >>> < Date: Thu, 10 Nov 2016 20:30:42 GMT >>> < >>> * Connection #0 to host localhost left intact >>> >>> # Let's try a currently valid OTP code >>> >>> curl -v \ >>> -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ >>> -H "Content-Type: application/json" \ >>> -d '[{"type":"totp", "value":"949784"}]' \ >>> $KC_URL/realms/$KC_REALM/credential-validation >>> * Trying 127.0.0.1... >>> * Connected to localhost (127.0.0.1) port 8081 (#0) >>>> POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 >>>> Host: localhost:8081 >>>> User-Agent: curl/7.45.0 >>>> Accept: */* >>>> Authorization: Bearer eyJhb...EW3Q >>>> Content-Type: application/json >>>> Content-Length: 35 >>>> >>> * upload completely sent off: 35 out of 35 bytes >>> < HTTP/1.1 200 OK >>> < Connection: keep-alive >>> < Content-Length: 0 >>> < Date: Thu, 10 Nov 2016 20:34:11 GMT >>> < >>> * Connection #0 to host localhost left intact >>> >>> Cheers, >>> Thomas >>> >>> 2016-11-10 14:27 GMT+01:00 Bill Burke : >>> >>>> Should be generic and not specific to a credential type. Should also >>>> hook into brute force detection. IMO though, one of the reasons for SSO >>>> and keycloak is that the application does not gather credentials. This >>>> is the job of the auth server. IMO, we'd be better off with expiring >>>> the login at the client side, redirecting to auth server, auth server >>>> sees that the user session is 3 hours old, and requests OTP. >>>> >>>> >>>> On 11/10/16 7:52 AM, Thomas Darimont wrote: >>>>> Hello Rohith, >>>>> >>>>> not that I know of - we'd also like to have this functionality. >>>>> >>>>> >>>>> What would be the best place to add that? Perhaps this could be added >>> to >>>>> the UsersResource with a new >>>>> endpoint like "/users/{userId}/otp-validation" or a (new) dedicated >>>>> resource. >>>>> >>>>> A client could then do a POST to that endpoint with the current >>> user's >>>>> access token and the entered OTP code. >>>>> Keycloak could then lookup and check the provided otp code. >>>>> If the code is corret, response could indicate that via status HTTP >>> 200 >>>> or >>>>> HTTP 400 otherwise. >>>>> >>>>> Cheers, >>>>> Thomas >>>>> >>>>> 2016-11-10 12:11 GMT+01:00 gambol : >>>>> >>>>>> Hiya >>>>>> >>>>>> Does the latest version of Keycloak provide any means of verifying a >>>> user's >>>>>> TOTP?. Our use-case at the moment, we have an application which once >>> the >>>>>> user is authenticated we issue a token of sorts ... however, we wish >>> to >>>>>> provide a popup that requests a user's TOPT every few hours which we >>>>>> "could" verify via service account ... I can't see any access at the >>>> moment >>>>>> via the rest api >>>>>> >>>>>> Rohith >>>>>> _______________________________________________ >>>>>> keycloak-dev mailing list >>>>>> keycloak-dev at lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >>>>> _______________________________________________ >>>>> keycloak-dev mailing list >>>>> keycloak-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From sthorger at redhat.com Fri Nov 11 05:31:28 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Fri, 11 Nov 2016 11:31:28 +0100 Subject: [keycloak-dev] Travis issues again Message-ID: Travis has an upper limit on 50 min for a single "script". Seems like our tests are now regularly taking more than 50 min to execute so that's the reason so many PRs are failing on Travis at the moment. I'll try to sort this out asap. From sthorger at redhat.com Fri Nov 11 05:42:55 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Fri, 11 Nov 2016 11:42:55 +0100 Subject: [keycloak-dev] Keycloak 2.4.0 Status Message-ID: Due to meetings and public holidays 2.4.0.CR1 will be delayed and will be released on Monday 21st. Then 2.4.0.Final is on Wednesday 23rd. We will follow-up with 2.4.1.Final on 7th December. All changes for 2.4.0 needs to be in by end of Friday 18th December. Marek will drive testing of the release on Monday 21st. The release will be done in the evening (EU time) on Monday 21st. Remaining work for 2.4.0.CR1 includes: * LDAP porting to User Storage SPI including migration. If not ready it'll get postponed to 2.4.1. * Infinispan cache issues - Marek is working on this and the aim is to include. If not ready it'll get postponed to 2.4.1. * Remove native libraries included in SSSD jar - Bruno is working on this. * Fuse issues - Marek is on these * A few more minor issues, see JIRA for list Postponed to 2.4.1: * Kerberos and SSSD porting to User Storage SPI * LDAP without syncing users to Keycloak db * Removal of old User Federation SPI From thomas.darimont at googlemail.com Fri Nov 11 09:06:46 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Fri, 11 Nov 2016 15:06:46 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: <81b6eaa5-aeae-a031-1f1f-2ff294d7ee5a@redhat.com> References: <81b6eaa5-aeae-a031-1f1f-2ff294d7ee5a@redhat.com> Message-ID: Hello Marek, I'm already using the direct grant auth flow for authenticating a desktop app against the Keycloak userbase, but thanks for the hint with the totp parameter, for the record I added a simple curl example below. Does anyone know about OIDC providers that support the display=popup parameter - I could not find a project / product that supports this. Btw. the usage for OTP to verify sensitive operations or transaction authorization seems indeed to be quite common. http://www.transmitsecurity.com/use_case/otp-one-time-password/ https://bankapi.net/docs/public/authorization-getstarted.html https://www.owasp.org/index.php/Transaction_Authorization_Cheat_Sheet Cheers, Thomas -- KC_REALM=otp-validation-test KC_USERNAME=tester KC_PASSWORD=test KC_OTP_CODE=654328 KC_CLIENT=test-client KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 KC_URL="http://localhost:8081/auth" ## Request Tokens for credentials KC_RESPONSE=$( \ curl -v \ -d "username=$KC_USERNAME" \ -d "password=$KC_PASSWORD" \ -d 'grant_type=password' \ -d "client_id=$KC_CLIENT" \ -d "client_secret=$KC_CLIENT_SECRET" \ -d "totp=$KC_OTP_CODE" \ "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ | jq . ) KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) ## Validate otp token with custom credential validation endpoint KC_OTP_CODE=027253 curl -v \ -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d "[{"\""type"\"":"\""totp"\"","\""value"\"":"\""$KC_OTP_CODE"\""}]" \ $KC_URL/realms/$KC_REALM/credential-validation 2016-11-11 11:26 GMT+01:00 Marek Posolda : > On 11/11/16 11:00, Thomas Darimont wrote: > >> Hi Stian, >> >> thanks for the feedback.I understand the philosophy behind not exposing >> credentials to an application and agree that >> one way to solve this is by using a custom endpoint. >> >> Question: Is there a general recommendation or guideline about resource >> names / paths? E.g. something like custom resources should be located >> beneath a special base path (e.g. /custom) to avoid potential clashes with >> Keycloak resources in newer versions? >> >> I also like the idea for (partial) re-auth via some sort of popup / >> iframe. >> Are there any JIRAs logged for this already? >> Another thing is that this might not work in every case, e.g. when >> integrating Desktop applications where a >> the "client" sends the user credentials to a server which then needs to >> check those credentials against keycloak, e.g. by trying to acquire an >> access token. >> > One related JIRA is the support for OIDC "display" parameter. This allows > to open login page in the popup (among other things). See > https://issues.jboss.org/browse/KEYCLOAK-3322 . > > Regarding desktop client, we already have "Direct Grant" authentication > flow, which has some support for OTP verification. You just need to add > "totp" parameter to the request though. See the ValidateOTP class for > details. > > Marek > > >> Cheers, >> Thomas >> >> 2016-11-11 8:48 GMT+01:00 Stian Thorgersen : >> >> This is not something we should add IMO. As Bill points out the whole idea >>> is to not expose credentials to application. I'd rather see something >>> that >>> works without exposing credentials to the users. Could do a redirect with >>> re-auth required. Could require just adding otp on re-auth. Could >>> possibly >>> do a popup with embedded login from KC. I'm sure there's cleaner ways >>> than >>> the applicaiton collection OTP. >>> >>> However, it's perfectly possible to implement this using a custom rest >>> resource so if you really want it to do this then do it that way. >>> >>> On 10 November 2016 at 21:55, Thomas Darimont >>> >> com> wrote: >>> >>> Hello guys, >>>> >>>> I agree with you Bill that requiring a user to re-auth by redirecting to >>>> the >>>> auth-server is often the best thing to do. >>>> However in business applications one often sees the requirement to >>>> periodically >>>> reauthenticate the user without requiring the user to login again (e.g. >>>> without leaving the application). >>>> A common example for this is that users need to reenter a OTP credential >>>> before performing a certain critical operation... >>>> >>>> I gave this a quick spin... >>>> https://github.com/thomasdarimont/keycloak/commit/e891ca604a >>>> cf5fd494525092c7a88628d9fb20bd >>>> as you can see I took some inspiration from the userinfo endpoint. >>>> >>>> Works quite well so far, but bruteforce protection is still missing. >>>> >>>> A demo session via curl looks like this: >>>> >>>> $ KC_REALM=otp-validation-test >>>> KC_USERNAME=tester >>>> KC_PASSWORD=test >>>> KC_CLIENT=test-client >>>> KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 >>>> KC_URL="http://localhost:8081/auth" >>>> >>>> ## Request Tokens for credentials >>>> KC_RESPONSE=$( \ >>>> curl -v \ >>>> -d "username=$KC_USERNAME" \ >>>> -d "password=$KC_PASSWORD" \ >>>> -d 'grant_type=password' \ >>>> -d "client_id=$KC_CLIENT" \ >>>> -d "client_secret=$KC_CLIENT_SECRET" \ >>>> "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ >>>> | jq . >>>> ) >>>> >>>> KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) >>>> KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) >>>> KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) >>>> * Trying 127.0.0.1... >>>> % Total % Received % Xferd Average Speed Time Time Time >>>> Current >>>> Dload Upload Total Spent Left >>>> Speed >>>> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- >>>> --:--:-- >>>> 0* Connected to localhost (127.0.0.1) port 8081 (#0) >>>> >>>>> POST /auth/realms/otp-validation-test/protocol/openid-connect/token >>>>> >>>> HTTP/1.1 >>>> >>>>> Host: localhost:8081 >>>>> User-Agent: curl/7.45.0 >>>>> Accept: */* >>>>> Content-Length: 122 >>>>> Content-Type: application/x-www-form-urlencoded >>>>> >>>>> } [122 bytes data] >>>> * upload completely sent off: 122 out of 122 bytes >>>> < HTTP/1.1 200 OK >>>> < Connection: keep-alive >>>> < Content-Type: application/json >>>> < Content-Length: 3713 >>>> < Date: Thu, 10 Nov 2016 20:32:16 GMT >>>> < >>>> { [3713 bytes data] >>>> >>>> # We got an acces token ... now lets try to validate a wrong OTP code >>>> >>>> curl -v \ >>>> -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ >>>> -H "Content-Type: application/json" \ >>>> -d '[{"type":"totp", "value":"24861"}]' \ >>>> $KC_URL/realms/$KC_REALM/credential-validation >>>> * Trying 127.0.0.1... >>>> * Connected to localhost (127.0.0.1) port 8081 (#0) >>>> >>>>> POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 >>>>> Host: localhost:8081 >>>>> User-Agent: curl/7.45.0 >>>>> Accept: */* >>>>> Authorization: Bearer eyJhb...EW3Q >>>>> Content-Type: application/json >>>>> Content-Length: 34 >>>>> >>>>> * upload completely sent off: 34 out of 34 bytes >>>> < HTTP/1.1 400 Bad Request >>>> < Connection: keep-alive >>>> < Content-Length: 0 >>>> < Date: Thu, 10 Nov 2016 20:30:42 GMT >>>> < >>>> * Connection #0 to host localhost left intact >>>> >>>> # Let's try a currently valid OTP code >>>> >>>> curl -v \ >>>> -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ >>>> -H "Content-Type: application/json" \ >>>> -d '[{"type":"totp", "value":"949784"}]' \ >>>> $KC_URL/realms/$KC_REALM/credential-validation >>>> * Trying 127.0.0.1... >>>> * Connected to localhost (127.0.0.1) port 8081 (#0) >>>> >>>>> POST /auth/realms/otp-validation-test/credential-validation HTTP/1.1 >>>>> Host: localhost:8081 >>>>> User-Agent: curl/7.45.0 >>>>> Accept: */* >>>>> Authorization: Bearer eyJhb...EW3Q >>>>> Content-Type: application/json >>>>> Content-Length: 35 >>>>> >>>>> * upload completely sent off: 35 out of 35 bytes >>>> < HTTP/1.1 200 OK >>>> < Connection: keep-alive >>>> < Content-Length: 0 >>>> < Date: Thu, 10 Nov 2016 20:34:11 GMT >>>> < >>>> * Connection #0 to host localhost left intact >>>> >>>> Cheers, >>>> Thomas >>>> >>>> 2016-11-10 14:27 GMT+01:00 Bill Burke : >>>> >>>> Should be generic and not specific to a credential type. Should also >>>>> hook into brute force detection. IMO though, one of the reasons for >>>>> SSO >>>>> and keycloak is that the application does not gather credentials. This >>>>> is the job of the auth server. IMO, we'd be better off with expiring >>>>> the login at the client side, redirecting to auth server, auth server >>>>> sees that the user session is 3 hours old, and requests OTP. >>>>> >>>>> >>>>> On 11/10/16 7:52 AM, Thomas Darimont wrote: >>>>> >>>>>> Hello Rohith, >>>>>> >>>>>> not that I know of - we'd also like to have this functionality. >>>>>> >>>>>> >>>>>> What would be the best place to add that? Perhaps this could be added >>>>>> >>>>> to >>>> >>>>> the UsersResource with a new >>>>>> endpoint like "/users/{userId}/otp-validation" or a (new) dedicated >>>>>> resource. >>>>>> >>>>>> A client could then do a POST to that endpoint with the current >>>>>> >>>>> user's >>>> >>>>> access token and the entered OTP code. >>>>>> Keycloak could then lookup and check the provided otp code. >>>>>> If the code is corret, response could indicate that via status HTTP >>>>>> >>>>> 200 >>>> >>>>> or >>>>> >>>>>> HTTP 400 otherwise. >>>>>> >>>>>> Cheers, >>>>>> Thomas >>>>>> >>>>>> 2016-11-10 12:11 GMT+01:00 gambol : >>>>>> >>>>>> Hiya >>>>>>> >>>>>>> Does the latest version of Keycloak provide any means of verifying a >>>>>>> >>>>>> user's >>>>> >>>>>> TOTP?. Our use-case at the moment, we have an application which once >>>>>>> >>>>>> the >>>> >>>>> user is authenticated we issue a token of sorts ... however, we wish >>>>>>> >>>>>> to >>>> >>>>> provide a popup that requests a user's TOPT every few hours which we >>>>>>> "could" verify via service account ... I can't see any access at the >>>>>>> >>>>>> moment >>>>> >>>>>> via the rest api >>>>>>> >>>>>>> Rohith >>>>>>> _______________________________________________ >>>>>>> keycloak-dev mailing list >>>>>>> keycloak-dev at lists.jboss.org >>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>> >>>>>>> _______________________________________________ >>>>>> keycloak-dev mailing list >>>>>> keycloak-dev at lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >>>>> _______________________________________________ >>>>> keycloak-dev mailing list >>>>> keycloak-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>> >>>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >>>> >>> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > > From sthorger at redhat.com Fri Nov 11 10:06:53 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Fri, 11 Nov 2016 16:06:53 +0100 Subject: [keycloak-dev] Travis issues again In-Reply-To: References: Message-ID: Update - it's still broken Trying to break tests up and use Matrix jobs. This should hopefully work, but only time will tell: https://travis-ci.org/keycloak/keycloak/builds/175090036 On 11 November 2016 at 11:31, Stian Thorgersen wrote: > Travis has an upper limit on 50 min for a single "script". Seems like our > tests are now regularly taking more than 50 min to execute so that's the > reason so many PRs are failing on Travis at the moment. > > I'll try to sort this out asap. > From sthorger at redhat.com Fri Nov 11 11:01:23 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Fri, 11 Nov 2016 17:01:23 +0100 Subject: [keycloak-dev] Travis issues again In-Reply-To: References: Message-ID: Sorted. It's now running as 5 parallel jobs. This actually reduces total time to check a PR to ~25min, which is half of what it was before. On 11 November 2016 at 16:06, Stian Thorgersen wrote: > Update - it's still broken > > Trying to break tests up and use Matrix jobs. This should hopefully work, > but only time will tell: > https://travis-ci.org/keycloak/keycloak/builds/175090036 > > On 11 November 2016 at 11:31, Stian Thorgersen > wrote: > >> Travis has an upper limit on 50 min for a single "script". Seems like our >> tests are now regularly taking more than 50 min to execute so that's the >> reason so many PRs are failing on Travis at the moment. >> >> I'll try to sort this out asap. >> > > From davey.chu at gmail.com Sat Nov 12 05:54:14 2016 From: davey.chu at gmail.com (Davey Chu) Date: Sat, 12 Nov 2016 11:54:14 +0100 Subject: [keycloak-dev] Run Keycloak side by side Message-ID: Hello, We have a use case where we initially want to run Keycloak side by side with our own authentication system. Our current way of handling it is by effectively overriding KeycloakOIDCFilter::shouldSkip method to only execute the filter if the user is on a special Keycloak login url or if the attribute KeycloakAccount.class.getName() is on the session (naive way to deduce that the user has logged in with Keycloak). As this feels like a fragile way of doing things, we were wondering if others have had the same use case and could point us in a better direction. Additionally, we deploy our single war in DTAP. We would therefore like to suggest to add a possibility to set the keycloak.config.file path via a system environment variable. Looking forward to any response or advice, Davey Chu From sena at ananse.im Sun Nov 13 00:53:36 2016 From: sena at ananse.im (Sena Gbeckor-Kove) Date: Sun, 13 Nov 2016 05:53:36 +0000 Subject: [keycloak-dev] OrientDB Support Message-ID: Hi, If nobody has implemented it yet. I'd be interested in adding OrientDB support. That way we can avoid having to manage Mongo as well. Cheers -- ----- Sena Gbeckor-Kove Founder & CEO ananse 19-09, 1 North Bridge Road Singapore 179094 m: +65 3163 5466 e: sena at ananse.im w: www.ananse.im From mposolda at redhat.com Mon Nov 14 02:44:26 2016 From: mposolda at redhat.com (Marek Posolda) Date: Mon, 14 Nov 2016 08:44:26 +0100 Subject: [keycloak-dev] OTP API In-Reply-To: References: <81b6eaa5-aeae-a031-1f1f-2ff294d7ee5a@redhat.com> Message-ID: <3e56c640-0659-969f-07c1-63aedb05a1c0@redhat.com> On 11/11/16 15:06, Thomas Darimont wrote: > Hello Marek, > > I'm already using the direct grant auth flow for authenticating a > desktop app against the Keycloak userbase, > but thanks for the hint with the totp parameter, for the record I > added a simple curl example below. > > Does anyone know about OIDC providers that support the display=popup > parameter - I could not find a > project / product that supports this. > > Btw. the usage for OTP to verify sensitive operations or transaction > authorization seems indeed to be quite common. > http://www.transmitsecurity.com/use_case/otp-one-time-password/ > https://bankapi.net/docs/public/authorization-getstarted.html > https://www.owasp.org/index.php/Transaction_Authorization_Cheat_Sheet Yeah, I agree. I hope that in future versions, we will have time to add better support for things like step-up authentication, authentication levels, display=popup etc. So the usecase when application wants to display popup window where it needs to re-authenticate against Keycloak server with TOTP (or any other secure auth mechanism) and this popup will be shown for confirm every important transaction on application side, will be possible. Marek > > Cheers, > Thomas > > -- > > KC_REALM=otp-validation-test > KC_USERNAME=tester > KC_PASSWORD=test > KC_OTP_CODE=654328 > KC_CLIENT=test-client > KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 > KC_URL="http://localhost:8081/auth" > > ## Request Tokens for credentials > KC_RESPONSE=$( \ > curl -v \ > -d "username=$KC_USERNAME" \ > -d "password=$KC_PASSWORD" \ > -d 'grant_type=password' \ > -d "client_id=$KC_CLIENT" \ > -d "client_secret=$KC_CLIENT_SECRET" \ > -d "totp=$KC_OTP_CODE" \ > "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" \ > | jq . > ) > > KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) > KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) > KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r .refresh_token) > > ## Validate otp token with custom credential validation endpoint > > KC_OTP_CODE=027253 curl -v \ > -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ > -H "Content-Type: application/json" \ > -d > "[{"\""type"\"":"\""totp"\"","\""value"\"":"\""$KC_OTP_CODE"\""}]" \ > $KC_URL/realms/$KC_REALM/credential-validation > > > 2016-11-11 11:26 GMT+01:00 Marek Posolda >: > > On 11/11/16 11:00, Thomas Darimont wrote: > > Hi Stian, > > thanks for the feedback.I understand the philosophy behind not > exposing > credentials to an application and agree that > one way to solve this is by using a custom endpoint. > > Question: Is there a general recommendation or guideline about > resource > names / paths? E.g. something like custom resources should be > located > beneath a special base path (e.g. /custom) to avoid potential > clashes with > Keycloak resources in newer versions? > > I also like the idea for (partial) re-auth via some sort of > popup / iframe. > Are there any JIRAs logged for this already? > Another thing is that this might not work in every case, e.g. when > integrating Desktop applications where a > the "client" sends the user credentials to a server which then > needs to > check those credentials against keycloak, e.g. by trying to > acquire an > access token. > > One related JIRA is the support for OIDC "display" parameter. This > allows to open login page in the popup (among other things). See > https://issues.jboss.org/browse/KEYCLOAK-3322 > . > > Regarding desktop client, we already have "Direct Grant" > authentication flow, which has some support for OTP verification. > You just need to add "totp" parameter to the request though. See > the ValidateOTP class for details. > > Marek > > > Cheers, > Thomas > > 2016-11-11 8:48 GMT+01:00 Stian Thorgersen > >: > > This is not something we should add IMO. As Bill points > out the whole idea > is to not expose credentials to application. I'd rather > see something that > works without exposing credentials to the users. Could do > a redirect with > re-auth required. Could require just adding otp on > re-auth. Could possibly > do a popup with embedded login from KC. I'm sure there's > cleaner ways than > the applicaiton collection OTP. > > However, it's perfectly possible to implement this using a > custom rest > resource so if you really want it to do this then do it > that way. > > On 10 November 2016 at 21:55, Thomas Darimont > com> wrote: > > Hello guys, > > I agree with you Bill that requiring a user to re-auth > by redirecting to > the > auth-server is often the best thing to do. > However in business applications one often sees the > requirement to > periodically > reauthenticate the user without requiring the user to > login again (e.g. > without leaving the application). > A common example for this is that users need to > reenter a OTP credential > before performing a certain critical operation... > > I gave this a quick spin... > https://github.com/thomasdarimont/keycloak/commit/e891ca604a > > cf5fd494525092c7a88628d9fb20bd > as you can see I took some inspiration from the > userinfo endpoint. > > Works quite well so far, but bruteforce protection is > still missing. > > A demo session via curl looks like this: > > $ KC_REALM=otp-validation-test > KC_USERNAME=tester > KC_PASSWORD=test > KC_CLIENT=test-client > KC_CLIENT_SECRET=c57dc179-09bb-4bb7-9128-91b29dd7fc35 > KC_URL="http://localhost:8081/auth > " > > ## Request Tokens for credentials > KC_RESPONSE=$( \ > curl -v \ > -d "username=$KC_USERNAME" \ > -d "password=$KC_PASSWORD" \ > -d 'grant_type=password' \ > -d "client_id=$KC_CLIENT" \ > -d "client_secret=$KC_CLIENT_SECRET" \ > > "$KC_URL/realms/$KC_REALM/protocol/openid-connect/token" > \ > | jq . > ) > > KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) > KC_ID_TOKEN=$(echo $KC_RESPONSE| jq -r .id_token) > KC_REFRESH_TOKEN=$(echo $KC_RESPONSE| jq -r > .refresh_token) > * Trying 127.0.0.1... > % Total % Received % Xferd Average Speed Time > Time Time > Current > Dload Upload > Total Spent Left > Speed > 0 0 0 0 0 0 0 0 > --:--:-- --:--:-- --:--:-- > 0* Connected to localhost (127.0.0.1) port 8081 (#0) > > POST > /auth/realms/otp-validation-test/protocol/openid-connect/token > > HTTP/1.1 > > Host: localhost:8081 > User-Agent: curl/7.45.0 > Accept: */* > Content-Length: 122 > Content-Type: application/x-www-form-urlencoded > > } [122 bytes data] > * upload completely sent off: 122 out of 122 bytes > < HTTP/1.1 200 OK > < Connection: keep-alive > < Content-Type: application/json > < Content-Length: 3713 > < Date: Thu, 10 Nov 2016 20:32:16 GMT > < > { [3713 bytes data] > > # We got an acces token ... now lets try to validate a > wrong OTP code > > curl -v \ > -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ > -H "Content-Type: application/json" \ > -d '[{"type":"totp", "value":"24861"}]' \ > $KC_URL/realms/$KC_REALM/credential-validation > * Trying 127.0.0.1... > * Connected to localhost (127.0.0.1) port 8081 (#0) > > POST > /auth/realms/otp-validation-test/credential-validation > HTTP/1.1 > Host: localhost:8081 > User-Agent: curl/7.45.0 > Accept: */* > Authorization: Bearer eyJhb...EW3Q > Content-Type: application/json > Content-Length: 34 > > * upload completely sent off: 34 out of 34 bytes > < HTTP/1.1 400 Bad Request > < Connection: keep-alive > < Content-Length: 0 > < Date: Thu, 10 Nov 2016 20:30:42 GMT > < > * Connection #0 to host localhost left intact > > # Let's try a currently valid OTP code > > curl -v \ > -H "Authorization: Bearer $KC_ACCESS_TOKEN" \ > -H "Content-Type: application/json" \ > -d '[{"type":"totp", "value":"949784"}]' \ > $KC_URL/realms/$KC_REALM/credential-validation > * Trying 127.0.0.1... > * Connected to localhost (127.0.0.1) port 8081 (#0) > > POST > /auth/realms/otp-validation-test/credential-validation > HTTP/1.1 > Host: localhost:8081 > User-Agent: curl/7.45.0 > Accept: */* > Authorization: Bearer eyJhb...EW3Q > Content-Type: application/json > Content-Length: 35 > > * upload completely sent off: 35 out of 35 bytes > < HTTP/1.1 200 OK > < Connection: keep-alive > < Content-Length: 0 > < Date: Thu, 10 Nov 2016 20:34:11 GMT > < > * Connection #0 to host localhost left intact > > Cheers, > Thomas > > 2016-11-10 14:27 GMT+01:00 Bill Burke > >: > > Should be generic and not specific to a credential > type. Should also > hook into brute force detection. IMO though, one > of the reasons for SSO > and keycloak is that the application does not > gather credentials. This > is the job of the auth server. IMO, we'd be > better off with expiring > the login at the client side, redirecting to auth > server, auth server > sees that the user session is 3 hours old, and > requests OTP. > > > On 11/10/16 7:52 AM, Thomas Darimont wrote: > > Hello Rohith, > > not that I know of - we'd also like to have > this functionality. > > > What would be the best place to add that? > Perhaps this could be added > > to > > the UsersResource with a new > endpoint like "/users/{userId}/otp-validation" > or a (new) dedicated > resource. > > A client could then do a POST to that > endpoint with the current > > user's > > access token and the entered OTP code. > Keycloak could then lookup and check the > provided otp code. > If the code is corret, response could indicate > that via status HTTP > > 200 > > or > > HTTP 400 otherwise. > > Cheers, > Thomas > > 2016-11-10 12:11 GMT+01:00 gambol > >: > > Hiya > > Does the latest version of Keycloak > provide any means of verifying a > > user's > > TOTP?. Our use-case at the moment, we have > an application which once > > the > > user is authenticated we issue a token of > sorts ... however, we wish > > to > > provide a popup that requests a user's > TOPT every few hours which we > "could" verify via service account ... I > can't see any access at the > > moment > > via the rest api > > Rohith > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > From mposolda at redhat.com Mon Nov 14 02:59:13 2016 From: mposolda at redhat.com (Marek Posolda) Date: Mon, 14 Nov 2016 08:59:13 +0100 Subject: [keycloak-dev] OrientDB Support In-Reply-To: References: Message-ID: <9f3f1fe6-b4db-3fb3-5984-ae197cea367b@redhat.com> Hi, IMO support for OrientDB will be interesting, but I think that we don't want to accept the contribution for it as it will add the overhead of maintain another type of model SPI. So for now, I think that we just stick with JPA and Mongo. Marek On 13/11/16 06:53, Sena Gbeckor-Kove wrote: > Hi, > > If nobody has implemented it yet. I'd be interested in adding OrientDB > support. That way we can avoid having to manage Mongo as well. > > Cheers From sena at ananse.im Mon Nov 14 03:09:22 2016 From: sena at ananse.im (Sena Gbeckor-Kove) Date: Mon, 14 Nov 2016 08:09:22 +0000 Subject: [keycloak-dev] OrientDB Support In-Reply-To: <9f3f1fe6-b4db-3fb3-5984-ae197cea367b@redhat.com> References: <9f3f1fe6-b4db-3fb3-5984-ae197cea367b@redhat.com> Message-ID: Thanks, If there's not going to be much help available I'll have to evaluate whether it's worth doing. It may make sense to try running it in jdbc mode against the jpa tests to see how compliant it is first. S On Mon, 14 Nov 2016, 14:59 Marek Posolda, wrote: > Hi, > > IMO support for OrientDB will be interesting, but I think that we don't > want to accept the contribution for it as it will add the overhead of > maintain another type of model SPI. So for now, I think that we just > stick with JPA and Mongo. > > Marek > > On 13/11/16 06:53, Sena Gbeckor-Kove wrote: > > Hi, > > > > If nobody has implemented it yet. I'd be interested in adding OrientDB > > support. That way we can avoid having to manage Mongo as well. > > > > Cheers > > > -- ----- Sena Gbeckor-Kove Founder & CEO ananse 19-09, 1 North Bridge Road Singapore 179094 m: +65 3163 5466 e: sena at ananse.im w: www.ananse.im From bruno at abstractj.org Mon Nov 14 14:21:07 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Mon, 14 Nov 2016 17:21:07 -0200 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider Message-ID: <20161114192107.GA1030@abstractj.org> Good morning, I believe I found a bug while playing with LDAP Federation provider. But before file any Jira, I would like to confirm. Steps to reproduce: 1. Just follow all the steps described here https://github.com/keycloak/keycloak/tree/master/examples/ldap 2. Change the federation provider to read-only mode 3. Synchronize all users 4. Delete all the roles associated with LDAP (For example: ldap-user, ldap-admin) 5. Now go to "Users" > "Edit" 6. Click on "Role Mappings" You might get an exception like this: https://gist.github.com/abstractj/19eb8e5f25fee3b469110246066ecd08 Is this a bug? -- abstractj PGP: 0x84DC9914 From mposolda at redhat.com Tue Nov 15 06:46:39 2016 From: mposolda at redhat.com (Marek Posolda) Date: Tue, 15 Nov 2016 12:46:39 +0100 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: <20161114192107.GA1030@abstractj.org> References: <20161114192107.GA1030@abstractj.org> Message-ID: Hey Bruno, it seems Bill pushed already some LDAP changes last night and migrated LDAP FederationProvider to UserStorage SPI. Do you have an opportunity to retry with latest master? If it can be still reproduced, then you can create JIRA for 2.4.1 fix version. We already have bunch of LDAP related issues planned for that version, so this can be possibly another one to address. Marek On 14/11/16 20:21, Bruno Oliveira wrote: > Good morning, I believe I found a bug while playing with LDAP Federation > provider. But before file any Jira, I would like to confirm. > > Steps to reproduce: > > 1. Just follow all the steps described here https://github.com/keycloak/keycloak/tree/master/examples/ldap > 2. Change the federation provider to read-only mode > 3. Synchronize all users > 4. Delete all the roles associated with LDAP (For example: ldap-user, > ldap-admin) > 5. Now go to "Users" > "Edit" > 6. Click on "Role Mappings" > > You might get an exception like this: https://gist.github.com/abstractj/19eb8e5f25fee3b469110246066ecd08 > > Is this a bug? > > > -- > > abstractj > PGP: 0x84DC9914 > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From sthorger at redhat.com Wed Nov 16 06:36:33 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Wed, 16 Nov 2016 12:36:33 +0100 Subject: [keycloak-dev] Keycloak is now OpenID Connect certified Message-ID: I'm pleased to announce that Keycloak is now OpenID Connect certified for all 5 conformance profiles. For details check out http://openid.net/certification/. From thomas.raehalme at aitiofinland.com Wed Nov 16 06:38:18 2016 From: thomas.raehalme at aitiofinland.com (Thomas Raehalme) Date: Wed, 16 Nov 2016 13:38:18 +0200 Subject: [keycloak-dev] Keycloak is now OpenID Connect certified In-Reply-To: References: Message-ID: That's awesome! Good work! Best regards, Thomas On Wed, Nov 16, 2016 at 1:36 PM, Stian Thorgersen wrote: > I'm pleased to announce that Keycloak is now OpenID Connect certified for > all 5 conformance profiles. For details check out > http://openid.net/certification/. > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From bruno at abstractj.org Wed Nov 16 08:16:38 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Wed, 16 Nov 2016 11:16:38 -0200 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: References: <20161114192107.GA1030@abstractj.org> Message-ID: <20161116131638.GA31972@abstractj.org> Hi Marek, After rebase against master and go to "Users > Edit" I get: "Resource not found... We could not find the resource you are looking for. Please make sure the URL you entered is correct." I can be wrong, but I believe that if a role is associated to some user, admin should not be able to delete it. Or, if we would like to remove roles, even if they have users associated to this, user should be displayed without these roles. Does it make sense? I can file a jira if we agree on that. On 2016-11-15, Marek Posolda wrote: > Hey Bruno, > > it seems Bill pushed already some LDAP changes last night and migrated LDAP > FederationProvider to UserStorage SPI. Do you have an opportunity to retry > with latest master? If it can be still reproduced, then you can create JIRA > for 2.4.1 fix version. We already have bunch of LDAP related issues planned > for that version, so this can be possibly another one to address. > > Marek > > On 14/11/16 20:21, Bruno Oliveira wrote: > > Good morning, I believe I found a bug while playing with LDAP Federation > > provider. But before file any Jira, I would like to confirm. > > > > Steps to reproduce: > > > > 1. Just follow all the steps described here https://github.com/keycloak/keycloak/tree/master/examples/ldap > > 2. Change the federation provider to read-only mode > > 3. Synchronize all users > > 4. Delete all the roles associated with LDAP (For example: ldap-user, > > ldap-admin) > > 5. Now go to "Users" > "Edit" > > 6. Click on "Role Mappings" > > > > You might get an exception like this: https://gist.github.com/abstractj/19eb8e5f25fee3b469110246066ecd08 > > > > Is this a bug? > > > > > > -- > > > > abstractj > > PGP: 0x84DC9914 > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > -- abstractj PGP: 0x84DC9914 From nielsbne at gmail.com Wed Nov 16 08:29:42 2016 From: nielsbne at gmail.com (Niels Bertram) Date: Wed, 16 Nov 2016 23:29:42 +1000 Subject: [keycloak-dev] Keycloak is now OpenID Connect certified In-Reply-To: References: Message-ID: Great effort guys. Awesome product. On Wed, Nov 16, 2016 at 9:38 PM, Thomas Raehalme < thomas.raehalme at aitiofinland.com> wrote: > That's awesome! Good work! > > Best regards, > Thomas > > > On Wed, Nov 16, 2016 at 1:36 PM, Stian Thorgersen > wrote: > > > I'm pleased to announce that Keycloak is now OpenID Connect certified for > > all 5 conformance profiles. For details check out > > http://openid.net/certification/. > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From thomas.darimont at googlemail.com Wed Nov 16 09:55:36 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Wed, 16 Nov 2016 15:55:36 +0100 Subject: [keycloak-dev] Keycloak is now OpenID Connect certified In-Reply-To: References: Message-ID: Wooho, congratulations! Great Job - keep on the great work! Cheers, Thomas 2016-11-16 14:29 GMT+01:00 Niels Bertram : > Great effort guys. Awesome product. > > On Wed, Nov 16, 2016 at 9:38 PM, Thomas Raehalme < > thomas.raehalme at aitiofinland.com> wrote: > > > That's awesome! Good work! > > > > Best regards, > > Thomas > > > > > > On Wed, Nov 16, 2016 at 1:36 PM, Stian Thorgersen > > wrote: > > > > > I'm pleased to announce that Keycloak is now OpenID Connect certified > for > > > all 5 conformance profiles. For details check out > > > http://openid.net/certification/. > > > _______________________________________________ > > > keycloak-dev mailing list > > > keycloak-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From murrat.demirr at gmail.com Wed Nov 16 09:58:56 2016 From: murrat.demirr at gmail.com (Murat Demir) Date: Wed, 16 Nov 2016 17:58:56 +0300 Subject: [keycloak-dev] Keycloak is now OpenID Connect certified In-Reply-To: References: Message-ID: Congratulations guys. Demir 2016-11-16 17:55 GMT+03:00 Thomas Darimont : > Wooho, congratulations! Great Job - keep on the great work! > > Cheers, > Thomas > > > 2016-11-16 14:29 GMT+01:00 Niels Bertram : > > > Great effort guys. Awesome product. > > > > On Wed, Nov 16, 2016 at 9:38 PM, Thomas Raehalme < > > thomas.raehalme at aitiofinland.com> wrote: > > > > > That's awesome! Good work! > > > > > > Best regards, > > > Thomas > > > > > > > > > On Wed, Nov 16, 2016 at 1:36 PM, Stian Thorgersen > > > > wrote: > > > > > > > I'm pleased to announce that Keycloak is now OpenID Connect certified > > for > > > > all 5 conformance profiles. For details check out > > > > http://openid.net/certification/. > > > > _______________________________________________ > > > > keycloak-dev mailing list > > > > keycloak-dev at lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > > > _______________________________________________ > > > keycloak-dev mailing list > > > keycloak-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > -- Te?ekk?rler, iyi ?al??malar . Murat Demir From abhi.raghav007 at gmail.com Wed Nov 16 11:03:08 2016 From: abhi.raghav007 at gmail.com (abhishek raghav) Date: Wed, 16 Nov 2016 21:33:08 +0530 Subject: [keycloak-dev] Keycloak is now OpenID Connect certified In-Reply-To: References: Message-ID: Wow...Another feather in the cap. Amazing product, truly worth it. Congrats guys...keep up the good work. From e.berdoncesbonelo at campus.tu-berlin.de Wed Nov 16 13:31:06 2016 From: e.berdoncesbonelo at campus.tu-berlin.de (Erik Berdonces Bonelo) Date: Wed, 16 Nov 2016 19:31:06 +0100 Subject: [keycloak-dev] Realm role to see groups and roles (only) Message-ID: Hello everyone, I?m checking the different realm roles (listed here: https://keycloak.gitbooks.io/server-adminstration-guide/content/topics/admin-console-permissions/master-realm.html). I?m trying to allow a user to only view the list of groups and roles available in the realm. I?d like not to allow him to list all the users, as this would be a bit too dangerous. Is there any combination for this? Mostly because with the `view-realm` role I can see the realm roles. However I need to enable `view-users`role ?to also see the groups, and that enables all the users as well. Otherwise I?ll open a feature request to Jira? The idea is that I want to allow a client in the admin API to fetch this information (roles and groups only), so I?d give to the Client?s Service Account the minimum roles, so it can fetch the data. This way, I?d fill up an autocomplete form so the users can setup easily a role mapping between Keycloak and my local website. We have lots of roles, and remembering them by memory it?s not a case! Thanks a lot in advance! ?? Best Regards,? Erik Berdonces Bonelo From mposolda at redhat.com Wed Nov 16 17:32:48 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 16 Nov 2016 23:32:48 +0100 Subject: [keycloak-dev] Issues with clustered invalidation caches In-Reply-To: References: Message-ID: So cache changes are in. Summary of changes: - "realms" and "users" caches are now always local caches (even in cluster) - The replicated cache is used to send the messages between cluster nodes to notify what should be invalidated and the records in "realms" and "users" caches are always invalidated locally and their revisions is increased - I've reused the existing "work" replicated cache to send the messages between cluster nodes. This was already here before. So no needed to add another replicated cache. - I've also tested the cross-DC (2 keycloak nodes not communicating directly between each other, but through external JDG server). It's working fine and invalidation messages are correctly propagated as expected. I've added some temporary documentation here https://github.com/keycloak/keycloak/blob/master/misc/CrossDataCenter.md . However userSessions, offlineSessions and loginFailures doesn't yet work in cross-DC setup. It seems this will need to wait for Keycloak 3.X though... Marek On 04/11/16 09:02, Marek Posolda wrote: > On 03/11/16 14:09, Stian Thorgersen wrote: >> Sounds like a plan to me and I can't see us fixing it in a more >> trivial matter either. >> >> This also aligns better with what we've recently discussed with >> regards to cross DC support. Cross DC support would use JDG and have a >> replicate cache between DCs to send invalidation messages, which is >> exactly what you are proposing so this shoulod be a step towards that. >> It may make sense that you work that in straight away. Basically we >> should support propagating invalidation messages by using an external >> Infinispan/JDG cluster as the store for the cache. > So it seems, we may need something like InvalidationManager SPI, as an > abstraction just for the actual transport (sending/receiving > invalidation messages) ? > > By default, it will just be replicated infinispan cache, but it will be > possible to use something completely different for actual message > transport (JMS messaging, external JDG for cross DC support etc). WDYT? > > Marek >> On 3 November 2016 at 12:06, Marek Posolda > > wrote: >> >> I was looking at the cache issue reported by customer. I found the >> cause >> of it and couple of other related issues: >> >> KEYCLOAK-3857 - Bad performance with clustered invalidation cache when >> updating object >> KEYCLOAK-3858 - Removing model object send lots of invalidation >> messages >> across cluster >> KEYCLOAK-3859 - Lots of userCache invalidation messages when >> invalidating realm >> KEYCLOAK-3860 - All realm users are invalidated from cache when >> removing >> some realm object >> >> >> In shortcut, our cache works fine in local mode. But in cluster, there >> are issues with the invalidation caches . We don't have issues with >> stale entries, but this is purchased but lots of various performance >> issues like: >> >> - There are lots of invalidation messages sent across the cluster >> >> - Eviction on the node, which received invalidation event, is also >> very >> uneffective. For example evicting realm with 1000 roles needs to call >> 1000 predicates, which iterates the cache 1000 times. >> >> - Invalidation cache doesn't allow to differ between the context >> why the >> object was invalidated. For example when I update realm settings on >> node1, I need to invalidate just the CachedRealm object, but not >> all the >> other objects dependent on the realm. However the invalidation event >> received on node2 doesn't know, if I invalidated CachedRealm >> because of >> realm update or because of realm removal. So for more safety, it >> assumes >> removal, which evicts all realm objects! See >> https://issues.jboss.org/browse/KEYCLOAK-3857 >> for details. >> >> - Finally we have the workaround with the "invalidation.key" >> objects in >> our invalidation caches. This is currently needed because when >> invalidating object on node1, the invalidation event is NOT >> received on >> node2 unless the object is there. Hence the workaround with the >> "invalidation.key" records just to avoid this limitation of >> invalidation >> cache. >> >> >> For solve all these issues, I propose: >> - Instead of relying on invalidation caches, we will send notification >> across cluster what happened (eg. message "realm XY was updated"). All >> the nodes will receive this notification and will evict all their >> locally cached objects accordingly and bump their revisions locally. >> This would be much more stable, performant and will allow us to remove >> some workarounds. >> >> Some details: >> >> - The caches "realms" and "users" won't be "invalidation" caches, but >> they will be "local" caches. >> >> - When any object needs to be removed from cache because of some >> reason >> (eg. updating realm), the notification message will be sent from node1 >> to all other cluster nodes. We will use the replicated cache for that. >> Node1 will send the notification message like "realm XY was updated" . >> >> - Other cluster nodes will receive this message and they will locally >> trigger evictions of all the objects dependent on particular realm. In >> case of realm update, it's just the CachedRealm object itself. In case >> of realm removal, it is all realm objects etc. >> >> - Note message will contain also context "realm XY was updated" or >> "realm XY was removed" . Not just "invalidate realm XY". This allows >> much more flexibility and in particular avoid the issues like >> https://issues.jboss.org/browse/KEYCLOAK-3857 >> . >> >> - We already have replicated cache called "work", which we are >> using to >> notify other cluster nodes about various events. So we will just use >> this one. No need to add another replicated cache, we will >> probably just >> need to configure LRU eviction for the existing one. >> >> - Also note that messages will be always received. We won't need >> workaround with "invalidation.key" objects anymore. >> >> - Also we don't need recursive evictions (which has very poor >> performance. See https://issues.jboss.org/browse/KEYCLOAK-3857 >> ), >> because receiving node will know exactly what happened. It will remove >> objects just the same way like the "sender" node. >> >> - Finally the amount of traffic sent across the cluster will be >> much lower. >> >> This sounds like the big step, but IMO it's not that bad :) Note >> that we >> already have all the predicates in place for individual objects. The >> only change will be about sending/receiving notifications across >> cluster. I think I am able to prototype something by tomorrow to >> doublecheck this approach working and then finish it somewhen middle >> next week. WDYT? >> >> Marek >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> >> >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Wed Nov 16 17:35:51 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 16 Nov 2016 23:35:51 +0100 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: <20161116131638.GA31972@abstractj.org> References: <20161114192107.GA1030@abstractj.org> <20161116131638.GA31972@abstractj.org> Message-ID: <530a0d89-4462-fa07-a09b-9fccd9ee9422@redhat.com> On 16/11/16 14:16, Bruno Oliveira wrote: > Hi Marek, > > After rebase against master and go to "Users > Edit" I get: > > "Resource not found... > We could not find the resource you are looking for. Please make sure the URL you entered is correct." > > I can be wrong, but I believe that if a role is associated to some user, > admin should not be able to delete it. > > Or, if we would like to remove roles, even if they have users associated > to this, user should be displayed without these roles. +1 Currently when role is deleted, we update all the users and remove the role mapping from them. If there is an exception for this scenario with LDAP, it is likely a bug. Btv. the "Resource not found..." is often shown in admin console when you have stale browser cache. Couldn't that be the case? Marek > > Does it make sense? I can file a jira if we agree on that. > > > On 2016-11-15, Marek Posolda wrote: >> Hey Bruno, >> >> it seems Bill pushed already some LDAP changes last night and migrated LDAP >> FederationProvider to UserStorage SPI. Do you have an opportunity to retry >> with latest master? If it can be still reproduced, then you can create JIRA >> for 2.4.1 fix version. We already have bunch of LDAP related issues planned >> for that version, so this can be possibly another one to address. >> >> Marek >> >> On 14/11/16 20:21, Bruno Oliveira wrote: >>> Good morning, I believe I found a bug while playing with LDAP Federation >>> provider. But before file any Jira, I would like to confirm. >>> >>> Steps to reproduce: >>> >>> 1. Just follow all the steps described here https://github.com/keycloak/keycloak/tree/master/examples/ldap >>> 2. Change the federation provider to read-only mode >>> 3. Synchronize all users >>> 4. Delete all the roles associated with LDAP (For example: ldap-user, >>> ldap-admin) >>> 5. Now go to "Users" > "Edit" >>> 6. Click on "Role Mappings" >>> >>> You might get an exception like this: https://gist.github.com/abstractj/19eb8e5f25fee3b469110246066ecd08 >>> >>> Is this a bug? >>> >>> >>> -- >>> >>> abstractj >>> PGP: 0x84DC9914 >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > -- > > abstractj > PGP: 0x84DC9914 From glavoie at gmail.com Wed Nov 16 19:43:36 2016 From: glavoie at gmail.com (Gabriel Lavoie) Date: Wed, 16 Nov 2016 19:43:36 -0500 Subject: [keycloak-dev] Performance improvements with large number of realms Message-ID: Hi, following http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html that was discussed in the keycloak-user mailing list, I've been working in the last few weeks on fixing performance issues for Keycloak setup with a very large number of realms (tested with 500-600). As suggested by Stian, I have changes to submit as pull requests that I want to discuss here first to make sure they are acceptable. With this improvements, the following issues were solved in our setup: - Startup time reduced from more than 15 minute to under a minute. - Login time to the admin console reduced from ~5 minutes to ~20 secs when cache is being filled, to under 5 when cache is filled. - Realm creation time reduced from ~1 minute to under 10 secs. 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): >From what I see, the JPA entity model is very disconnected and many entities are retrieved through their ID using NamedQuery. I'm not sure why the mapping was done this way. This causes Hibernate to make a autoFlush check for dirty entities on every query, which is very CPU intensive. There are a few entities that can benefit from having a Collection/Set to their child entities and use it rather than a query to retrieve the required information. Also, with explicit mapping, I foresee multiple explicit em.flush() that could be removed when changing the data. In this changeset I've covered only what was necessary for us, but there are still a few areas that could be improved in my opinion: https://github.com/glavoie/keycloak/commit/93c8056498e38ed075523f5b9b78dbde2814d783 2 - Many missing index for FKs: We are using an Oracle database and indexes are not created automatically for FKs. I've added many but I'm not 100% sure if you want a new Liquibase changeset for every changes that could go in a version or only one per version. I still have one index to fix so don't take this as-is for now: https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e663695218524a0da4ce48a331 3 - Using Set rather than Collection with JPA entities: When using a Collection, Hibernate will delete all rows and re-insert them in the join table to update the content. This behavior was very slow and visible with the "admin" composite role that points to roles for all realms (through the different clients). This was slowing down a lot new realm creation. When using a Set, Hibernate does a better job at tracking changes. https://github.com/glavoie/keycloak/commit/9eb1bf013e80395affeec09625d5ee6157a7084a 4 - "session" cache when testing for "hasRole" with a composite role: The recursive search done KeycloakModelUtils.searchFor() can generate quite a lot of hits on the Infinispan cache when trying to test for role. With 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache just to login to the admin console. I've added a in-memory cache of the retrieved composite RoleModel list in the Infinispan RoleAdapter. As I understand, every instance of a RoleAdapter is bound to a KeycloakSession that can be seen as a transaction/request. Still relying on the delegate object if the RealmAdapter is being invalidated. This is the change that reduced the most the login time to the admin console or the time to obtain a Bearer token for the admin API: https://github.com/glavoie/keycloak/commit/0ab39670525315148761627e7d0f808d4ef9c3c0 5 - realmRevisions cache size: In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions cache creation was moved from standalone.xml to Java code. With our current Keycloak cluster we already added the eviction policy mentioned in the ticket, but had to increase the size to 50000 to support our number of realms. Having this size hardcoded in the Java code becomes a bit inconvenient with the latest versions. I didn't really fix the code as Stian mentioned that this part is being heavily reworked. I would guess this has something to do with http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a743941980bad3dc83be7965 Thank you! Gabriel -- Gabriel Lavoie glavoie at gmail.com From shmuein+keycloak-dev at gmail.com Wed Nov 16 19:52:26 2016 From: shmuein+keycloak-dev at gmail.com (Muein Muzamil) Date: Wed, 16 Nov 2016 18:52:26 -0600 Subject: [keycloak-dev] User SPI Message-ID: Hi all, We have implemented custom authenticators to integrate with an external authentication API. After successful authentication, the API returns user attributes back. Right now we save this user into DB as shown below, which works well. user = userFederationManager.addUser(context.getRealm(), username) user.setEnabled(true); mapUserAttributesToUserModel(user, userAttributes); context.setUser(user); We have some privacy and security related requirements because we which we don't want to store user information in KeyCloak database for a longer period. We were thinking to implement some scheduled job to clean up user data from KeyCloak database but I noticed that in KeyCloak 2.3 new User SPI is introduced which allows users to be pull in without sync in user into KeyCloak database. 1. So I was wondering how can I use this SPI to avoid storing user data in DB. 2. Can I imagine sticking user information in session and returning user information from User SPI? 3. Do we have any sample implementations or documentation available for User SPI? Regards, Muein From bruno at abstractj.org Thu Nov 17 08:43:25 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Thu, 17 Nov 2016 13:43:25 +0000 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: <530a0d89-4462-fa07-a09b-9fccd9ee9422@redhat.com> References: <20161114192107.GA1030@abstractj.org> <20161116131638.GA31972@abstractj.org> <530a0d89-4462-fa07-a09b-9fccd9ee9422@redhat.com> Message-ID: Hi Marek, I believe that's not the case. I tried it again removing any cache in my browser. Anyways, I created this Jira, to not miss this: https://issues.jboss.org/browse/KEYCLOAK-3923 On Wed, Nov 16, 2016 at 8:35 PM Marek Posolda wrote: > On 16/11/16 14:16, Bruno Oliveira wrote: > > Hi Marek, > > > > After rebase against master and go to "Users > Edit" I get: > > > > "Resource not found... > > We could not find the resource you are looking for. Please make sure the > URL you entered is correct." > > > > I can be wrong, but I believe that if a role is associated to some user, > > admin should not be able to delete it. > > > > Or, if we would like to remove roles, even if they have users associated > > to this, user should be displayed without these roles. > +1 > > Currently when role is deleted, we update all the users and remove the > role mapping from them. If there is an exception for this scenario with > LDAP, it is likely a bug. > > Btv. the "Resource not found..." is often shown in admin console when > you have stale browser cache. Couldn't that be the case? > > Marek > > > > > Does it make sense? I can file a jira if we agree on that. > > > > > > On 2016-11-15, Marek Posolda wrote: > >> Hey Bruno, > >> > >> it seems Bill pushed already some LDAP changes last night and migrated > LDAP > >> FederationProvider to UserStorage SPI. Do you have an opportunity to > retry > >> with latest master? If it can be still reproduced, then you can create > JIRA > >> for 2.4.1 fix version. We already have bunch of LDAP related issues > planned > >> for that version, so this can be possibly another one to address. > >> > >> Marek > >> > >> On 14/11/16 20:21, Bruno Oliveira wrote: > >>> Good morning, I believe I found a bug while playing with LDAP > Federation > >>> provider. But before file any Jira, I would like to confirm. > >>> > >>> Steps to reproduce: > >>> > >>> 1. Just follow all the steps described here > https://github.com/keycloak/keycloak/tree/master/examples/ldap > >>> 2. Change the federation provider to read-only mode > >>> 3. Synchronize all users > >>> 4. Delete all the roles associated with LDAP (For example: ldap-user, > >>> ldap-admin) > >>> 5. Now go to "Users" > "Edit" > >>> 6. Click on "Role Mappings" > >>> > >>> You might get an exception like this: > https://gist.github.com/abstractj/19eb8e5f25fee3b469110246066ecd08 > >>> > >>> Is this a bug? > >>> > >>> > >>> -- > >>> > >>> abstractj > >>> PGP: 0x84DC9914 > >>> _______________________________________________ > >>> keycloak-dev mailing list > >>> keycloak-dev at lists.jboss.org > >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev > >> > > -- > > > > abstractj > > PGP: 0x84DC9914 > > > From mposolda at redhat.com Fri Nov 18 15:32:28 2016 From: mposolda at redhat.com (Marek Posolda) Date: Fri, 18 Nov 2016 21:32:28 +0100 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: <20161116131638.GA31972@abstractj.org> References: <20161114192107.GA1030@abstractj.org> <20161116131638.GA31972@abstractj.org> Message-ID: I've reproduced the "Resource not found" in latest master. Should be fixed now. Marek On 16/11/16 14:16, Bruno Oliveira wrote: > After rebase against master and go to "Users > Edit" I get: > > "Resource not found... > We could not find the resource you are looking for. Please make sure the URL you entered is correct." From bburke at redhat.com Fri Nov 18 15:45:58 2016 From: bburke at redhat.com (Bill Burke) Date: Fri, 18 Nov 2016 15:45:58 -0500 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: References: <20161114192107.GA1030@abstractj.org> <20161116131638.GA31972@abstractj.org> Message-ID: This is my port? On 11/18/16 3:32 PM, Marek Posolda wrote: > I've reproduced the "Resource not found" in latest master. Should be > fixed now. > > Marek > > On 16/11/16 14:16, Bruno Oliveira wrote: >> After rebase against master and go to "Users > Edit" I get: >> >> "Resource not found... >> We could not find the resource you are looking for. Please make sure the URL you entered is correct." > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Fri Nov 18 16:11:43 2016 From: mposolda at redhat.com (Marek Posolda) Date: Fri, 18 Nov 2016 22:11:43 +0100 Subject: [keycloak-dev] A probable bug in read-only mode for LDAP Federation Provider In-Reply-To: References: <20161114192107.GA1030@abstractj.org> <20161116131638.GA31972@abstractj.org> Message-ID: <1d208dd7-4f8d-3753-ee27-62fad15f90e8@redhat.com> Yeah, now it's working fine. However I've added workaround with custom REST endpoint, which is able to find either UserStorageProvider model and fallback to UserFederationProvider model (if ComponentModel corresponding to UserStorageProvider is not found). I couldn't figure better way to have admin console working correctly with both UserStorage providers and old UserFederation providers. We can remove workaround once the UserFederation SPI is removed. I've added JIRA to track removing this workaround in next version https://issues.jboss.org/browse/KEYCLOAK-3933 Marek On 18/11/16 21:45, Bill Burke wrote: > This is my port? > > > On 11/18/16 3:32 PM, Marek Posolda wrote: >> I've reproduced the "Resource not found" in latest master. Should be >> fixed now. >> >> Marek >> >> On 16/11/16 14:16, Bruno Oliveira wrote: >>> After rebase against master and go to "Users > Edit" I get: >>> >>> "Resource not found... >>> We could not find the resource you are looking for. Please make sure the URL you entered is correct." >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From amorse at deloitte.com Fri Nov 18 16:26:08 2016 From: amorse at deloitte.com (Morse, Alexander (US - Newton)) Date: Fri, 18 Nov 2016 21:26:08 +0000 Subject: [keycloak-dev] Bug with Users and Groups Using Mongo in v2.30 Message-ID: Hi, Have run into a bug where assigning a user to a group when using Mongo DB does not work in version 2.3.0. Am using the base jboss/keycloak-mongo:2.3.0.Final docker image to run Keycloak. Simply create a group, then a user. Attempting to change the user's group membership to be part of the group yields an error message and no change. Everything seems to work fine when using the plain image with an internal user store. This is not a problem in version 2.2.1 either. Seems to be a regression specific to MongoDB in this version. I have copied the stack trace of the error below. Has this already been reported as a bug? I did not see it. This is the same issue reported by Jitendra Chouhan on the user list. Let me know if you need any more information. Thanks, Alex 21:12:05,078 ERROR [io.undertow.request] (default task-4) UT005023: Exception handling request to /auth/admin/realms/master/users/56e5eecc-85c3-4af5-84f2-84ad59c2c745/groups/a682283f-fd8e-4ce8-ad52-d928fec1d685: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76) at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212) at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90) at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at org.keycloak.models.mongo.keycloak.adapters.UserAdapter.isMemberOf(UserAdapter.java:263) at org.keycloak.models.cache.infinispan.UserAdapter.isMemberOf(UserAdapter.java:368) at org.keycloak.services.resources.admin.UsersResource.joinGroup(UsersResource.java:992) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:395) ... 37 more This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. v.E.1 From mposolda at redhat.com Sun Nov 20 15:00:00 2016 From: mposolda at redhat.com (Marek Posolda) Date: Sun, 20 Nov 2016 21:00:00 +0100 Subject: [keycloak-dev] Bug with Users and Groups Using Mongo in v2.30 In-Reply-To: References: Message-ID: <78527b8b-dac1-a56d-eba9-2345f167cf70@redhat.com> It's fixed in latest master and will be available in 2.4.0.CR1 release next week. Marek On 18/11/16 22:26, Morse, Alexander (US - Newton) wrote: > Hi, > > Have run into a bug where assigning a user to a group when using Mongo DB does not work in version 2.3.0. Am using the base jboss/keycloak-mongo:2.3.0.Final docker image to run Keycloak. Simply create a group, then a user. Attempting to change the user's group membership to be part of the group yields an error message and no change. > > Everything seems to work fine when using the plain image with an internal user store. This is not a problem in version 2.2.1 either. Seems to be a regression specific to MongoDB in this version. I have copied the stack trace of the error below. > > Has this already been reported as a bug? I did not see it. This is the same issue reported by Jitendra Chouhan on the user list. Let me know if you need any more information. > > Thanks, > Alex > > 21:12:05,078 ERROR [io.undertow.request] (default task-4) UT005023: Exception handling request to /auth/admin/realms/master/users/56e5eecc-85c3-4af5-84f2-84ad59c2c745/groups/a682283f-fd8e-4ce8-ad52-d928fec1d685: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException > at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76) > at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212) > at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168) > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411) > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202) > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) > at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) > at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90) > at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) > at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) > at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) > at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) > at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) > at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) > at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) > at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) > at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) > at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) > at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) > at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) > at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) > at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) > at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) > at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) > at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) > at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) > at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at org.keycloak.models.mongo.keycloak.adapters.UserAdapter.isMemberOf(UserAdapter.java:263) > at org.keycloak.models.cache.infinispan.UserAdapter.isMemberOf(UserAdapter.java:368) > at org.keycloak.services.resources.admin.UsersResource.joinGroup(UsersResource.java:992) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) > at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295) > at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249) > at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138) > at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) > at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) > at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107) > at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133) > at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101) > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:395) > ... 37 more > > This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited. > > v.E.1 > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From bburke at redhat.com Mon Nov 21 11:18:25 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 21 Nov 2016 11:18:25 -0500 Subject: [keycloak-dev] session.userLocalStorage() needs to be fronted by cache Message-ID: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> I think I made a mistake when User Storage Providers are importing into local cache. Currently KeycloakSession.userLocalStorage() does not have a cache in front of it. The LDAP and Kerberos providers call this to determine if the user has been imported or not. The thing is, the user may already be cached and I think there is a possibility of updating the user (on demand resync) and getting stale cache entries. Also, we don't want a database hit every time there is validation happening. So, I'm going to figure out a way to have the cache front the userLocalStorage() method too like we do for userStorage(). This will require some refactoring of UserCacheSession. Not sure if that will conflict with Marek's work. Bill From mauriciogiacomini at hotmail.com Mon Nov 21 13:04:38 2016 From: mauriciogiacomini at hotmail.com (=?iso-8859-1?Q?Maur=EDcio_Giacomini_Penteado?=) Date: Mon, 21 Nov 2016 18:04:38 +0000 Subject: [keycloak-dev] RuntimeException: Cannot find KieModule on example authz/photoz Message-ID: Hello everybody I am doing some tests of resources authorization in keycloak but I am having some problems. If I try run authz/photoz example in a windows 7 station with wildfly 10 and Keycloak 2.3.0 all works fine. But using windows 10 or ubuntu 16.04 stations I always receive "RuntimeException: Cannot find KieModule". In all tests I did the same sequence described in https://github.com/keycloak/keycloak/tree/master/examples/authz/photoz/ I do not understanding why just on windows 7 station all works fine. If someone can help me, please let me know. Regards, Mauricio. *A complete stack of the error: 15:20:52,311 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 65) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher) at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:162) at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2209) at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:299) at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:240) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:113) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36) at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117) at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78) at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103) at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:231) at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:132) at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82) ... 6 more Caused by: java.lang.RuntimeException: Cannot find KieModule: org.keycloak:photoz-authz-policy:2.1.0-SNAPSHOT at org.drools.compiler.kie.builder.impl.KieServicesImpl.newKieContainer(KieServicesImpl.java:117) at org.drools.compiler.kie.builder.impl.KieServicesImpl.newKieContainer(KieServicesImpl.java:111) at org.keycloak.authorization.policy.provider.drools.DroolsPolicy.(DroolsPolicy.java:31) at org.keycloak.authorization.policy.provider.drools.DroolsPolicyProviderFactory.update(DroolsPolicyProviderFactory.java:95) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.keycloak.authorization.policy.provider.drools.DroolsPolicyProviderFactory$1.onEvent(DroolsPolicyProviderFactory.java:75) at org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:67) at org.keycloak.services.resources.KeycloakApplication.(KeycloakApplication.java:162) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150) ... 19 more From jcain at redhat.com Mon Nov 21 13:21:11 2016 From: jcain at redhat.com (Josh Cain) Date: Mon, 21 Nov 2016 12:21:11 -0600 Subject: [keycloak-dev] Docker Protocol? In-Reply-To: References: <7e296080-8b29-d70c-e397-376146f023c7@redhat.com> Message-ID: <1479752471.17635.15.camel@redhat.com> Finally got some time to try to wrap this one up, but I had a few questions: - Was looking to add a new section to the server-administration-guide (7.4, under SSO Protocols), but am not sure how to do that using GitBooks. ?I can make comments, but am unsure how a new page would be added in a PR-style flow. ?I don't have permissions to clone the gitbook as a repo, so how would you like me to submit a new gitbook section for this protocol? - Automated testing is going to be a bit of a problem since standing up a docker registry requires either a) system binaries, or b) something like spinning up a docker image. ?Is it acceptable to use the supported java docker client and interact with what's effectively a registry fixture via the a Keycloak Docker client for the sake of integration testing? On Thu, 2016-08-18 at 06:01 +0200, Stian Thorgersen wrote: > I'm good with including it in community as long as it's disabled by > default, documented and tested ;) > > On 17 August 2016 at 16:06, Josh Cain wrote: > > I've been in POC mode lately, so the cursory testing I've done has > > been limited to manual, high-level tests.? I'd be happy to find a > > way to get some useful integrated test for this. > > > > Right now I'm using 3 things to test - a docker client, a docker > > registry, and the keycloak IDP.? Should be able to automate the > > registry container start and there's a java library for docker > > clients that we can use.? So automated integration testing looks > > possible, but I've not gone down that road just yet.? Would > > certainly do so if this was going upstream to Keycloak proper. > > > > > > Josh Cain | Software Applications Engineer > > Identity and Access Management > > Red Hat > > +1 256-452-0150 > > > > On Wed, Aug 17, 2016 at 7:56 AM, Bill Burke > > wrote: > > > Can't you just have "enabled": false in keycloak-server.json for > > > that provider? > > > > > > On 8/17/16 3:41 AM, Stian Thorgersen wrote: > > > > We could have it included in KC, but disabled by default. > > > > Having the ability to enable/disable features is something we > > > > need in either case. Especially for product as we'd like to > > > > have certain community only features (or tech preview features) > > > > disabled by default to make it obvious to customers they are > > > > using unsupported features. > > > > > > > > We'd need to figure out some way to automate testing of it > > > > though. We obviously can't manually test with Docker on a > > > > regular basis. Do you have any plans around how to test it? > > > > > > > > On 16 August 2016 at 21:24, Josh Cain > > > > wrote: > > > > > Couldn't agree more that Docker should use an > > > > > existing/standards-based auth protocol.? Problem is, we need > > > > > our Keycloak IDP to start talking to docker registries.? > > > > > Since we have a hard requirement to talk to docker > > > > > registries, our IDP has to speak docker auth.? There are > > > > > already a number of Docker registry authorization servers out > > > > > there, so there seems to be a need out there in the community > > > > > as well. > > > > > > > > > > We're going to develop one and push it out such that it's > > > > > publicly accessible in any case, I was just curious as to > > > > > whether the dev team would want to incorporate this into the > > > > > product. > > > > > > > > > > > > > > > Josh Cain | Software Applications Engineer > > > > > Identity and Access Management > > > > > Red Hat > > > > > +1 256-452-0150 > > > > > > > > > > On Tue, Aug 16, 2016 at 4:19 AM, Stian Thorgersen > > > > edhat.com> wrote: > > > > > > I'm not sure we'd like to have a Docker registry specific > > > > > > protocol added directly to Keycloak. Seems like Docker > > > > > > registry should rather get their act together and comply > > > > > > with existing protocols rather than invent their own. > > > > > > > > > > > > We did have an idea of creating some repository for > > > > > > extensions. These would be community maintained, not > > > > > > included in product and wouldn't receive the same level of > > > > > > testing. Maybe that would be a good place for this. With > > > > > > Bills recent deployer work it could be easier to allow > > > > > > users to deploy custom extensions as well. > > > > > > > > > > > > On 15 August 2016 at 16:41, Josh Cain > > > > > > wrote: > > > > > > > Hi Stian, > > > > > > > > > > > > > > Docker's auth V2 (docs link above) is Oauth-ish, but > > > > > > > doesn't seem to conform 100% to the specification.? I > > > > > > > started by just trying to stand up an OIDC endpoint to > > > > > > > talk to docker and Keycloak threw a "Missing parameters: > > > > > > > response_type" error.? Turns out, Docker sends the GET > > > > > > > request like this:? > > > > > > > > > > > > > > /auth/realms/redhat-external/protocol/docker- > > > > > > > v2/auth?account=jcain&scope=repository%3Acentos%3Apull&se > > > > > > > rvice=docker-registry > > > > > > > > > > > > > > Not only is the request missing the request_typer > > > > > > > paremeter, but Docker uses different nomenclature than > > > > > > > the OAuth/OIDC standard.? For instance, I would expect > > > > > > > the 'service' param to appear as the client_id param to > > > > > > > conform to the OAuth spec. > > > > > > > > > > > > > > Since it uses different nomenclature, I thought it would > > > > > > > be a much cleaner implementation to just implement it as > > > > > > > its own protocol.? Didn't want to muddy up a clean > > > > > > > OIDC/OAuth implemention. > > > > > > > > > > > > > > Are there workarounds to deal with these differences that > > > > > > > I'm missing? > > > > > > > > > > > > > > > > > > > > > Josh Cain | Software Applications Engineer > > > > > > > Identity and Access Management > > > > > > > Red Hat > > > > > > > +1 843-737-1735 > > > > > > > > > > > > > > On Mon, Aug 15, 2016 at 5:56 AM, Stian Thorgersen > > > > > > er at redhat.com> wrote: > > > > > > > > I'm not sure I fully understand. Are you using a Docker > > > > > > > > client to authenticate with Keycloak? That works with > > > > > > > > the standard OIDC flows, but it requires some > > > > > > > > additional claims in the token which you are adding > > > > > > > > with a protocol mapper? > > > > > > > > > > > > > > > > On 12 August 2016 at 15:31, Josh Cain > > > > > > > .com> wrote: > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > We want to use Keycloak as the IDP/Token issuer for > > > > > > > > > authentication with a docker registry as per the > > > > > > > > > specification found here: > > > > > > > > > > > > > > > > > > https://docs.docker.com/registry/spec/auth/? > > > > > > > > > > > > > > > > > > I've implemented a Protocol Mapper in Keycloak that > > > > > > > > > successfully uses the IDP to perform a login against > > > > > > > > > a registry/docker client.? Is this something that the > > > > > > > > > team is interested in building into the product?? If > > > > > > > > > so, I'd be happy to push back upstream. > > > > > > > > > > > > > > > > > > Josh Cain | Software Applications Engineer > > > > > > > > > Identity and Access Management > > > > > > > > > Red Hat > > > > > > > > > +1 843-737-1735 > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > keycloak-dev mailing list > > > > > > > > > keycloak-dev at lists.jboss.org > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > keycloak-dev mailing list > > > > keycloak-dev at lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > ? > > > > > > _______________________________________________ > > > keycloak-dev mailing list > > > keycloak-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From sthorger at redhat.com Mon Nov 21 13:42:00 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Mon, 21 Nov 2016 19:42:00 +0100 Subject: [keycloak-dev] session.userLocalStorage() needs to be fronted by cache In-Reply-To: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> References: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> Message-ID: We're about to release 2.4.0.CR1. When will this be ready? On 21 November 2016 at 17:18, Bill Burke wrote: > I think I made a mistake when User Storage Providers are importing into > local cache. Currently KeycloakSession.userLocalStorage() does not have > a cache in front of it. The LDAP and Kerberos providers call this to > determine if the user has been imported or not. The thing is, the user > may already be cached and I think there is a possibility of updating the > user (on demand resync) and getting stale cache entries. Also, we don't > want a database hit every time there is validation happening. > > So, I'm going to figure out a way to have the cache front the > userLocalStorage() method too like we do for userStorage(). This will > require some refactoring of UserCacheSession. Not sure if that will > conflict with Marek's work. > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From nafiux at gmail.com Mon Nov 21 14:43:08 2016 From: nafiux at gmail.com (Ignacio Ocampo) Date: Mon, 21 Nov 2016 11:43:08 -0800 Subject: [keycloak-dev] getAuthorizationContext from RefreshableKeycloakSecurityContext with Spring Security Message-ID: Hello Team, I've an Spring Boot application with Spring Security Core, everything is working fine in terms of authentication. The next step is to setup the authorization with "Authorization enabled" in the Client. I've a problem trying to obtain the authorizationContext from RefreshableKeycloakSecurityContext KeycloakSecurityContext keycloakSecurityContext = (KeycloakSecurityContext) request.getAttribute(KeycloakSecurityContext.class.getName()); > org.keycloak.adapters.RefreshableKeycloakSecurityContext at 69d7e12b > AuthorizationContext authzContext = keycloakSecurityContext.getAuthorizationContext(); > null > Could you please help me to understand how I can get the authorization context? In my keycloak.json I have: { > "realm": "MyRealName", > "auth-server-url": "http://myendpoint/auth", > "ssl-required": "none", > "resource": "serviceName", > "credentials": { > "secret": "XXX-XXX-XXX" > }, > "policy-enforcer": { > "enforcement-mode" : "ENFORCING" > } > } Thanks Regards. -- Ignacio Ocampo Mill?n From bburke at redhat.com Mon Nov 21 16:57:06 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 21 Nov 2016 16:57:06 -0500 Subject: [keycloak-dev] can master be bumped to 2.4.1 pom version? Message-ID: Let me know, Bill From bburke at redhat.com Mon Nov 21 16:57:22 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 21 Nov 2016 16:57:22 -0500 Subject: [keycloak-dev] session.userLocalStorage() needs to be fronted by cache In-Reply-To: References: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> Message-ID: <9db41ad7-a0b8-2e08-c2aa-efa2c009fdad@redhat.com> 2.4.1 On 11/21/16 1:42 PM, Stian Thorgersen wrote: > We're about to release 2.4.0.CR1. When will this be ready? > > On 21 November 2016 at 17:18, Bill Burke > wrote: > > I think I made a mistake when User Storage Providers are importing > into > local cache. Currently KeycloakSession.userLocalStorage() does > not have > a cache in front of it. The LDAP and Kerberos providers call this to > determine if the user has been imported or not. The thing is, the > user > may already be cached and I think there is a possibility of > updating the > user (on demand resync) and getting stale cache entries. Also, we > don't > want a database hit every time there is validation happening. > > So, I'm going to figure out a way to have the cache front the > userLocalStorage() method too like we do for userStorage(). This will > require some refactoring of UserCacheSession. Not sure if that will > conflict with Marek's work. > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > From sthorger at redhat.com Mon Nov 21 16:57:41 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Mon, 21 Nov 2016 22:57:41 +0100 Subject: [keycloak-dev] Keycloak 2.4.0.CR1 released Message-ID: We've just released Keycloak 2.4.0.CR1. This release is mainly a maintenance release and we've done a lot of minor improvements and bug fixes. For the full list of issues resolved check out JIRA and to download the release go to the Keycloak homepage . From bburke at redhat.com Mon Nov 21 17:00:20 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 21 Nov 2016 17:00:20 -0500 Subject: [keycloak-dev] Keycloak 2.4.0.CR1 released In-Reply-To: References: Message-ID: <30b31a5c-79f0-39c2-d1d9-77c3025a3bb9@redhat.com> LDAP has been ported to the new UserStorage SPI (next iteration of User Federation Provider). Should be seemless to those using this provider as the data model will be converted when upgrading Keycloak. On 11/21/16 4:57 PM, Stian Thorgersen wrote: > We've just released Keycloak 2.4.0.CR1. This release is mainly a > maintenance release and we've done a lot of minor improvements and bug > fixes. > For the full list of issues resolved check out JIRA > > and > to download the release go to the Keycloak homepage > . > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From nafiux at gmail.com Mon Nov 21 18:01:46 2016 From: nafiux at gmail.com (Ignacio Ocampo) Date: Mon, 21 Nov 2016 15:01:46 -0800 Subject: [keycloak-dev] getAuthorizationContext from RefreshableKeycloakSecurityContext with Spring Security In-Reply-To: References: Message-ID: I've a question, is this issue related? https://issues.jboss.org/browse/KEYCLOAK-3796 On Mon, Nov 21, 2016 at 11:43 AM, Ignacio Ocampo wrote: > Hello Team, > > I've an Spring Boot application with Spring Security Core, everything is > working fine in terms of authentication. > > The next step is to setup the authorization with "Authorization enabled" > in the Client. > > I've a problem trying to obtain the authorizationContext from > RefreshableKeycloakSecurityContext > > KeycloakSecurityContext keycloakSecurityContext = > (KeycloakSecurityContext) request.getAttribute( > KeycloakSecurityContext.class.getName()); > >> org.keycloak.adapters.RefreshableKeycloakSecurityContext at 69d7e12b >> > > AuthorizationContext authzContext = keycloakSecurityContext. > getAuthorizationContext(); > >> null >> > Could you please help me to understand how I can get the authorization > context? > > In my keycloak.json I have: > > { >> "realm": "MyRealName", >> "auth-server-url": "http://myendpoint/auth", >> "ssl-required": "none", >> "resource": "serviceName", >> "credentials": { >> "secret": "XXX-XXX-XXX" >> }, >> "policy-enforcer": { >> "enforcement-mode" : "ENFORCING" >> } >> } > > > Thanks > > Regards. > > -- > Ignacio Ocampo Mill?n > -- Ignacio Ocampo Mill?n From shmuein+keycloak-dev at gmail.com Mon Nov 21 19:39:01 2016 From: shmuein+keycloak-dev at gmail.com (Muein Muzamil) Date: Mon, 21 Nov 2016 18:39:01 -0600 Subject: [keycloak-dev] APIs to manage authorization policies and permissions Message-ID: Hi all, I am trying to integrate KeyCloak with our application, For that purpose, I need to be able to call KeyCloak APIs to manage authorization information. I came across protection API which allows you to manage resource and scopes. I am wondering, do we also have any APIs available to manage authorization policies and permissions? Regards, Muein From shmuein+keycloak-dev at gmail.com Mon Nov 21 19:40:25 2016 From: shmuein+keycloak-dev at gmail.com (Muein Muzamil) Date: Mon, 21 Nov 2016 18:40:25 -0600 Subject: [keycloak-dev] User SPI In-Reply-To: References: Message-ID: Hi all, A gentle reminder to my query regarding User SPI. Regards, Muein On Wed, Nov 16, 2016 at 6:52 PM, Muein Muzamil < shmuein+keycloak-dev at gmail.com> wrote: > Hi all, > > We have implemented custom authenticators to integrate with an external > authentication API. After successful authentication, the API returns user > attributes back. Right now we save this user into DB as shown below, which > works well. > > user = userFederationManager.addUser(context.getRealm(), username) > user.setEnabled(true); > > mapUserAttributesToUserModel(user, userAttributes); > > context.setUser(user); > > > We have some privacy and security related requirements because we which we > don't want to store user information in KeyCloak database for a longer > period. We were thinking to implement some scheduled job to clean up user > data from KeyCloak database but I noticed that in KeyCloak 2.3 new User SPI > is introduced which allows users to be pull in without sync in user into > KeyCloak database. > > 1. So I was wondering how can I use this SPI to avoid storing user > data in DB. > 2. Can I imagine sticking user information in session and returning > user information from User SPI? > 3. Do we have any sample implementations or documentation available > for User SPI? > > Regards, > Muein > From mposolda at redhat.com Tue Nov 22 03:09:58 2016 From: mposolda at redhat.com (Marek Posolda) Date: Tue, 22 Nov 2016 09:09:58 +0100 Subject: [keycloak-dev] session.userLocalStorage() needs to be fronted by cache In-Reply-To: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> References: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> Message-ID: My stuff is already in. So the conflict shouldn't be there though. The main thing is, that when you invalidate something, there needs to be put new entry to "invalidationEvents" set, which are events to be sent to other cluster nodes. The existing methods (eg. registerUserInvalidation) are already doing it. This will work regardless if cache is fronted with userStorage() or userLocalStorage() . Marek On 21/11/16 17:18, Bill Burke wrote: > I think I made a mistake when User Storage Providers are importing into > local cache. Currently KeycloakSession.userLocalStorage() does not have > a cache in front of it. The LDAP and Kerberos providers call this to > determine if the user has been imported or not. The thing is, the user > may already be cached and I think there is a possibility of updating the > user (on demand resync) and getting stale cache entries. Also, we don't > want a database hit every time there is validation happening. > > So, I'm going to figure out a way to have the cache front the > userLocalStorage() method too like we do for userStorage(). This will > require some refactoring of UserCacheSession. Not sure if that will > conflict with Marek's work. > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From psilva at redhat.com Tue Nov 22 06:21:28 2016 From: psilva at redhat.com (Pedro Igor) Date: Tue, 22 Nov 2016 09:21:28 -0200 Subject: [keycloak-dev] APIs to manage authorization policies and permissions In-Reply-To: References: Message-ID: Hi Muein, ? ? We do have the API you are looking for accessible from Keycloak Admin Client. Which is basically the same API used by the Admin Console and integration tests.? ? ? The Protection API is a UMA-compliant API. Spec-wise, UMA doesn't provide an API for policy management. ? ? You try to use the Admin Client, but in the future we want to provide a more developer-friendly API and make it accessible through the Authorization Client. There is a?KEYCLOAK-3135 [https://issues.jboss.org/browse/KEYCLOAK-3135]?for this. Feel free to put there what you need to get this integration done. Regards. On 11/21/2016 10:42:06 PM, Muein Muzamil wrote: Hi all, I am trying to integrate KeyCloak with our application, For that purpose, I need to be able to call KeyCloak APIs to manage authorization information. I came across protection API which allows you to manage resource and scopes. I am wondering, do we also have any APIs available to manage authorization policies and permissions? Regards, Muein _______________________________________________ keycloak-dev mailing list keycloak-dev at lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev From psilva at redhat.com Tue Nov 22 06:49:02 2016 From: psilva at redhat.com (Pedro Igor) Date: Tue, 22 Nov 2016 09:49:02 -0200 Subject: [keycloak-dev] getAuthorizationContext from RefreshableKeycloakSecurityContext with Spring Security In-Reply-To: References: Message-ID: Hey Ignacio, It is more related with?KEYCLOAK-3662 [https://issues.jboss.org/browse/KEYCLOAK-3662].?I think we don't have authz integrated with Spring adapters, yet. Will talk with Sebastien. That should explain why you are not getting that instance. Regards. Pedro Igor On 11/21/2016 9:04:00 PM, Ignacio Ocampo wrote: I've a question, is this issue related? https://issues.jboss.org/browse/KEYCLOAK-3796 On Mon, Nov 21, 2016 at 11:43 AM, Ignacio Ocampo wrote: > Hello Team, > > I've an Spring Boot application with Spring Security Core, everything is > working fine in terms of authentication. > > The next step is to setup the authorization with "Authorization enabled" > in the Client. > > I've a problem trying to obtain the authorizationContext from > RefreshableKeycloakSecurityContext > > KeycloakSecurityContext keycloakSecurityContext = > (KeycloakSecurityContext) request.getAttribute( > KeycloakSecurityContext.class.getName()); > >> org.keycloak.adapters.RefreshableKeycloakSecurityContext at 69d7e12b >> > > AuthorizationContext authzContext = keycloakSecurityContext. > getAuthorizationContext(); > >> null >> > Could you please help me to understand how I can get the authorization > context? > > In my keycloak.json I have: > > { >> "realm": "MyRealName", >> "auth-server-url": "http://myendpoint/auth", >> "ssl-required": "none", >> "resource": "serviceName", >> "credentials": { >> "secret": "XXX-XXX-XXX" >> }, >> "policy-enforcer": { >> "enforcement-mode" : "ENFORCING" >> } >> } > > > Thanks > > Regards. > > -- > Ignacio Ocampo Mill?n > -- Ignacio Ocampo Mill?n _______________________________________________ keycloak-dev mailing list keycloak-dev at lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev From sthorger at redhat.com Tue Nov 22 08:53:47 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Tue, 22 Nov 2016 14:53:47 +0100 Subject: [keycloak-dev] session.userLocalStorage() needs to be fronted by cache In-Reply-To: <9db41ad7-a0b8-2e08-c2aa-efa2c009fdad@redhat.com> References: <07188a4a-c88f-0603-e3f7-2cd6a0c58c53@redhat.com> <9db41ad7-a0b8-2e08-c2aa-efa2c009fdad@redhat.com> Message-ID: Can you create a JIRA for this please? On 21 November 2016 at 22:57, Bill Burke wrote: > 2.4.1 > > On 11/21/16 1:42 PM, Stian Thorgersen wrote: > > We're about to release 2.4.0.CR1. When will this be ready? > > On 21 November 2016 at 17:18, Bill Burke wrote: > >> I think I made a mistake when User Storage Providers are importing into >> local cache. Currently KeycloakSession.userLocalStorage() does not have >> a cache in front of it. The LDAP and Kerberos providers call this to >> determine if the user has been imported or not. The thing is, the user >> may already be cached and I think there is a possibility of updating the >> user (on demand resync) and getting stale cache entries. Also, we don't >> want a database hit every time there is validation happening. >> >> So, I'm going to figure out a way to have the cache front the >> userLocalStorage() method too like we do for userStorage(). This will >> require some refactoring of UserCacheSession. Not sure if that will >> conflict with Marek's work. >> >> Bill >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > > From sthorger at redhat.com Tue Nov 22 08:57:05 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Tue, 22 Nov 2016 14:57:05 +0100 Subject: [keycloak-dev] can master be bumped to 2.4.1 pom version? In-Reply-To: References: Message-ID: Forgot this as usual - changes on the way https://github.com/keycloak/keycloak/pull/3538 On 21 November 2016 at 22:57, Bill Burke wrote: > Let me know, > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From bburke at redhat.com Tue Nov 22 08:58:44 2016 From: bburke at redhat.com (Bill Burke) Date: Tue, 22 Nov 2016 08:58:44 -0500 Subject: [keycloak-dev] User SPI In-Reply-To: References: Message-ID: <9ac259d2-7540-e2cc-b0c4-7b0e2aaa3cc8@redhat.com> Documentation is coming soon in 2.4.1. Sorry for delay. Just in the middle of porting the LDAP and other providers to new SPI. https://github.com/keycloak/keycloak/tree/master/examples/providers/user-storage-jpa Example needs to be updated to tell you you need to configure ExampleDS or KeycloakDS as an xa-datasource. Wildfly barfs when you two datasources in the same transaction and neither are XA enabled. On 11/21/16 7:40 PM, Muein Muzamil wrote: > Hi all, > > A gentle reminder to my query regarding User SPI. > > Regards, > Muein > > On Wed, Nov 16, 2016 at 6:52 PM, Muein Muzamil < > shmuein+keycloak-dev at gmail.com> wrote: > >> Hi all, >> >> We have implemented custom authenticators to integrate with an external >> authentication API. After successful authentication, the API returns user >> attributes back. Right now we save this user into DB as shown below, which >> works well. >> >> user = userFederationManager.addUser(context.getRealm(), username) >> user.setEnabled(true); >> >> mapUserAttributesToUserModel(user, userAttributes); >> >> context.setUser(user); >> >> >> We have some privacy and security related requirements because we which we >> don't want to store user information in KeyCloak database for a longer >> period. We were thinking to implement some scheduled job to clean up user >> data from KeyCloak database but I noticed that in KeyCloak 2.3 new User SPI >> is introduced which allows users to be pull in without sync in user into >> KeyCloak database. >> >> 1. So I was wondering how can I use this SPI to avoid storing user >> data in DB. >> 2. Can I imagine sticking user information in session and returning >> user information from User SPI? >> 3. Do we have any sample implementations or documentation available >> for User SPI? >> >> Regards, >> Muein >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From bburke at redhat.com Tue Nov 22 10:08:43 2016 From: bburke at redhat.com (Bill Burke) Date: Tue, 22 Nov 2016 10:08:43 -0500 Subject: [keycloak-dev] cluster invalidation tests failing Message-ID: <94d6e55b-ec23-9190-81b4-9d19dfcc710c@redhat.com> Every travis build is failing with this: Tests in error: ClientInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds ClientInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds GroupInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds GroupInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds RealmInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds RealmInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds RoleInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds RoleInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds SessionFailoverClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds UserInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds UserInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds From mposolda at redhat.com Tue Nov 22 10:16:31 2016 From: mposolda at redhat.com (Marek Posolda) Date: Tue, 22 Nov 2016 16:16:31 +0100 Subject: [keycloak-dev] cluster invalidation tests failing In-Reply-To: <94d6e55b-ec23-9190-81b4-9d19dfcc710c@redhat.com> References: <94d6e55b-ec23-9190-81b4-9d19dfcc710c@redhat.com> Message-ID: <3ed0dc96-c343-1edf-37e2-72644bdce3ce@redhat.com> Yeah, I've did some workaround to fix that: https://github.com/keycloak/keycloak/pull/3539 JIRA with details: https://issues.jboss.org/browse/KEYCLOAK-3962 Hope workaround helps, Marek On 22/11/16 16:08, Bill Burke wrote: > Every travis build is failing with this: > > Tests in error: > ClientInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > ClientInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > GroupInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > GroupInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > RealmInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > RealmInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > RoleInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > RoleInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > SessionFailoverClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > UserInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > UserInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Tue Nov 22 10:37:08 2016 From: mposolda at redhat.com (Marek Posolda) Date: Tue, 22 Nov 2016 16:37:08 +0100 Subject: [keycloak-dev] cluster invalidation tests failing In-Reply-To: <3ed0dc96-c343-1edf-37e2-72644bdce3ce@redhat.com> References: <94d6e55b-ec23-9190-81b4-9d19dfcc710c@redhat.com> <3ed0dc96-c343-1edf-37e2-72644bdce3ce@redhat.com> Message-ID: Should be fixed in master now. But not sure if all existing PRs should be rebased to have the travis fix visible. I've tried to rerun one of previously failed travis builds to doublecheck that.. Marek On 22/11/16 16:16, Marek Posolda wrote: > Yeah, I've did some workaround to fix that: > https://github.com/keycloak/keycloak/pull/3539 > > JIRA with details: https://issues.jboss.org/browse/KEYCLOAK-3962 > > Hope workaround helps, > Marek > > On 22/11/16 16:08, Bill Burke wrote: >> Every travis build is failing with this: >> >> Tests in error: >> ClientInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> ClientInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> GroupInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> GroupInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> RealmInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> RealmInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> RoleInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> RoleInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> SessionFailoverClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> UserInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> UserInvalidationClusterTest>AbstractClusterTest.beforeClusterTest:122->AbstractClusterTest.logFailoverSetup:60->AbstractClusterTest.getCurrentFailNode:49->AbstractClusterTest.backendNode:85 ? IndexOutOfBounds >> >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mauriciogiacomini at hotmail.com Tue Nov 22 13:08:39 2016 From: mauriciogiacomini at hotmail.com (=?iso-8859-1?Q?Maur=EDcio_Giacomini_Penteado?=) Date: Tue, 22 Nov 2016 18:08:39 +0000 Subject: [keycloak-dev] RuntimeException: Cannot find KieModule on example authz/photoz In-Reply-To: References: Message-ID: Hello I would like inform that my problem was in maven cache. Deleting the folders photoz-authz-policy, photoz-html5-client and photoz-restful-api from $M2_HOME\repository\org\keycloak and executing the commands to rebuild using mvn with "-U" parameter all works fine. Ex: cd keycloak/examples/authz/photoz/ mvn -U clean install cd photoz-restful-api/ mvn -U clean package wildfly:deploy cd .. cd photoz-html5-client mvn -U clean package wildfly:deploy I would ask a correction in branch 2.3.x the file https://github.com/keycloak/keycloak/blob/2.3.x/examples/authz/photoz/photoz-restful-api-authz-service.json The "Only Owner Policy" block is relationed with "mavenArtifactVersion": "2.1.0-SNAPSHOT". I think that it should be corrected to 2.3.0.Final version. Regards, Maur?cio. ________________________________ De: keycloak-dev-bounces at lists.jboss.org em nome de Maur?cio Giacomini Penteado Enviado: segunda-feira, 21 de novembro de 2016 18:04 Para: keycloak-dev at lists.jboss.org Assunto: [keycloak-dev] RuntimeException: Cannot find KieModule on example authz/photoz Hello everybody I am doing some tests of resources authorization in keycloak but I am having some problems. If I try run authz/photoz example in a windows 7 station with wildfly 10 and Keycloak 2.3.0 all works fine. But using windows 10 or ubuntu 16.04 stations I always receive "RuntimeException: Cannot find KieModule". In all tests I did the same sequence described in https://github.com/keycloak/keycloak/tree/master/examples/authz/photoz/ I do not understanding why just on windows 7 station all works fine. If someone can help me, please let me know. Regards, Mauricio. *A complete stack of the error: 15:20:52,311 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 65) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher) at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:162) at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2209) at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:299) at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:240) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:113) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36) at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117) at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78) at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103) at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:231) at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:132) at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82) ... 6 more Caused by: java.lang.RuntimeException: Cannot find KieModule: org.keycloak:photoz-authz-policy:2.1.0-SNAPSHOT at org.drools.compiler.kie.builder.impl.KieServicesImpl.newKieContainer(KieServicesImpl.java:117) at org.drools.compiler.kie.builder.impl.KieServicesImpl.newKieContainer(KieServicesImpl.java:111) at org.keycloak.authorization.policy.provider.drools.DroolsPolicy.(DroolsPolicy.java:31) at org.keycloak.authorization.policy.provider.drools.DroolsPolicyProviderFactory.update(DroolsPolicyProviderFactory.java:95) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.keycloak.authorization.policy.provider.drools.DroolsPolicyProviderFactory$1.onEvent(DroolsPolicyProviderFactory.java:75) at org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:67) at org.keycloak.services.resources.KeycloakApplication.(KeycloakApplication.java:162) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150) ... 19 more _______________________________________________ keycloak-dev mailing list keycloak-dev at lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev keycloak-dev Info Page - lists.jboss.org lists.jboss.org To see the collection of prior postings to the list, visit the keycloak-dev Archives. Using keycloak-dev: To post a message to all the list members ... From psilva at redhat.com Tue Nov 22 15:20:14 2016 From: psilva at redhat.com (Pedro Igor) Date: Tue, 22 Nov 2016 18:20:14 -0200 Subject: [keycloak-dev] RuntimeException: Cannot find KieModule on example authz/photoz In-Reply-To: References: Message-ID: Good to know. Btw, we have changed the photoz example to update the policy artifact version during the build to avoid errors like this one. On 11/22/2016 4:12:22 PM, Maur?cio Giacomini Penteado wrote: Hello I would like inform that my problem was in maven cache. Deleting the folders photoz-authz-policy, photoz-html5-client and photoz-restful-api from $M2_HOME\repository\org\keycloak and executing the commands to rebuild using mvn with "-U" parameter all works fine. Ex: cd keycloak/examples/authz/photoz/ mvn -U clean install cd photoz-restful-api/ mvn -U clean package wildfly:deploy cd .. cd photoz-html5-client mvn -U clean package wildfly:deploy I would ask a correction in branch 2.3.x the file https://github.com/keycloak/keycloak/blob/2.3.x/examples/authz/photoz/photoz-restful-api-authz-service.json The "Only Owner Policy" block is relationed with "mavenArtifactVersion": "2.1.0-SNAPSHOT". I think that it should be corrected to 2.3.0.Final version. Regards, Maur?cio. ________________________________ De: keycloak-dev-bounces at lists.jboss.org em nome de Maur?cio Giacomini Penteado Enviado: segunda-feira, 21 de novembro de 2016 18:04 Para: keycloak-dev at lists.jboss.org Assunto: [keycloak-dev] RuntimeException: Cannot find KieModule on example authz/photoz Hello everybody I am doing some tests of resources authorization in keycloak but I am having some problems. If I try run authz/photoz example in a windows 7 station with wildfly 10 and Keycloak 2.3.0 all works fine. But using windows 10 or ubuntu 16.04 stations I always receive "RuntimeException: Cannot find KieModule". In all tests I did the same sequence described in https://github.com/keycloak/keycloak/tree/master/examples/authz/photoz/ I do not understanding why just on windows 7 station all works fine. If someone can help me, please let me know. Regards, Mauricio. *A complete stack of the error: 15:20:52,311 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 65) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:85) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher) at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:162) at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2209) at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:299) at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:240) at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:113) at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36) at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117) at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78) at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103) at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:231) at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:132) at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:526) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:101) at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82) ... 6 more Caused by: java.lang.RuntimeException: Cannot find KieModule: org.keycloak:photoz-authz-policy:2.1.0-SNAPSHOT at org.drools.compiler.kie.builder.impl.KieServicesImpl.newKieContainer(KieServicesImpl.java:117) at org.drools.compiler.kie.builder.impl.KieServicesImpl.newKieContainer(KieServicesImpl.java:111) at org.keycloak.authorization.policy.provider.drools.DroolsPolicy.(DroolsPolicy.java:31) at org.keycloak.authorization.policy.provider.drools.DroolsPolicyProviderFactory.update(DroolsPolicyProviderFactory.java:95) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.keycloak.authorization.policy.provider.drools.DroolsPolicyProviderFactory$1.onEvent(DroolsPolicyProviderFactory.java:75) at org.keycloak.services.DefaultKeycloakSessionFactory.publish(DefaultKeycloakSessionFactory.java:67) at org.keycloak.services.resources.KeycloakApplication.(KeycloakApplication.java:162) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150) ... 19 more _______________________________________________ keycloak-dev mailing list keycloak-dev at lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev keycloak-dev Info Page - lists.jboss.org lists.jboss.org To see the collection of prior postings to the list, visit the keycloak-dev Archives. Using keycloak-dev: To post a message to all the list members ... _______________________________________________ keycloak-dev mailing list keycloak-dev at lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev From bburke at redhat.com Tue Nov 22 15:55:53 2016 From: bburke at redhat.com (Bill Burke) Date: Tue, 22 Nov 2016 15:55:53 -0500 Subject: [keycloak-dev] issue supporting older migrations Message-ID: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> In removing UserFederationModel I found that we have generic Migration objects that use this api to upgrade LDAP. Specifically in 1.3.0 1.4.0 I vote we don't support migration anymore for anything older than 1.9.8 From ronyjoy at gmail.com Wed Nov 23 00:35:27 2016 From: ronyjoy at gmail.com (rony joy) Date: Wed, 23 Nov 2016 05:35:27 +0000 Subject: [keycloak-dev] Saml authentication Signature verification Exception when Special Characters is the username Message-ID: Hi All, We are getting signature verification exception at the client side after Idp successfully authenticated the user("Ro????????????????") when the user id contains special characters. *UserName : Ro????????????????* *Following are the keycloak settings. * *Encryption req: false* *Sign Document : true* *Please find the below exception at the client side* 05:25:23at org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler.verifyPostBindingSignature(AbstractSamlAuthenticationHandler.java:480)se signature: org.keycloak.com mon.Veriat org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler.validateSamlSignature(AbstractSamlAuthenticationHandler.java:261) at org.keycloak.adapters.saml.profile.webbrowsersso.SamlEndpoint.handle(SamlEndpoint.java:44)stractSamlAuthenticationHandler.java:183) at org.keycloak.adapters.saml.undertow.AbstractSamlAuthMech.authenticate(AbstractSamlAuthMech.java:115) at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:263) at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)31) at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)9) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)(ServletAuthenticationCallHandler.java:55) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)3) at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)ndler.java:64) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)java:292) at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)tupAction.java:48) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)44) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) From nafiux at gmail.com Wed Nov 23 01:53:09 2016 From: nafiux at gmail.com (Ignacio Ocampo) Date: Tue, 22 Nov 2016 22:53:09 -0800 Subject: [keycloak-dev] getAuthorizationContext from RefreshableKeycloakSecurityContext with Spring Security In-Reply-To: References: Message-ID: Hello Pedro, I have implemented something to solve my requirement, it was in a Zuul filter: @Override > public Object run() { > RequestContext ctx = RequestContext.getCurrentContext(); > HttpServletRequest request = ctx.getRequest(); > HttpServletResponse response = ctx.getResponse(); > KeycloakSecurityContext keycloakSecurityContext = > (KeycloakSecurityContext) > request.getAttribute(KeycloakSecurityContext.class.getName()); > OIDCServletHttpFacade facade = new OIDCServletHttpFacade(request, > response); > AuthenticatedActionsHandler actions = new > AuthenticatedActionsHandler(keycloakSecurityContext.deployment, facade); > if(actions.handledRequest()) { // true if !isAuthorized() > // disable route filters > ctx.setSendZuulResponse(); > } > return null; > } Dependencies for keycloakVersion = '2.3.0.Final' compile > "org.keycloak:keycloak-spring-security-adapter:${keycloakVersion}" > compile "org.keycloak:keycloak-authz-client:${keycloakVersion}" > compile "org.keycloak:keycloak-adapter-core:${keycloakVersion}" > compile > "org.keycloak:keycloak-servlet-filter-adapter:${keycloakVersion}" This is working perfectly! But I don't understand so much in deep what is behind the code that I did. Does it make sense for you? Thank you very much. Regards. On Tue, Nov 22, 2016 at 3:49 AM, Pedro Igor wrote: > Hey Ignacio, > > It is more related with KEYCLOAK-3662 > . I think we don't have > authz integrated with Spring adapters, yet. Will talk with Sebastien. > > That should explain why you are not getting that instance. > > Regards. > Pedro Igor > > On 11/21/2016 9:04:00 PM, Ignacio Ocampo wrote: > I've a question, is this issue related? > https://issues.jboss.org/browse/KEYCLOAK-3796 > > On Mon, Nov 21, 2016 at 11:43 AM, Ignacio Ocampo wrote: > > > Hello Team, > > > > I've an Spring Boot application with Spring Security Core, everything is > > working fine in terms of authentication. > > > > The next step is to setup the authorization with "Authorization enabled" > > in the Client. > > > > I've a problem trying to obtain the authorizationContext from > > RefreshableKeycloakSecurityContext > > > > KeycloakSecurityContext keycloakSecurityContext = > > (KeycloakSecurityContext) request.getAttribute( > > KeycloakSecurityContext.class.getName()); > > > >> org.keycloak.adapters.RefreshableKeycloakSecurityContext at 69d7e12b > >> > > > > AuthorizationContext authzContext = keycloakSecurityContext. > > getAuthorizationContext(); > > > >> null > >> > > Could you please help me to understand how I can get the authorization > > context? > > > > In my keycloak.json I have: > > > > { > >> "realm": "MyRealName", > >> "auth-server-url": "http://myendpoint/auth", > >> "ssl-required": "none", > >> "resource": "serviceName", > >> "credentials": { > >> "secret": "XXX-XXX-XXX" > >> }, > >> "policy-enforcer": { > >> "enforcement-mode" : "ENFORCING" > >> } > >> } > > > > > > Thanks > > > > Regards. > > > > -- > > Ignacio Ocampo Mill?n > > > > > > -- > Ignacio Ocampo Mill?n > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > -- Ignacio Ocampo Mill?n From hmlnarik at redhat.com Wed Nov 23 02:35:21 2016 From: hmlnarik at redhat.com (Hynek Mlnarik) Date: Wed, 23 Nov 2016 08:35:21 +0100 Subject: [keycloak-dev] Saml authentication Signature verification Exception when Special Characters is the username In-Reply-To: References: Message-ID: Could you please create JIRA? On Wed, Nov 23, 2016 at 6:35 AM, rony joy wrote: > Hi All, > > We are getting signature verification exception at the client side after > Idp successfully authenticated the user("Ro????????????????") when the user > id contains special characters. > > *UserName : Ro????????????????* > *Following are the keycloak settings. * > *Encryption req: false* > *Sign Document : true* > > *Please find the below exception at the client side* > > 05:25:23at > org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler. > verifyPostBindingSignature(AbstractSamlAuthenticationHandler.java:480)se > signature: org.keycloak.com > mon.Veriat > org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHand > ler.validateSamlSignature(AbstractSamlAuthenticationHandler.java:261) > at > org.keycloak.adapters.saml.profile.webbrowsersso.SamlEndpoint.handle( > SamlEndpoint.java:44)stractSamlAuthenticationHandler.java:183) > at > org.keycloak.adapters.saml.undertow.AbstractSamlAuthMech.authenticate( > AbstractSamlAuthMech.java:115) > at > io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition( > SecurityContextImpl.java:263) > at > io.undertow.security.impl.SecurityContextImpl.attemptAuthentication( > SecurityContextImpl.java:125)31) > at > io.undertow.security.impl.SecurityContextImpl.authenticate( > SecurityContextImpl.java:92)9) > at > io.undertow.server.handlers.PredicateHandler.handleRequest( > PredicateHandler.java:43)(ServletAuthenticationCallHandler.java:55) > at > io.undertow.security.handlers.AbstractConfidentialityHandler > .handleRequest(AbstractConfidentialityHandler.java:46)3) > at > io.undertow.servlet.handlers.security.ServletSecurityConstraintHandl > er.handleRequest(ServletSecurityConstraintHandler.java:59)ndler.java:64) > at > io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHand > ler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > at > io.undertow.security.handlers.AbstractSecurityContextAssocia > tionHandler.handleRequest(AbstractSecurityContextAssocia > tionHandler.java:43) > at > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler. > handleRequest(JACCContextIdHandler.java:61) > at > io.undertow.server.handlers.PredicateHandler.handleRequest( > PredicateHandler.java:43) > at > io.undertow.servlet.handlers.ServletInitialHandler.access$ > 100(ServletInitialHandler.java:81)java:292) > at > io.undertow.servlet.handlers.ServletInitialHandler$2.call( > ServletInitialHandler.java:135) > at > io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call( > ContextClassLoaderSetupAction.java:43)tupAction.java:48) > at > io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call( > LegacyThreadSetupActionWrapper.java:44) > at > io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call( > LegacyThreadSetupActionWrapper.java:44) > at > io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( > ServletInitialHandler.java:272)44) > at > io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest( > ServletInitialHandler.java:104) > at > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev -- --Hynek From sthorger at redhat.com Wed Nov 23 04:05:54 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Wed, 23 Nov 2016 10:05:54 +0100 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> Message-ID: I vote against that. There's a lot of users that are still on older versions. We could ask them to upgrade to 1.9.8 first, but there's a chance that could fail (we've seen this a few times) and as they can't upgrade to the most recent version we can't fix it either. On 22 Nov 2016 21:57, "Bill Burke" wrote: > In removing UserFederationModel I found that we have generic Migration > objects that use this api to upgrade LDAP. Specifically in > > 1.3.0 > > 1.4.0 > > I vote we don't support migration anymore for anything older than 1.9.8 > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From glavoie at gmail.com Wed Nov 23 08:35:25 2016 From: glavoie at gmail.com (Gabriel Lavoie) Date: Wed, 23 Nov 2016 08:35:25 -0500 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: Hi, I received some feedback on the Liquibase changeset for the missing indexes. Any comment on the other suggested changes? Thank you, Gabriel 2016-11-16 19:43 GMT-05:00 Gabriel Lavoie : > Hi, > following http://lists.jboss.org/pipermail/keycloak-user/ > 2016-October/008032.html that was discussed in the keycloak-user mailing > list, I've been working in the last few weeks on fixing performance issues > for Keycloak setup with a very large number of realms (tested with > 500-600). As suggested by Stian, I have changes to submit as pull requests > that I want to discuss here first to make sure they are acceptable. > > With this improvements, the following issues were solved in our setup: > - Startup time reduced from more than 15 minute to under a minute. > - Login time to the admin console reduced from ~5 minutes to ~20 secs when > cache is being filled, to under 5 when cache is filled. > - Realm creation time reduced from ~1 minute to under 10 secs. > > 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): > From what I see, the JPA entity model is very disconnected and many > entities are retrieved through their ID using NamedQuery. I'm not sure why > the mapping was done this way. > > This causes Hibernate to make a autoFlush check for dirty entities on > every query, which is very CPU intensive. There are a few entities that can > benefit from having a Collection/Set to their child entities and use it > rather than a query to retrieve the required information. Also, with > explicit mapping, I foresee multiple explicit em.flush() that could be > removed when changing the data. > > In this changeset I've covered only what was necessary for us, but there > are still a few areas that could be improved in my opinion: > > https://github.com/glavoie/keycloak/commit/93c8056498e38ed075523f5b9b78db > de2814d783 > > > 2 - Many missing index for FKs: > We are using an Oracle database and indexes are not created automatically > for FKs. I've added many but I'm not 100% sure if you want a new Liquibase > changeset for every changes that could go in a version or only one per > version. I still have one index to fix so don't take this as-is for now: > > https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e663695218524a0d > a4ce48a331 > > 3 - Using Set rather than Collection with JPA entities: > When using a Collection, Hibernate will delete all rows and re-insert them > in the join table to update the content. This behavior was very slow and > visible with the "admin" composite role that points to roles for all realms > (through the different clients). This was slowing down a lot new realm > creation. When using a Set, Hibernate does a better job at tracking changes. > > https://github.com/glavoie/keycloak/commit/9eb1bf013e80395affeec09625d5ee > 6157a7084a > > > 4 - "session" cache when testing for "hasRole" with a composite role: > The recursive search done KeycloakModelUtils.searchFor() can generate > quite a lot of hits on the Infinispan cache when trying to test for role. > With 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache > just to login to the admin console. I've added a in-memory cache of the > retrieved composite RoleModel list in the Infinispan RoleAdapter. As I > understand, every instance of a RoleAdapter is bound to a KeycloakSession > that can be seen as a transaction/request. Still relying on the delegate > object if the RealmAdapter is being invalidated. > > This is the change that reduced the most the login time to the admin > console or the time to obtain a Bearer token for the admin API: > > https://github.com/glavoie/keycloak/commit/0ab39670525315148761627e7d0f80 > 8d4ef9c3c0 > > > 5 - realmRevisions cache size: > In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions > cache creation was moved from standalone.xml to Java code. With our current > Keycloak cluster we already added the eviction policy mentioned in the > ticket, but had to increase the size to 50000 to support our number of > realms. Having this size hardcoded in the Java code becomes a bit > inconvenient with the latest versions. > > I didn't really fix the code as Stian mentioned that this part is being > heavily reworked. I would guess this has something to do with > http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? > > https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a743941980bad3 > dc83be7965 > > > Thank you! > > Gabriel > > -- > Gabriel Lavoie > glavoie at gmail.com > -- Gabriel Lavoie glavoie at gmail.com From ronyjoy at gmail.com Wed Nov 23 10:20:46 2016 From: ronyjoy at gmail.com (rony joy) Date: Wed, 23 Nov 2016 15:20:46 +0000 Subject: [keycloak-dev] Saml authentication Signature verification Exception when Special Characters is the username In-Reply-To: References: Message-ID: Sure I will add it to Jira On Wed, Nov 23, 2016 at 1:35 AM Hynek Mlnarik wrote: > Could you please create JIRA? > > On Wed, Nov 23, 2016 at 6:35 AM, rony joy wrote: > > Hi All, > > We are getting signature verification exception at the client side after > Idp successfully authenticated the user("Ro????????????????") when the user > id contains special characters. > > *UserName : Ro????????????????* > *Following are the keycloak settings. * > *Encryption req: false* > *Sign Document : true* > > *Please find the below exception at the client side* > > > > 05:25:23at > > org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler.verifyPostBindingSignature(AbstractSamlAuthenticationHandler.java:480)se > signature: org.keycloak.com > mon.Veriat > > org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler.validateSamlSignature(AbstractSamlAuthenticationHandler.java:261) > at > > org.keycloak.adapters.saml.profile.webbrowsersso.SamlEndpoint.handle(SamlEndpoint.java:44)stractSamlAuthenticationHandler.java:183) > at > > org.keycloak.adapters.saml.undertow.AbstractSamlAuthMech.authenticate(AbstractSamlAuthMech.java:115) > at > > io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:263) > at > > io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)31) > at > > io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)9) > at > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)(ServletAuthenticationCallHandler.java:55) > at > > io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)3) > at > > io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)ndler.java:64) > at > > io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > at > > io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) > at > > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) > at > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > > io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)java:292) > at > > io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > at > > io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)tupAction.java:48) > at > > io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) > at > > io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) > at > > io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)44) > at > > io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > at > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > -- > > --Hynek > From bburke at redhat.com Wed Nov 23 10:35:17 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 10:35:17 -0500 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> Message-ID: Not sure I have time to port those migration scripts to Liquibase and Mongo specific migration scripts. Its a day or more of work to implement and manually test. FYI, its actually any version < 1.9.0 as there are generic LDAP migration scripts for 1.8.0 as well. On 11/23/16 4:05 AM, Stian Thorgersen wrote: > > I vote against that. There's a lot of users that are still on older > versions. We could ask them to upgrade to 1.9.8 first, but there's a > chance that could fail (we've seen this a few times) and as they can't > upgrade to the most recent version we can't fix it either. > > > On 22 Nov 2016 21:57, "Bill Burke" > wrote: > > In removing UserFederationModel I found that we have generic Migration > objects that use this api to upgrade LDAP. Specifically in > > 1.3.0 > > 1.4.0 > > I vote we don't support migration anymore for anything older than > 1.9.8 > > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > From bburke at redhat.com Wed Nov 23 10:51:57 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 10:51:57 -0500 Subject: [keycloak-dev] migrating custom User Fed Providers Message-ID: <6d394414-06f5-63ec-e39c-730fdf0dfb64@redhat.com> Not sure what to do about migration of custom User Fed providers. The issue is around imported users as they have a federation link specified. What I think I can do is check to see if the provider exists for the linked user when queried, if it doesn't remove the user. This would slowly remove old linked users. This is the easiest solution. I can do something similar to LDAP in which if a UserStorage with same provider id exists, then just port it to the new component model. If there isn't a similar provider remove all users that are linked. This becomes much harder as this isn't as simple as deleting the user from the user table. I'll have to port all the queries that are executed from JPA to JDBC when a user is removed. More work.... From ronyjoy at gmail.com Wed Nov 23 10:52:47 2016 From: ronyjoy at gmail.com (rony joy) Date: Wed, 23 Nov 2016 15:52:47 +0000 Subject: [keycloak-dev] Saml authentication Signature verification Exception when Special Characters is the username In-Reply-To: References: Message-ID: Created Jira Issue https://issues.jboss.org/browse/KEYCLOAK-3971 thanks Rony Joy On Wed, Nov 23, 2016 at 1:35 AM Hynek Mlnarik wrote: > Could you please create JIRA? > > On Wed, Nov 23, 2016 at 6:35 AM, rony joy wrote: > > Hi All, > > We are getting signature verification exception at the client side after > Idp successfully authenticated the user("Ro????????????????") when the user > id contains special characters. > > *UserName : Ro????????????????* > *Following are the keycloak settings. * > *Encryption req: false* > *Sign Document : true* > > *Please find the below exception at the client side* > > > > 05:25:23at > > org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler.verifyPostBindingSignature(AbstractSamlAuthenticationHandler.java:480)se > signature: org.keycloak.com > mon.Veriat > > org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler.validateSamlSignature(AbstractSamlAuthenticationHandler.java:261) > at > > org.keycloak.adapters.saml.profile.webbrowsersso.SamlEndpoint.handle(SamlEndpoint.java:44)stractSamlAuthenticationHandler.java:183) > at > > org.keycloak.adapters.saml.undertow.AbstractSamlAuthMech.authenticate(AbstractSamlAuthMech.java:115) > at > > io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:263) > at > > io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)31) > at > > io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)9) > at > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)(ServletAuthenticationCallHandler.java:55) > at > > io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)3) > at > > io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)ndler.java:64) > at > > io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > at > > io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) > at > > org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) > at > > io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > at > > io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)java:292) > at > > io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) > at > > io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)tupAction.java:48) > at > > io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) > at > > io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44) > at > > io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)44) > at > > io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) > at > io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:805) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > -- > > --Hynek > From mposolda at redhat.com Wed Nov 23 12:13:52 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 23 Nov 2016 18:13:52 +0100 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> Message-ID: <870b773b-6ee3-767b-ae90-0da1892c1f17@redhat.com> The generic migration is very cool, but there is problem that it has hard dependency on the API, so you can't easily remove the stuff from model API... How about deprecate UserFederationProviderModel and deprecate all UserFederationProviderModel CRUD methods on RealmModel? Or maybe even remove those CRUD methods from RealmModel, but keep them just on JPA RealmAdapter and Mongo RealmAdapter? The MigrateTo1_3_0 would need to cast the RealmModel somehow to particular RealmAdapter though. Marek On 23/11/16 16:35, Bill Burke wrote: > Not sure I have time to port those migration scripts to Liquibase and > Mongo specific migration scripts. Its a day or more of work to > implement and manually test. > > FYI, its actually any version < 1.9.0 as there are generic LDAP > migration scripts for 1.8.0 as well. > > > On 11/23/16 4:05 AM, Stian Thorgersen wrote: >> I vote against that. There's a lot of users that are still on older >> versions. We could ask them to upgrade to 1.9.8 first, but there's a >> chance that could fail (we've seen this a few times) and as they can't >> upgrade to the most recent version we can't fix it either. >> >> >> On 22 Nov 2016 21:57, "Bill Burke" > > wrote: >> >> In removing UserFederationModel I found that we have generic Migration >> objects that use this api to upgrade LDAP. Specifically in >> >> 1.3.0 >> >> 1.4.0 >> >> I vote we don't support migration anymore for anything older than >> 1.9.8 >> >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From sthorger at redhat.com Wed Nov 23 14:27:47 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Wed, 23 Nov 2016 20:27:47 +0100 Subject: [keycloak-dev] Changes for Keycloak 2.4.1 should go into master Message-ID: Just add changes for 2.4.1 directly to master rather than 2.4.x branch! Thanks From bburke at redhat.com Wed Nov 23 15:22:20 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 15:22:20 -0500 Subject: [keycloak-dev] next release should be 2.5 not 2.4.1 Message-ID: Next community release should be 2.5 not 2.4.1. Since we are removing support for the User Federation SPI, this will break backward campatibility with 2.4.0.Final. Point releases should not break backward compatibility so, IMO, the next community release should be 2.5.0, not 2.4.1. Bill From bburke at redhat.com Wed Nov 23 15:24:56 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 15:24:56 -0500 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: <870b773b-6ee3-767b-ae90-0da1892c1f17@redhat.com> References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> <870b773b-6ee3-767b-ae90-0da1892c1f17@redhat.com> Message-ID: <8eb23c51-b53b-15af-4f5b-4e44577bb21b@redhat.com> Good idea on keeping them on just the JPA/Mongo adapters...problem is, we won't be able to remove the tables. On 11/23/16 12:13 PM, Marek Posolda wrote: > The generic migration is very cool, but there is problem that it has > hard dependency on the API, so you can't easily remove the stuff from > model API... > > How about deprecate UserFederationProviderModel and deprecate all > UserFederationProviderModel CRUD methods on RealmModel? Or maybe even > remove those CRUD methods from RealmModel, but keep them just on JPA > RealmAdapter and Mongo RealmAdapter? The MigrateTo1_3_0 would need to > cast the RealmModel somehow to particular RealmAdapter though. > > Marek > > On 23/11/16 16:35, Bill Burke wrote: >> Not sure I have time to port those migration scripts to Liquibase and >> Mongo specific migration scripts. Its a day or more of work to >> implement and manually test. >> >> FYI, its actually any version < 1.9.0 as there are generic LDAP >> migration scripts for 1.8.0 as well. >> >> >> On 11/23/16 4:05 AM, Stian Thorgersen wrote: >>> I vote against that. There's a lot of users that are still on older >>> versions. We could ask them to upgrade to 1.9.8 first, but there's a >>> chance that could fail (we've seen this a few times) and as they can't >>> upgrade to the most recent version we can't fix it either. >>> >>> >>> On 22 Nov 2016 21:57, "Bill Burke" >> > wrote: >>> >>> In removing UserFederationModel I found that we have generic >>> Migration >>> objects that use this api to upgrade LDAP. Specifically in >>> >>> 1.3.0 >>> >>> 1.4.0 >>> >>> I vote we don't support migration anymore for anything older than >>> 1.9.8 >>> >>> >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >>> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > > From bburke at redhat.com Wed Nov 23 15:38:31 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 15:38:31 -0500 Subject: [keycloak-dev] turning import on/off implications Message-ID: When it is finished, you will be able to choose whether LDAP provider imports users or not. The thing is, if you run with import On (which will be the default setting for older migrated ldap deployments)...then you switch it to OFF what should happen? My first thought is that we remove all imported users when that switch is fllicked off. This would require: * an onUpdate(ComponentModel old, ComponentModel new) callback to the UserStorageProvider so it can trigger deletion. * A method deleteLinkedUsers(String federationLink) on userLocalStorage() I'm also wondering if the generic console should have a DELETE_LINKED_USERS and UNLINK USERS button if the provider supports import. See any problems with this? From bburke at redhat.com Wed Nov 23 15:41:03 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 15:41:03 -0500 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: <8eb23c51-b53b-15af-4f5b-4e44577bb21b@redhat.com> References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> <870b773b-6ee3-767b-ae90-0da1892c1f17@redhat.com> <8eb23c51-b53b-15af-4f5b-4e44577bb21b@redhat.com> Message-ID: <78346a09-6e2e-f09e-e538-9bff9b176e77@redhat.com> This actually wouldn't work at all. The Liquibase/Mongo scripts would already have converted the ldap provider to the new component datamodel....HMMM.....I guess that means I can just modify the component model! problem solved! On 11/23/16 3:24 PM, Bill Burke wrote: > Good idea on keeping them on just the JPA/Mongo adapters...problem is, > we won't be able to remove the tables. > > > On 11/23/16 12:13 PM, Marek Posolda wrote: >> The generic migration is very cool, but there is problem that it has >> hard dependency on the API, so you can't easily remove the stuff from >> model API... >> >> How about deprecate UserFederationProviderModel and deprecate all >> UserFederationProviderModel CRUD methods on RealmModel? Or maybe even >> remove those CRUD methods from RealmModel, but keep them just on JPA >> RealmAdapter and Mongo RealmAdapter? The MigrateTo1_3_0 would need to >> cast the RealmModel somehow to particular RealmAdapter though. >> >> Marek >> >> On 23/11/16 16:35, Bill Burke wrote: >>> Not sure I have time to port those migration scripts to Liquibase and >>> Mongo specific migration scripts. Its a day or more of work to >>> implement and manually test. >>> >>> FYI, its actually any version < 1.9.0 as there are generic LDAP >>> migration scripts for 1.8.0 as well. >>> >>> >>> On 11/23/16 4:05 AM, Stian Thorgersen wrote: >>>> I vote against that. There's a lot of users that are still on older >>>> versions. We could ask them to upgrade to 1.9.8 first, but there's a >>>> chance that could fail (we've seen this a few times) and as they can't >>>> upgrade to the most recent version we can't fix it either. >>>> >>>> >>>> On 22 Nov 2016 21:57, "Bill Burke" >>> > wrote: >>>> >>>> In removing UserFederationModel I found that we have generic >>>> Migration >>>> objects that use this api to upgrade LDAP. Specifically in >>>> >>>> 1.3.0 >>>> >>>> 1.4.0 >>>> >>>> I vote we don't support migration anymore for anything older than >>>> 1.9.8 >>>> >>>> >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >>>> >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From mposolda at redhat.com Wed Nov 23 15:47:38 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 23 Nov 2016 21:47:38 +0100 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: <8eb23c51-b53b-15af-4f5b-4e44577bb21b@redhat.com> References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> <870b773b-6ee3-767b-ae90-0da1892c1f17@redhat.com> <8eb23c51-b53b-15af-4f5b-4e44577bb21b@redhat.com> Message-ID: <9205d522-3945-79ab-8742-8a440ad4feda@redhat.com> Yes, we won't . But I don't have any better idea. It seems there are possibilities like: 1) Keep the userFederation model API and tables in DB. Just deprecate the API 2) Converting generic migrations to Liquibase. This is lots of work, but also it's problematic because of Liquibase checksums. For example if you add new "customChange" to the existing jpa-changelog-1.3.0.xml the migration will break for those, who are already on newer version than 1.3.0. This can be handled with liquibase preconditions, but preconditions adds some additional mess and complexity... 3) Rewrite the old 1.3 generic migrations to use UserStorage SPI. This sounds error prone and probably complex... 4) Migrate in the paths ( 1.2.0 to 1.9.8 , then 1.9.8 to 2.3), but that has some other limitations Stian mentioned. My vote is 4 or 1. Marek On 23/11/16 21:24, Bill Burke wrote: > Good idea on keeping them on just the JPA/Mongo adapters...problem is, > we won't be able to remove the tables. > > > On 11/23/16 12:13 PM, Marek Posolda wrote: >> The generic migration is very cool, but there is problem that it has >> hard dependency on the API, so you can't easily remove the stuff from >> model API... >> >> How about deprecate UserFederationProviderModel and deprecate all >> UserFederationProviderModel CRUD methods on RealmModel? Or maybe even >> remove those CRUD methods from RealmModel, but keep them just on JPA >> RealmAdapter and Mongo RealmAdapter? The MigrateTo1_3_0 would need to >> cast the RealmModel somehow to particular RealmAdapter though. >> >> Marek >> >> On 23/11/16 16:35, Bill Burke wrote: >>> Not sure I have time to port those migration scripts to Liquibase and >>> Mongo specific migration scripts. Its a day or more of work to >>> implement and manually test. >>> >>> FYI, its actually any version < 1.9.0 as there are generic LDAP >>> migration scripts for 1.8.0 as well. >>> >>> >>> On 11/23/16 4:05 AM, Stian Thorgersen wrote: >>>> I vote against that. There's a lot of users that are still on older >>>> versions. We could ask them to upgrade to 1.9.8 first, but there's a >>>> chance that could fail (we've seen this a few times) and as they can't >>>> upgrade to the most recent version we can't fix it either. >>>> >>>> >>>> On 22 Nov 2016 21:57, "Bill Burke" >>> > wrote: >>>> >>>> In removing UserFederationModel I found that we have generic >>>> Migration >>>> objects that use this api to upgrade LDAP. Specifically in >>>> >>>> 1.3.0 >>>> >>>> 1.4.0 >>>> >>>> I vote we don't support migration anymore for anything older than >>>> 1.9.8 >>>> >>>> >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev at lists.jboss.org >>>> >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >>>> >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> >> > From mposolda at redhat.com Wed Nov 23 15:49:57 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 23 Nov 2016 21:49:57 +0100 Subject: [keycloak-dev] issue supporting older migrations In-Reply-To: <78346a09-6e2e-f09e-e538-9bff9b176e77@redhat.com> References: <11ba83f3-3c07-e409-ee5c-f0ce2820160d@redhat.com> <870b773b-6ee3-767b-ae90-0da1892c1f17@redhat.com> <8eb23c51-b53b-15af-4f5b-4e44577bb21b@redhat.com> <78346a09-6e2e-f09e-e538-9bff9b176e77@redhat.com> Message-ID: Cool :-) On 23/11/16 21:41, Bill Burke wrote: > This actually wouldn't work at all. The Liquibase/Mongo scripts would > already have converted the ldap provider to the new component > datamodel....HMMM.....I guess that means I can just modify the component > model! problem solved! > > > On 11/23/16 3:24 PM, Bill Burke wrote: >> Good idea on keeping them on just the JPA/Mongo adapters...problem is, >> we won't be able to remove the tables. >> >> >> On 11/23/16 12:13 PM, Marek Posolda wrote: >>> The generic migration is very cool, but there is problem that it has >>> hard dependency on the API, so you can't easily remove the stuff from >>> model API... >>> >>> How about deprecate UserFederationProviderModel and deprecate all >>> UserFederationProviderModel CRUD methods on RealmModel? Or maybe even >>> remove those CRUD methods from RealmModel, but keep them just on JPA >>> RealmAdapter and Mongo RealmAdapter? The MigrateTo1_3_0 would need to >>> cast the RealmModel somehow to particular RealmAdapter though. >>> >>> Marek >>> >>> On 23/11/16 16:35, Bill Burke wrote: >>>> Not sure I have time to port those migration scripts to Liquibase and >>>> Mongo specific migration scripts. Its a day or more of work to >>>> implement and manually test. >>>> >>>> FYI, its actually any version < 1.9.0 as there are generic LDAP >>>> migration scripts for 1.8.0 as well. >>>> >>>> >>>> On 11/23/16 4:05 AM, Stian Thorgersen wrote: >>>>> I vote against that. There's a lot of users that are still on older >>>>> versions. We could ask them to upgrade to 1.9.8 first, but there's a >>>>> chance that could fail (we've seen this a few times) and as they can't >>>>> upgrade to the most recent version we can't fix it either. >>>>> >>>>> >>>>> On 22 Nov 2016 21:57, "Bill Burke" >>>> > wrote: >>>>> >>>>> In removing UserFederationModel I found that we have generic >>>>> Migration >>>>> objects that use this api to upgrade LDAP. Specifically in >>>>> >>>>> 1.3.0 >>>>> >>>>> 1.4.0 >>>>> >>>>> I vote we don't support migration anymore for anything older than >>>>> 1.9.8 >>>>> >>>>> >>>>> _______________________________________________ >>>>> keycloak-dev mailing list >>>>> keycloak-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>> >>>>> >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From bburke at redhat.com Wed Nov 23 16:59:28 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 16:59:28 -0500 Subject: [keycloak-dev] migrating custom User Fed Providers In-Reply-To: <6d394414-06f5-63ec-e39c-730fdf0dfb64@redhat.com> References: <6d394414-06f5-63ec-e39c-730fdf0dfb64@redhat.com> Message-ID: <88ed4a12-2778-2136-1809-cf11217a16da@redhat.com> Ok, I added the logic to remove a user with a federation link that doesn't have a corresponding UserStorageProvider. The question remains: * Should I automatically convert UserFederationProviderModels to ComponentsModels that have a user storage provider with the same id? * Should I remove users imported from custom providers in Liquibase/Model migration scripts? I'm wondering if I should do this on boot up by invoking a new method on the userLocalStorage() UserProvider.removeStaleFederationLinks() Just worried this could be a very long action in the case where there are thousands of imported users. On 11/23/16 10:51 AM, Bill Burke wrote: > Not sure what to do about migration of custom User Fed providers. The > issue is around imported users as they have a federation link > specified. What I think I can do is check to see if the provider exists > for the linked user when queried, if it doesn't remove the user. This > would slowly remove old linked users. This is the easiest solution. > > I can do something similar to LDAP in which if a UserStorage with same > provider id exists, then just port it to the new component model. If > there isn't a similar provider remove all users that are linked. This > becomes much harder as this isn't as simple as deleting the user from > the user table. I'll have to port all the queries that are executed > from JPA to JDBC when a user is removed. > > More work.... > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From bburke at redhat.com Wed Nov 23 18:27:52 2016 From: bburke at redhat.com (Bill Burke) Date: Wed, 23 Nov 2016 18:27:52 -0500 Subject: [keycloak-dev] deprecated User Fed SPI removed in master Message-ID: <592ce475-8ca5-6bc2-05e5-7747260a3706@redhat.com> FYI From bruno at abstractj.org Thu Nov 24 04:43:10 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Thu, 24 Nov 2016 09:43:10 +0000 Subject: [keycloak-dev] next release should be 2.5 not 2.4.1 In-Reply-To: References: Message-ID: +1 makes sense On Wed, Nov 23, 2016, 6:22 PM Bill Burke wrote: > Next community release should be 2.5 not 2.4.1. Since we are removing > support for the User Federation SPI, this will break backward > campatibility with 2.4.0.Final. Point releases should not break > backward compatibility so, IMO, the next community release should be > 2.5.0, not 2.4.1. > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From sthorger at redhat.com Thu Nov 24 05:00:28 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Thu, 24 Nov 2016 11:00:28 +0100 Subject: [keycloak-dev] Keycloak 2.4.0.Final Released Message-ID: Keycloak 2.4.0.Final has just been released. There are no changes since 2.4.0.CR1. To download the release go to the Keycloak homepage . Before you upgrade refer to the migration guide From sthorger at redhat.com Thu Nov 24 05:07:29 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Thu, 24 Nov 2016 11:07:29 +0100 Subject: [keycloak-dev] next release should be 2.5 not 2.4.1 In-Reply-To: References: Message-ID: Actually thinking about this some more. Why don't we just keep the old user federation SPI in for now and mark them as deprecated. Then we can remove in 3.0.0C.CR1 release instead? Makes more sense to remove in a major update rather than a minor update. That would also give you more time on figuring out the migration issue? On 23 November 2016 at 21:22, Bill Burke wrote: > Next community release should be 2.5 not 2.4.1. Since we are removing > support for the User Federation SPI, this will break backward > campatibility with 2.4.0.Final. Point releases should not break > backward compatibility so, IMO, the next community release should be > 2.5.0, not 2.4.1. > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From sthorger at redhat.com Thu Nov 24 05:10:24 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Thu, 24 Nov 2016 11:10:24 +0100 Subject: [keycloak-dev] next release should be 2.5 not 2.4.1 In-Reply-To: References: Message-ID: Just saw the other threads that it's already removed. We could either recover them and mark them deprecated or remove and do a 2.5 release. Whatever you think is best. On 24 November 2016 at 11:07, Stian Thorgersen wrote: > Actually thinking about this some more. Why don't we just keep the old > user federation SPI in for now and mark them as deprecated. Then we can > remove in 3.0.0C.CR1 release instead? Makes more sense to remove in a major > update rather than a minor update. That would also give you more time on > figuring out the migration issue? > > On 23 November 2016 at 21:22, Bill Burke wrote: > >> Next community release should be 2.5 not 2.4.1. Since we are removing >> support for the User Federation SPI, this will break backward >> campatibility with 2.4.0.Final. Point releases should not break >> backward compatibility so, IMO, the next community release should be >> 2.5.0, not 2.4.1. >> >> Bill >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > From tikovsky.tomas at gmail.com Thu Nov 24 07:38:55 2016 From: tikovsky.tomas at gmail.com (Tomas Tikovsky) Date: Thu, 24 Nov 2016 13:38:55 +0100 Subject: [keycloak-dev] Suggestion and fix for e-Directory federation provider Message-ID: Hello everyone, im using e-directory federation ldap provider and came to this bug KEYCLOAK-3099 as i was experiencing the same problem. e-Directory sends guid attribute as byte[] so it needs to be declared as binary the same way as its done for activeDirectory. Sending simple diff to fix this issue if you consider this as helpfull. Novell was acquired by microfocus and their product has been renamed to netIQ eDirectory so i incorporated that change as well. Another thing i noted were 2 incorrect attribute mappings in administration console. "username" -> "uid" correct as long as users are enabled for linux (not default) otherwise cn. So cn should work for more cases than uid. "firstname" -> "cn" wrong, should be "givenname" Cheers Tom From bruno at abstractj.org Thu Nov 24 23:14:48 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Fri, 25 Nov 2016 02:14:48 -0200 Subject: [keycloak-dev] Admin console broken on master? Message-ID: <20161125041448.GB8055@abstractj.org> Good morning, I'm fixing the SSSD provider testsuite and noticed that User Federation provider list is broken on master. It looks like some endpoints changed: Error: http://localhost:8081/auth/admin/realms/master/user-federation/providers Failed to load resource: the server responded with a status of 404 (Not Found) http://localhost:8081/auth/admin/realms/master/user-federation/instances Failed to load resource: the server responded with a status of 404 (Not Found) To reproduce it, based on commit 9dd4cbb40e1579394aa2a4618b2dc941fffa4af1 run: 1. mvn clean install -DskipTests=true && mvn -f testsuite/integration/pom.xml exec:java -Pkeycloak-server 2. Login and go to "User Federation" The expected page is: Resource not found... We could not find the resource you are looking for. Please make sure the URL you entered is correct. Go to the home page ? Am I the only one? -- abstractj PGP: 0x84DC9914 From hmlnarik at redhat.com Fri Nov 25 02:06:19 2016 From: hmlnarik at redhat.com (Hynek Mlnarik) Date: Fri, 25 Nov 2016 08:06:19 +0100 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: Hi, thank you for you changes. Could you please turn them into PRs so that we can review and discuss them individually? I'm currently in favor of merging changes 2 (the new version) and 3. The rest of changes need more time for review. Thanks --Hynek On Wed, Nov 23, 2016 at 2:35 PM, Gabriel Lavoie wrote: > Hi, > I received some feedback on the Liquibase changeset for the missing > indexes. Any comment on the other suggested changes? > > Thank you, > > Gabriel > > 2016-11-16 19:43 GMT-05:00 Gabriel Lavoie : > > > Hi, > > following http://lists.jboss.org/pipermail/keycloak-user/ > > 2016-October/008032.html that was discussed in the keycloak-user mailing > > list, I've been working in the last few weeks on fixing performance > issues > > for Keycloak setup with a very large number of realms (tested with > > 500-600). As suggested by Stian, I have changes to submit as pull > requests > > that I want to discuss here first to make sure they are acceptable. > > > > With this improvements, the following issues were solved in our setup: > > - Startup time reduced from more than 15 minute to under a minute. > > - Login time to the admin console reduced from ~5 minutes to ~20 secs > when > > cache is being filled, to under 5 when cache is filled. > > - Realm creation time reduced from ~1 minute to under 10 secs. > > > > 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): > > From what I see, the JPA entity model is very disconnected and many > > entities are retrieved through their ID using NamedQuery. I'm not sure > why > > the mapping was done this way. > > > > This causes Hibernate to make a autoFlush check for dirty entities on > > every query, which is very CPU intensive. There are a few entities that > can > > benefit from having a Collection/Set to their child entities and use it > > rather than a query to retrieve the required information. Also, with > > explicit mapping, I foresee multiple explicit em.flush() that could be > > removed when changing the data. > > > > In this changeset I've covered only what was necessary for us, but there > > are still a few areas that could be improved in my opinion: > > > > https://github.com/glavoie/keycloak/commit/ > 93c8056498e38ed075523f5b9b78db > > de2814d783 > > > > > > 2 - Many missing index for FKs: > > We are using an Oracle database and indexes are not created automatically > > for FKs. I've added many but I'm not 100% sure if you want a new > Liquibase > > changeset for every changes that could go in a version or only one per > > version. I still have one index to fix so don't take this as-is for now: > > > > https://github.com/glavoie/keycloak/commit/ > 7bb44b7dfeea31e663695218524a0d > > a4ce48a331 > > > > 3 - Using Set rather than Collection with JPA entities: > > When using a Collection, Hibernate will delete all rows and re-insert > them > > in the join table to update the content. This behavior was very slow and > > visible with the "admin" composite role that points to roles for all > realms > > (through the different clients). This was slowing down a lot new realm > > creation. When using a Set, Hibernate does a better job at tracking > changes. > > > > https://github.com/glavoie/keycloak/commit/ > 9eb1bf013e80395affeec09625d5ee > > 6157a7084a > > > > > > 4 - "session" cache when testing for "hasRole" with a composite role: > > The recursive search done KeycloakModelUtils.searchFor() can generate > > quite a lot of hits on the Infinispan cache when trying to test for role. > > With 500 realms, we saw up to 1.5 millions hits on the realmsRevision > cache > > just to login to the admin console. I've added a in-memory cache of the > > retrieved composite RoleModel list in the Infinispan RoleAdapter. As I > > understand, every instance of a RoleAdapter is bound to a KeycloakSession > > that can be seen as a transaction/request. Still relying on the delegate > > object if the RealmAdapter is being invalidated. > > > > This is the change that reduced the most the login time to the admin > > console or the time to obtain a Bearer token for the admin API: > > > > https://github.com/glavoie/keycloak/commit/ > 0ab39670525315148761627e7d0f80 > > 8d4ef9c3c0 > > > > > > 5 - realmRevisions cache size: > > In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions > > cache creation was moved from standalone.xml to Java code. With our > current > > Keycloak cluster we already added the eviction policy mentioned in the > > ticket, but had to increase the size to 50000 to support our number of > > realms. Having this size hardcoded in the Java code becomes a bit > > inconvenient with the latest versions. > > > > I didn't really fix the code as Stian mentioned that this part is being > > heavily reworked. I would guess this has something to do with > > http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? > > > > https://github.com/glavoie/keycloak/commit/ > 6e3e7e8400da3440a743941980bad3 > > dc83be7965 > > > > > > Thank you! > > > > Gabriel > > > > -- > > Gabriel Lavoie > > glavoie at gmail.com > > > > > > -- > Gabriel Lavoie > glavoie at gmail.com > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > -- --Hynek From bruno at abstractj.org Fri Nov 25 05:06:47 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Fri, 25 Nov 2016 08:06:47 -0200 Subject: [keycloak-dev] Admin console broken on master? In-Reply-To: <20161125041448.GB8055@abstractj.org> References: <20161125041448.GB8055@abstractj.org> Message-ID: <20161125100647.GA11943@abstractj.org> I identified that after this commit d5925b8ccfbfa3e42f3a0396fa60f233552e9b8f things were broken, because "user-federation" endpoint was removed. What's the new endpoint to list federation providers now? On 2016-11-25, Bruno Oliveira wrote: > Good morning, > > I'm fixing the SSSD provider testsuite and noticed that User Federation > provider list is broken on master. It looks like some endpoints changed: > > Error: > > http://localhost:8081/auth/admin/realms/master/user-federation/providers Failed to load resource: the server responded with a status of 404 (Not Found) > http://localhost:8081/auth/admin/realms/master/user-federation/instances Failed to load resource: the server responded with a status of 404 (Not Found) > > To reproduce it, based on commit 9dd4cbb40e1579394aa2a4618b2dc941fffa4af1 run: > > 1. mvn clean install -DskipTests=true && mvn -f testsuite/integration/pom.xml exec:java -Pkeycloak-server > 2. Login and go to "User Federation" > > The expected page is: > > Resource not found... > We could not find the resource you are looking for. Please make sure the URL you entered is correct. > > Go to the home page ? > > Am I the only one? > > -- > > abstractj > PGP: 0x84DC9914 -- abstractj PGP: 0x84DC9914 From mposolda at redhat.com Fri Nov 25 06:43:53 2016 From: mposolda at redhat.com (Marek Posolda) Date: Fri, 25 Nov 2016 12:43:53 +0100 Subject: [keycloak-dev] Admin console broken on master? In-Reply-To: <20161125100647.GA11943@abstractj.org> References: <20161125041448.GB8055@abstractj.org> <20161125100647.GA11943@abstractj.org> Message-ID: I can see it too. It seems UserFederation were not yet fully cleaned from admin console. Feel free to create JIRA and/or fix it or wait for Bill after return from thanksgiving. AFAIK SSSD was already migrated from UserFederation to new UserStorage SPI. So new endpoint should look for userStorage component rather than federation. There should be already existing endpoint for it and I think that things will work fine after UserFederation stuff is fully cleared from angular. Marek On 25/11/16 11:06, Bruno Oliveira wrote: > I identified that after this commit d5925b8ccfbfa3e42f3a0396fa60f233552e9b8f things were broken, > because "user-federation" endpoint was removed. > > What's the new endpoint to list federation providers now? > > On 2016-11-25, Bruno Oliveira wrote: >> Good morning, >> >> I'm fixing the SSSD provider testsuite and noticed that User Federation >> provider list is broken on master. It looks like some endpoints changed: >> >> Error: >> >> http://localhost:8081/auth/admin/realms/master/user-federation/providers Failed to load resource: the server responded with a status of 404 (Not Found) >> http://localhost:8081/auth/admin/realms/master/user-federation/instances Failed to load resource: the server responded with a status of 404 (Not Found) >> >> To reproduce it, based on commit 9dd4cbb40e1579394aa2a4618b2dc941fffa4af1 run: >> >> 1. mvn clean install -DskipTests=true && mvn -f testsuite/integration/pom.xml exec:java -Pkeycloak-server >> 2. Login and go to "User Federation" >> >> The expected page is: >> >> Resource not found... >> We could not find the resource you are looking for. Please make sure the URL you entered is correct. >> >> Go to the home page ? >> >> Am I the only one? >> >> -- >> >> abstractj >> PGP: 0x84DC9914 > -- > > abstractj > PGP: 0x84DC9914 > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From bruno at abstractj.org Fri Nov 25 08:41:19 2016 From: bruno at abstractj.org (Bruno Oliveira) Date: Fri, 25 Nov 2016 11:41:19 -0200 Subject: [keycloak-dev] Admin console broken on master? In-Reply-To: References: <20161125041448.GB8055@abstractj.org> <20161125100647.GA11943@abstractj.org> Message-ID: <20161125134119.GB11943@abstractj.org> Created: https://issues.jboss.org/browse/KEYCLOAK-3984 On 2016-11-25, Marek Posolda wrote: > I can see it too. It seems UserFederation were not yet fully cleaned from > admin console. Feel free to create JIRA and/or fix it or wait for Bill after > return from thanksgiving. > > AFAIK SSSD was already migrated from UserFederation to new UserStorage SPI. > So new endpoint should look for userStorage component rather than > federation. There should be already existing endpoint for it and I think > that things will work fine after UserFederation stuff is fully cleared from > angular. Thank you Marek, I started to work on https://issues.jboss.org/browse/KEYCLOAK-3970. The testsuite is broken, but from what I test manually before that commit, everything is working as expected. > > Marek > > On 25/11/16 11:06, Bruno Oliveira wrote: > > I identified that after this commit d5925b8ccfbfa3e42f3a0396fa60f233552e9b8f things were broken, > > because "user-federation" endpoint was removed. > > > > What's the new endpoint to list federation providers now? > > > > On 2016-11-25, Bruno Oliveira wrote: > > > Good morning, > > > > > > I'm fixing the SSSD provider testsuite and noticed that User Federation > > > provider list is broken on master. It looks like some endpoints changed: > > > > > > Error: > > > > > > http://localhost:8081/auth/admin/realms/master/user-federation/providers Failed to load resource: the server responded with a status of 404 (Not Found) > > > http://localhost:8081/auth/admin/realms/master/user-federation/instances Failed to load resource: the server responded with a status of 404 (Not Found) > > > > > > To reproduce it, based on commit 9dd4cbb40e1579394aa2a4618b2dc941fffa4af1 run: > > > > > > 1. mvn clean install -DskipTests=true && mvn -f testsuite/integration/pom.xml exec:java -Pkeycloak-server > > > 2. Login and go to "User Federation" > > > > > > The expected page is: > > > > > > Resource not found... > > > We could not find the resource you are looking for. Please make sure the URL you entered is correct. > > > > > > Go to the home page ? > > > > > > Am I the only one? > > > > > > -- > > > > > > abstractj > > > PGP: 0x84DC9914 > > -- > > > > abstractj > > PGP: 0x84DC9914 > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > -- abstractj PGP: 0x84DC9914 From bburke at redhat.com Mon Nov 28 09:12:08 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 28 Nov 2016 09:12:08 -0500 Subject: [keycloak-dev] next release should be 2.5 not 2.4.1 In-Reply-To: References: Message-ID: Recovery would be a bit of work as I also removed methods from KeycloakSession, the UserFederationManager, and ported relevant tests. Do a 2.5 release. We've been telling people for a long time that this SPI would be removed. Then again, its been undocumented for a long time too. On 11/24/16 5:10 AM, Stian Thorgersen wrote: > Just saw the other threads that it's already removed. We could either > recover them and mark them deprecated or remove and do a 2.5 release. > Whatever you think is best. > > On 24 November 2016 at 11:07, Stian Thorgersen > wrote: > > Actually thinking about this some more. Why don't we just keep the > old user federation SPI in for now and mark them as deprecated. > Then we can remove in 3.0.0C.CR1 release instead? Makes more sense > to remove in a major update rather than a minor update. That would > also give you more time on figuring out the migration issue? > > On 23 November 2016 at 21:22, Bill Burke > wrote: > > Next community release should be 2.5 not 2.4.1. Since we are > removing > support for the User Federation SPI, this will break backward > campatibility with 2.4.0.Final. Point releases should not break > backward compatibility so, IMO, the next community release > should be > 2.5.0, not 2.4.1. > > Bill > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > From sthorger at redhat.com Mon Nov 28 10:34:44 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Mon, 28 Nov 2016 16:34:44 +0100 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: On a quick glance these changes (1, 2 and 3) seems all to make sense, but there's a lot of them and we just don't have the capacity to review and incorporate for 2.x (we're only weeks away from last 2.x release) so I think it'll have to wait for 3.x. 4 and 5 have been significantly reworked so please review 2.4.0.Final and see if this release fixed issues around caching. On 17 November 2016 at 01:43, Gabriel Lavoie wrote: > Hi, > following > http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html > that was discussed in the keycloak-user mailing list, I've been working in > the last few weeks on fixing performance issues for Keycloak setup with a > very large number of realms (tested with 500-600). As suggested by Stian, I > have changes to submit as pull requests that I want to discuss here first > to make sure they are acceptable. > > With this improvements, the following issues were solved in our setup: > - Startup time reduced from more than 15 minute to under a minute. > - Login time to the admin console reduced from ~5 minutes to ~20 secs when > cache is being filled, to under 5 when cache is filled. > - Realm creation time reduced from ~1 minute to under 10 secs. > > 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): > >From what I see, the JPA entity model is very disconnected and many > entities are retrieved through their ID using NamedQuery. I'm not sure why > the mapping was done this way. > > This causes Hibernate to make a autoFlush check for dirty entities on every > query, which is very CPU intensive. There are a few entities that can > benefit from having a Collection/Set to their child entities and use it > rather than a query to retrieve the required information. Also, with > explicit mapping, I foresee multiple explicit em.flush() that could be > removed when changing the data. > > In this changeset I've covered only what was necessary for us, but there > are still a few areas that could be improved in my opinion: > > https://github.com/glavoie/keycloak/commit/93c8056498e38ed075523f5b9b78db > de2814d783 > > > 2 - Many missing index for FKs: > We are using an Oracle database and indexes are not created automatically > for FKs. I've added many but I'm not 100% sure if you want a new Liquibase > changeset for every changes that could go in a version or only one per > version. I still have one index to fix so don't take this as-is for now: > > https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e663695218524a0d > a4ce48a331 > > 3 - Using Set rather than Collection with JPA entities: > When using a Collection, Hibernate will delete all rows and re-insert them > in the join table to update the content. This behavior was very slow and > visible with the "admin" composite role that points to roles for all realms > (through the different clients). This was slowing down a lot new realm > creation. When using a Set, Hibernate does a better job at tracking > changes. > > https://github.com/glavoie/keycloak/commit/9eb1bf013e80395affeec09625d5ee > 6157a7084a > > > 4 - "session" cache when testing for "hasRole" with a composite role: > The recursive search done KeycloakModelUtils.searchFor() can generate quite > a lot of hits on the Infinispan cache when trying to test for role. With > 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache just > to login to the admin console. I've added a in-memory cache of the > retrieved composite RoleModel list in the Infinispan RoleAdapter. As I > understand, every instance of a RoleAdapter is bound to a KeycloakSession > that can be seen as a transaction/request. Still relying on the delegate > object if the RealmAdapter is being invalidated. > > This is the change that reduced the most the login time to the admin > console or the time to obtain a Bearer token for the admin API: > > https://github.com/glavoie/keycloak/commit/0ab39670525315148761627e7d0f80 > 8d4ef9c3c0 > > > 5 - realmRevisions cache size: > In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions cache > creation was moved from standalone.xml to Java code. With our current > Keycloak cluster we already added the eviction policy mentioned in the > ticket, but had to increase the size to 50000 to support our number of > realms. Having this size hardcoded in the Java code becomes a bit > inconvenient with the latest versions. > > I didn't really fix the code as Stian mentioned that this part is being > heavily reworked. I would guess this has something to do with > http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? > > https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a743941980bad3 > dc83be7965 > > > Thank you! > > Gabriel > > -- > Gabriel Lavoie > glavoie at gmail.com > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From bburke at redhat.com Mon Nov 28 11:26:17 2016 From: bburke at redhat.com (Bill Burke) Date: Mon, 28 Nov 2016 11:26:17 -0500 Subject: [keycloak-dev] migrating custom User Fed Providers In-Reply-To: <88ed4a12-2778-2136-1809-cf11217a16da@redhat.com> References: <6d394414-06f5-63ec-e39c-730fdf0dfb64@redhat.com> <88ed4a12-2778-2136-1809-cf11217a16da@redhat.com> Message-ID: bump...Any thoughts? On 11/23/16 4:59 PM, Bill Burke wrote: > Ok, I added the logic to remove a user with a federation link that > doesn't have a corresponding UserStorageProvider. The question remains: > > * Should I automatically convert UserFederationProviderModels to > ComponentsModels that have a user storage provider with the same id? > > * Should I remove users imported from custom providers in > Liquibase/Model migration scripts? > > I'm wondering if I should do this on boot up by invoking a new method on > the userLocalStorage() > > UserProvider.removeStaleFederationLinks() > > Just worried this could be a very long action in the case where there > are thousands of imported users. > > > On 11/23/16 10:51 AM, Bill Burke wrote: >> Not sure what to do about migration of custom User Fed providers. The >> issue is around imported users as they have a federation link >> specified. What I think I can do is check to see if the provider exists >> for the linked user when queried, if it doesn't remove the user. This >> would slowly remove old linked users. This is the easiest solution. >> >> I can do something similar to LDAP in which if a UserStorage with same >> provider id exists, then just port it to the new component model. If >> there isn't a similar provider remove all users that are linked. This >> becomes much harder as this isn't as simple as deleting the user from >> the user table. I'll have to port all the queries that are executed >> from JPA to JDBC when a user is removed. >> >> More work.... >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From glavoie at gmail.com Mon Nov 28 15:56:04 2016 From: glavoie at gmail.com (Gabriel Lavoie) Date: Mon, 28 Nov 2016 15:56:04 -0500 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: Thanks for all the comments. I opened the PRs for 1-3. I will re-test on 2.4.0.Final a soon as possible for the other two points and report back. Gabriel 2016-11-28 10:34 GMT-05:00 Stian Thorgersen : > On a quick glance these changes (1, 2 and 3) seems all to make sense, but > there's a lot of them and we just don't have the capacity to review and > incorporate for 2.x (we're only weeks away from last 2.x release) so I > think it'll have to wait for 3.x. > > 4 and 5 have been significantly reworked so please review 2.4.0.Final and > see if this release fixed issues around caching. > > > On 17 November 2016 at 01:43, Gabriel Lavoie wrote: > >> Hi, >> following >> http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html >> that was discussed in the keycloak-user mailing list, I've been working in >> the last few weeks on fixing performance issues for Keycloak setup with a >> very large number of realms (tested with 500-600). As suggested by Stian, >> I >> have changes to submit as pull requests that I want to discuss here first >> to make sure they are acceptable. >> >> With this improvements, the following issues were solved in our setup: >> - Startup time reduced from more than 15 minute to under a minute. >> - Login time to the admin console reduced from ~5 minutes to ~20 secs when >> cache is being filled, to under 5 when cache is filled. >> - Realm creation time reduced from ~1 minute to under 10 secs. >> >> 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): >> >From what I see, the JPA entity model is very disconnected and many >> entities are retrieved through their ID using NamedQuery. I'm not sure why >> the mapping was done this way. >> >> This causes Hibernate to make a autoFlush check for dirty entities on >> every >> query, which is very CPU intensive. There are a few entities that can >> benefit from having a Collection/Set to their child entities and use it >> rather than a query to retrieve the required information. Also, with >> explicit mapping, I foresee multiple explicit em.flush() that could be >> removed when changing the data. >> >> In this changeset I've covered only what was necessary for us, but there >> are still a few areas that could be improved in my opinion: >> >> https://github.com/glavoie/keycloak/commit/93c8056498e38ed07 >> 5523f5b9b78dbde2814d783 > > >> >> >> 2 - Many missing index for FKs: >> We are using an Oracle database and indexes are not created automatically >> for FKs. I've added many but I'm not 100% sure if you want a new Liquibase >> changeset for every changes that could go in a version or only one per >> version. I still have one index to fix so don't take this as-is for now: >> >> https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e66 >> 3695218524a0da4ce48a331 >> >> 3 - Using Set rather than Collection with JPA entities: >> When using a Collection, Hibernate will delete all rows and re-insert them >> in the join table to update the content. This behavior was very slow and >> visible with the "admin" composite role that points to roles for all >> realms >> (through the different clients). This was slowing down a lot new realm >> creation. When using a Set, Hibernate does a better job at tracking >> changes. >> >> https://github.com/glavoie/keycloak/commit/9eb1bf013e80395af >> feec09625d5ee6157a7084a >> >> >> 4 - "session" cache when testing for "hasRole" with a composite role: >> The recursive search done KeycloakModelUtils.searchFor() can generate >> quite >> a lot of hits on the Infinispan cache when trying to test for role. With >> 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache >> just >> to login to the admin console. I've added a in-memory cache of the >> retrieved composite RoleModel list in the Infinispan RoleAdapter. As I >> understand, every instance of a RoleAdapter is bound to a KeycloakSession >> that can be seen as a transaction/request. Still relying on the delegate >> object if the RealmAdapter is being invalidated. >> >> This is the change that reduced the most the login time to the admin >> console or the time to obtain a Bearer token for the admin API: >> >> https://github.com/glavoie/keycloak/commit/0ab39670525315148 >> 761627e7d0f808d4ef9c3c0 >> >> >> 5 - realmRevisions cache size: >> In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions >> cache >> creation was moved from standalone.xml to Java code. With our current >> Keycloak cluster we already added the eviction policy mentioned in the >> ticket, but had to increase the size to 50000 to support our number of >> realms. Having this size hardcoded in the Java code becomes a bit >> inconvenient with the latest versions. >> >> I didn't really fix the code as Stian mentioned that this part is being >> heavily reworked. I would guess this has something to do with >> http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? >> >> https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a >> 743941980bad3dc83be7965 >> >> >> Thank you! >> >> Gabriel >> >> -- >> Gabriel Lavoie >> glavoie at gmail.com >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > -- Gabriel Lavoie glavoie at gmail.com From sthorger at redhat.com Tue Nov 29 01:14:26 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Tue, 29 Nov 2016 07:14:26 +0100 Subject: [keycloak-dev] migrating custom User Fed Providers In-Reply-To: <88ed4a12-2778-2136-1809-cf11217a16da@redhat.com> References: <6d394414-06f5-63ec-e39c-730fdf0dfb64@redhat.com> <88ed4a12-2778-2136-1809-cf11217a16da@redhat.com> Message-ID: On 23 November 2016 at 22:59, Bill Burke wrote: > Ok, I added the logic to remove a user with a federation link that > doesn't have a corresponding UserStorageProvider. The question remains: > > * Should I automatically convert UserFederationProviderModels to > ComponentsModels that have a user storage provider with the same id? > Yes, wouldn't anything else mess things up for users? > > * Should I remove users imported from custom providers in > Liquibase/Model migration scripts? > No, users could have aggregated information in the Keycloak database not stored in the custom user federation provider. > > I'm wondering if I should do this on boot up by invoking a new method on > the userLocalStorage() > > UserProvider.removeStaleFederationLinks() > > Just worried this could be a very long action in the case where there > are thousands of imported users. > I really don't follow the logic of deleting users. Makes no sense to me. > > > On 11/23/16 10:51 AM, Bill Burke wrote: > > Not sure what to do about migration of custom User Fed providers. The > > issue is around imported users as they have a federation link > > specified. What I think I can do is check to see if the provider exists > > for the linked user when queried, if it doesn't remove the user. This > > would slowly remove old linked users. This is the easiest solution. > > > > I can do something similar to LDAP in which if a UserStorage with same > > provider id exists, then just port it to the new component model. If > > there isn't a similar provider remove all users that are linked. This > > becomes much harder as this isn't as simple as deleting the user from > > the user table. I'll have to port all the queries that are executed > > from JPA to JDBC when a user is removed. > > > > More work.... > > > > _______________________________________________ > > keycloak-dev mailing list > > keycloak-dev at lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev > From sthorger at redhat.com Tue Nov 29 01:17:21 2016 From: sthorger at redhat.com (Stian Thorgersen) Date: Tue, 29 Nov 2016 07:17:21 +0100 Subject: [keycloak-dev] Docker Protocol? In-Reply-To: <1479752471.17635.15.camel@redhat.com> References: <7e296080-8b29-d70c-e397-376146f023c7@redhat.com> <1479752471.17635.15.camel@redhat.com> Message-ID: On 21 November 2016 at 19:21, Josh Cain wrote: > Finally got some time to try to wrap this one up, but I had a few > questions: > > - Was looking to add a new section to the server-administration-guide > (7.4, under SSO Protocols), but am not sure how to do that using > GitBooks. I can make comments, but am unsure how a new page would be > added in a PR-style flow. I don't have permissions to clone the > gitbook as a repo, so how would you like me to submit a new gitbook > section for this protocol? > Clone the GitHub repo (https://github.com/keycloak/server_admin_guide) and send a PR there. > - Automated testing is going to be a bit of a problem since standing up > a docker registry requires either a) system binaries, or b) something > like spinning up a docker image. Is it acceptable to use the supported > java docker client and interact with what's effectively a registry > fixture via the a Keycloak Docker client for the sake of integration > testing? > Can you explain a bit further? I'm not following how you could use the java docker client to test integration with the registry. > On Thu, 2016-08-18 at 06:01 +0200, Stian Thorgersen wrote: > > I'm good with including it in community as long as it's disabled by > > default, documented and tested ;) > > > > On 17 August 2016 at 16:06, Josh Cain wrote: > > > I've been in POC mode lately, so the cursory testing I've done has > > > been limited to manual, high-level tests. I'd be happy to find a > > > way to get some useful integrated test for this. > > > > > > Right now I'm using 3 things to test - a docker client, a docker > > > registry, and the keycloak IDP. Should be able to automate the > > > registry container start and there's a java library for docker > > > clients that we can use. So automated integration testing looks > > > possible, but I've not gone down that road just yet. Would > > > certainly do so if this was going upstream to Keycloak proper. > > > > > > > > > Josh Cain | Software Applications Engineer > > > Identity and Access Management > > > Red Hat > > > +1 256-452-0150 > > > > > > On Wed, Aug 17, 2016 at 7:56 AM, Bill Burke > > > wrote: > > > > Can't you just have "enabled": false in keycloak-server.json for > > > > that provider? > > > > > > > > On 8/17/16 3:41 AM, Stian Thorgersen wrote: > > > > > We could have it included in KC, but disabled by default. > > > > > Having the ability to enable/disable features is something we > > > > > need in either case. Especially for product as we'd like to > > > > > have certain community only features (or tech preview features) > > > > > disabled by default to make it obvious to customers they are > > > > > using unsupported features. > > > > > > > > > > We'd need to figure out some way to automate testing of it > > > > > though. We obviously can't manually test with Docker on a > > > > > regular basis. Do you have any plans around how to test it? > > > > > > > > > > On 16 August 2016 at 21:24, Josh Cain > > > > > wrote: > > > > > > Couldn't agree more that Docker should use an > > > > > > existing/standards-based auth protocol. Problem is, we need > > > > > > our Keycloak IDP to start talking to docker registries. > > > > > > Since we have a hard requirement to talk to docker > > > > > > registries, our IDP has to speak docker auth. There are > > > > > > already a number of Docker registry authorization servers out > > > > > > there, so there seems to be a need out there in the community > > > > > > as well. > > > > > > > > > > > > We're going to develop one and push it out such that it's > > > > > > publicly accessible in any case, I was just curious as to > > > > > > whether the dev team would want to incorporate this into the > > > > > > product. > > > > > > > > > > > > > > > > > > Josh Cain | Software Applications Engineer > > > > > > Identity and Access Management > > > > > > Red Hat > > > > > > +1 256-452-0150 > > > > > > > > > > > > On Tue, Aug 16, 2016 at 4:19 AM, Stian Thorgersen > > > > > edhat.com> wrote: > > > > > > > I'm not sure we'd like to have a Docker registry specific > > > > > > > protocol added directly to Keycloak. Seems like Docker > > > > > > > registry should rather get their act together and comply > > > > > > > with existing protocols rather than invent their own. > > > > > > > > > > > > > > We did have an idea of creating some repository for > > > > > > > extensions. These would be community maintained, not > > > > > > > included in product and wouldn't receive the same level of > > > > > > > testing. Maybe that would be a good place for this. With > > > > > > > Bills recent deployer work it could be easier to allow > > > > > > > users to deploy custom extensions as well. > > > > > > > > > > > > > > On 15 August 2016 at 16:41, Josh Cain > > > > > > > wrote: > > > > > > > > Hi Stian, > > > > > > > > > > > > > > > > Docker's auth V2 (docs link above) is Oauth-ish, but > > > > > > > > doesn't seem to conform 100% to the specification. I > > > > > > > > started by just trying to stand up an OIDC endpoint to > > > > > > > > talk to docker and Keycloak threw a "Missing parameters: > > > > > > > > response_type" error. Turns out, Docker sends the GET > > > > > > > > request like this: > > > > > > > > > > > > > > > > /auth/realms/redhat-external/protocol/docker- > > > > > > > > v2/auth?account=jcain&scope=repository%3Acentos%3Apull&se > > > > > > > > rvice=docker-registry > > > > > > > > > > > > > > > > Not only is the request missing the request_typer > > > > > > > > paremeter, but Docker uses different nomenclature than > > > > > > > > the OAuth/OIDC standard. For instance, I would expect > > > > > > > > the 'service' param to appear as the client_id param to > > > > > > > > conform to the OAuth spec. > > > > > > > > > > > > > > > > Since it uses different nomenclature, I thought it would > > > > > > > > be a much cleaner implementation to just implement it as > > > > > > > > its own protocol. Didn't want to muddy up a clean > > > > > > > > OIDC/OAuth implemention. > > > > > > > > > > > > > > > > Are there workarounds to deal with these differences that > > > > > > > > I'm missing? > > > > > > > > > > > > > > > > > > > > > > > > Josh Cain | Software Applications Engineer > > > > > > > > Identity and Access Management > > > > > > > > Red Hat > > > > > > > > +1 843-737-1735 > > > > > > > > > > > > > > > > On Mon, Aug 15, 2016 at 5:56 AM, Stian Thorgersen > > > > > > > er at redhat.com> wrote: > > > > > > > > > I'm not sure I fully understand. Are you using a Docker > > > > > > > > > client to authenticate with Keycloak? That works with > > > > > > > > > the standard OIDC flows, but it requires some > > > > > > > > > additional claims in the token which you are adding > > > > > > > > > with a protocol mapper? > > > > > > > > > > > > > > > > > > On 12 August 2016 at 15:31, Josh Cain > > > > > > > > .com> wrote: > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > We want to use Keycloak as the IDP/Token issuer for > > > > > > > > > > authentication with a docker registry as per the > > > > > > > > > > specification found here: > > > > > > > > > > > > > > > > > > > > https://docs.docker.com/registry/spec/auth/ > > > > > > > > > > > > > > > > > > > > I've implemented a Protocol Mapper in Keycloak that > > > > > > > > > > successfully uses the IDP to perform a login against > > > > > > > > > > a registry/docker client. Is this something that the > > > > > > > > > > team is interested in building into the product? If > > > > > > > > > > so, I'd be happy to push back upstream. > > > > > > > > > > > > > > > > > > > > Josh Cain | Software Applications Engineer > > > > > > > > > > Identity and Access Management > > > > > > > > > > Red Hat > > > > > > > > > > +1 843-737-1735 > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > keycloak-dev mailing list > > > > > > > > > > keycloak-dev at lists.jboss.org > > > > > > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > keycloak-dev mailing list > > > > > keycloak-dev at lists.jboss.org > > > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > > > > > > > > _______________________________________________ > > > > keycloak-dev mailing list > > > > keycloak-dev at lists.jboss.org > > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > > > > > > > _______________________________________________ > > > keycloak-dev mailing list > > > keycloak-dev at lists.jboss.org > > > https://lists.jboss.org/mailman/listinfo/keycloak-dev > > > From bburke at redhat.com Tue Nov 29 09:26:33 2016 From: bburke at redhat.com (Bill Burke) Date: Tue, 29 Nov 2016 09:26:33 -0500 Subject: [keycloak-dev] migrating custom User Fed Providers In-Reply-To: References: <6d394414-06f5-63ec-e39c-730fdf0dfb64@redhat.com> <88ed4a12-2778-2136-1809-cf11217a16da@redhat.com> Message-ID: <39f2ee11-df11-a575-314e-98bb4f070254@redhat.com> On 11/29/16 1:14 AM, Stian Thorgersen wrote: > > > On 23 November 2016 at 22:59, Bill Burke > wrote: > > Ok, I added the logic to remove a user with a federation link that > doesn't have a corresponding UserStorageProvider. The question > remains: > > * Should I automatically convert UserFederationProviderModels to > ComponentsModels that have a user storage provider with the same id? > > > Yes, wouldn't anything else mess things up for users? > > > * Should I remove users imported from custom providers in > Liquibase/Model migration scripts? > > > No, users could have aggregated information in the Keycloak database > not stored in the custom user federation provider. > > > I'm wondering if I should do this on boot up by invoking a new > method on > the userLocalStorage() > > UserProvider.removeStaleFederationLinks() > > Just worried this could be a very long action in the case where there > are thousands of imported users. > > > I really don't follow the logic of deleting users. Makes no sense to me. Case 1: New provider does do import anymore. 1. Deploy the new provider 2. Provider has to have same provider id as the old one 3. Remove the provider in the realm (this will delete linked users) 4. Create new instance of provider Case 2: New provider doesn't exist * Remove linked users on boot up or * Remove linked users as they are found (this is the current implementation). Bill From thomas.darimont at googlemail.com Tue Nov 29 16:15:31 2016 From: thomas.darimont at googlemail.com (Thomas Darimont) Date: Tue, 29 Nov 2016 22:15:31 +0100 Subject: [keycloak-dev] Major themes for Keycloak 3.0? Message-ID: Hello guys, after having read that the next Keycloak release will be the last one of the 2.x line I was wondering what the major themes for Keycloak 3.0 will be. Do you guys already have a roadmap (apart from the JIRA issues) or information about major features / changes that are planned which you could share with the list? Would be very interesting to hear what's next :) Cheers, Thomas From bburke at redhat.com Tue Nov 29 20:15:34 2016 From: bburke at redhat.com (Bill Burke) Date: Tue, 29 Nov 2016 20:15:34 -0500 Subject: [keycloak-dev] Major themes for Keycloak 3.0? In-Reply-To: References: Message-ID: Another iteration on the Auth SPI. Adding step-up Authentication. Adding conditionals and better scripting to Authentication Flows. Cross-datacenter support. Streamlining and optimizing clustering are just some of the features planned. I really wish we could spend some time streamlining the admin REST api and the admin console too. On 11/29/16 4:15 PM, Thomas Darimont wrote: > Hello guys, > > after having read that the next Keycloak release will be the last one > of the 2.x line I was wondering what the major themes for Keycloak 3.0 will > be. > > Do you guys already have a roadmap (apart from the JIRA issues) or > information about major features / changes that are planned which you > could share with the list? > > Would be very interesting to hear what's next :) > > Cheers, > Thomas > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From glavoie at gmail.com Wed Nov 30 09:07:25 2016 From: glavoie at gmail.com (Gabriel Lavoie) Date: Wed, 30 Nov 2016 09:07:25 -0500 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: Hi Stian, I re-tested for points 4 and 5 on master (2.4.1) and they are still both issues. 4 - The suggested fix still speed-up considerably the admin authentication for both the admin console and the admin REST API. For the admin console, with the fix I'm getting around 300K hit on the realmRevisions cache to display 1000 realms (number increased with all my test on my perf database ;). Without that fix, the admin console display attempt ends up with expired Bearer token errors after around 238M hits on the realmsRevisions cache. I'll submit the PR for that one right away. 5 - realmRevisions cache size is still hardcoded and is too small. With the 10000 elements limit (from InfinispanConnectionProvider), many operations generate an endless loop of eviction/refresh. The 50K value I used previously still fixes that with 1000 realms, but that should be configurable. I guess I could just submit a PR with the same pattern as the users cache where the revisions cache size is based on the cache size or the default value if <= 0? Gabriel 2016-11-28 15:56 GMT-05:00 Gabriel Lavoie : > Thanks for all the comments. I opened the PRs for 1-3. I will re-test on > 2.4.0.Final a soon as possible for the other two points and report back. > > Gabriel > > 2016-11-28 10:34 GMT-05:00 Stian Thorgersen : > >> On a quick glance these changes (1, 2 and 3) seems all to make sense, but >> there's a lot of them and we just don't have the capacity to review and >> incorporate for 2.x (we're only weeks away from last 2.x release) so I >> think it'll have to wait for 3.x. >> >> 4 and 5 have been significantly reworked so please review 2.4.0.Final and >> see if this release fixed issues around caching. >> >> >> On 17 November 2016 at 01:43, Gabriel Lavoie wrote: >> >>> Hi, >>> following >>> http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html >>> that was discussed in the keycloak-user mailing list, I've been working >>> in >>> the last few weeks on fixing performance issues for Keycloak setup with a >>> very large number of realms (tested with 500-600). As suggested by >>> Stian, I >>> have changes to submit as pull requests that I want to discuss here first >>> to make sure they are acceptable. >>> >>> With this improvements, the following issues were solved in our setup: >>> - Startup time reduced from more than 15 minute to under a minute. >>> - Login time to the admin console reduced from ~5 minutes to ~20 secs >>> when >>> cache is being filled, to under 5 when cache is filled. >>> - Realm creation time reduced from ~1 minute to under 10 secs. >>> >>> 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): >>> >From what I see, the JPA entity model is very disconnected and many >>> entities are retrieved through their ID using NamedQuery. I'm not sure >>> why >>> the mapping was done this way. >>> >>> This causes Hibernate to make a autoFlush check for dirty entities on >>> every >>> query, which is very CPU intensive. There are a few entities that can >>> benefit from having a Collection/Set to their child entities and use it >>> rather than a query to retrieve the required information. Also, with >>> explicit mapping, I foresee multiple explicit em.flush() that could be >>> removed when changing the data. >>> >>> In this changeset I've covered only what was necessary for us, but there >>> are still a few areas that could be improved in my opinion: >>> >>> https://github.com/glavoie/keycloak/commit/93c8056498e38ed07 >>> 5523f5b9b78dbde2814d783 >> >> >>> >>> >>> 2 - Many missing index for FKs: >>> We are using an Oracle database and indexes are not created automatically >>> for FKs. I've added many but I'm not 100% sure if you want a new >>> Liquibase >>> changeset for every changes that could go in a version or only one per >>> version. I still have one index to fix so don't take this as-is for now: >>> >>> https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e66 >>> 3695218524a0da4ce48a331 >>> >>> 3 - Using Set rather than Collection with JPA entities: >>> When using a Collection, Hibernate will delete all rows and re-insert >>> them >>> in the join table to update the content. This behavior was very slow and >>> visible with the "admin" composite role that points to roles for all >>> realms >>> (through the different clients). This was slowing down a lot new realm >>> creation. When using a Set, Hibernate does a better job at tracking >>> changes. >>> >>> https://github.com/glavoie/keycloak/commit/9eb1bf013e80395af >>> feec09625d5ee6157a7084a >>> >>> >>> 4 - "session" cache when testing for "hasRole" with a composite role: >>> The recursive search done KeycloakModelUtils.searchFor() can generate >>> quite >>> a lot of hits on the Infinispan cache when trying to test for role. With >>> 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache >>> just >>> to login to the admin console. I've added a in-memory cache of the >>> retrieved composite RoleModel list in the Infinispan RoleAdapter. As I >>> understand, every instance of a RoleAdapter is bound to a KeycloakSession >>> that can be seen as a transaction/request. Still relying on the delegate >>> object if the RealmAdapter is being invalidated. >>> >>> This is the change that reduced the most the login time to the admin >>> console or the time to obtain a Bearer token for the admin API: >>> >>> https://github.com/glavoie/keycloak/commit/0ab39670525315148 >>> 761627e7d0f808d4ef9c3c0 >>> >>> >>> 5 - realmRevisions cache size: >>> In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions >>> cache >>> creation was moved from standalone.xml to Java code. With our current >>> Keycloak cluster we already added the eviction policy mentioned in the >>> ticket, but had to increase the size to 50000 to support our number of >>> realms. Having this size hardcoded in the Java code becomes a bit >>> inconvenient with the latest versions. >>> >>> I didn't really fix the code as Stian mentioned that this part is being >>> heavily reworked. I would guess this has something to do with >>> http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? >>> >>> https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a >>> 743941980bad3dc83be7965 >>> >>> >>> Thank you! >>> >>> Gabriel >>> >>> -- >>> Gabriel Lavoie >>> glavoie at gmail.com >>> _______________________________________________ >>> keycloak-dev mailing list >>> keycloak-dev at lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >> >> > > > -- > Gabriel Lavoie > glavoie at gmail.com > -- Gabriel Lavoie glavoie at gmail.com From mposolda at redhat.com Wed Nov 30 09:53:51 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 30 Nov 2016 15:53:51 +0100 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: On 30/11/16 15:07, Gabriel Lavoie wrote: > Hi Stian, > I re-tested for points 4 and 5 on master (2.4.1) and they are still > both issues. > > 4 - The suggested fix still speed-up considerably the admin authentication > for both the admin console and the admin REST API. For the admin console, > with the fix I'm getting around 300K hit on the realmRevisions cache to > display 1000 realms (number increased with all my test on my perf database > ;). Without that fix, the admin console display attempt ends up with > expired Bearer token errors after around 238M hits on the realmsRevisions > cache. > > I'll submit the PR for that one right away. Feel free to submit PR for your composite role fixes. From the 1st look it seems ok, but it will need a bit deeper review likely. We will see whether to include it in next release or rather postpone. > > 5 - realmRevisions cache size is still hardcoded and is too small. With the > 10000 elements limit (from InfinispanConnectionProvider), many operations > generate an endless loop of eviction/refresh. The 50K value I used > previously still fixes that with 1000 realms, but that should be > configurable. I guess I could just submit a PR with the same pattern as the > users cache where the revisions cache size is based on the cache size or > the default value if <= 0? Again, feel free to submit PR. My vote is to include this one soon. Just note that "realms" cache doesn't have any limit by default, but "realmRevisions" cache always must have limit. Otherwise there is risk of memory leak. So in case that "realms" (or "users" ) cache doesn't have limit configured, then realmRevisions (or userRevisions) cache should use some default limit. I guess that's what you were suggesting, but just doublechecking. Thanks, Marek > > Gabriel > > > 2016-11-28 15:56 GMT-05:00 Gabriel Lavoie : > >> Thanks for all the comments. I opened the PRs for 1-3. I will re-test on >> 2.4.0.Final a soon as possible for the other two points and report back. >> >> Gabriel >> >> 2016-11-28 10:34 GMT-05:00 Stian Thorgersen : >> >>> On a quick glance these changes (1, 2 and 3) seems all to make sense, but >>> there's a lot of them and we just don't have the capacity to review and >>> incorporate for 2.x (we're only weeks away from last 2.x release) so I >>> think it'll have to wait for 3.x. >>> >>> 4 and 5 have been significantly reworked so please review 2.4.0.Final and >>> see if this release fixed issues around caching. >>> >>> >>> On 17 November 2016 at 01:43, Gabriel Lavoie wrote: >>> >>>> Hi, >>>> following >>>> http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html >>>> that was discussed in the keycloak-user mailing list, I've been working >>>> in >>>> the last few weeks on fixing performance issues for Keycloak setup with a >>>> very large number of realms (tested with 500-600). As suggested by >>>> Stian, I >>>> have changes to submit as pull requests that I want to discuss here first >>>> to make sure they are acceptable. >>>> >>>> With this improvements, the following issues were solved in our setup: >>>> - Startup time reduced from more than 15 minute to under a minute. >>>> - Login time to the admin console reduced from ~5 minutes to ~20 secs >>>> when >>>> cache is being filled, to under 5 when cache is filled. >>>> - Realm creation time reduced from ~1 minute to under 10 secs. >>>> >>>> 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): >>>> >From what I see, the JPA entity model is very disconnected and many >>>> entities are retrieved through their ID using NamedQuery. I'm not sure >>>> why >>>> the mapping was done this way. >>>> >>>> This causes Hibernate to make a autoFlush check for dirty entities on >>>> every >>>> query, which is very CPU intensive. There are a few entities that can >>>> benefit from having a Collection/Set to their child entities and use it >>>> rather than a query to retrieve the required information. Also, with >>>> explicit mapping, I foresee multiple explicit em.flush() that could be >>>> removed when changing the data. >>>> >>>> In this changeset I've covered only what was necessary for us, but there >>>> are still a few areas that could be improved in my opinion: >>>> >>>> https://github.com/glavoie/keycloak/commit/93c8056498e38ed07 >>>> 5523f5b9b78dbde2814d783 >>> >>>> >>>> 2 - Many missing index for FKs: >>>> We are using an Oracle database and indexes are not created automatically >>>> for FKs. I've added many but I'm not 100% sure if you want a new >>>> Liquibase >>>> changeset for every changes that could go in a version or only one per >>>> version. I still have one index to fix so don't take this as-is for now: >>>> >>>> https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e66 >>>> 3695218524a0da4ce48a331 >>>> >>>> 3 - Using Set rather than Collection with JPA entities: >>>> When using a Collection, Hibernate will delete all rows and re-insert >>>> them >>>> in the join table to update the content. This behavior was very slow and >>>> visible with the "admin" composite role that points to roles for all >>>> realms >>>> (through the different clients). This was slowing down a lot new realm >>>> creation. When using a Set, Hibernate does a better job at tracking >>>> changes. >>>> >>>> https://github.com/glavoie/keycloak/commit/9eb1bf013e80395af >>>> feec09625d5ee6157a7084a >>>> >>>> >>>> 4 - "session" cache when testing for "hasRole" with a composite role: >>>> The recursive search done KeycloakModelUtils.searchFor() can generate >>>> quite >>>> a lot of hits on the Infinispan cache when trying to test for role. With >>>> 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache >>>> just >>>> to login to the admin console. I've added a in-memory cache of the >>>> retrieved composite RoleModel list in the Infinispan RoleAdapter. As I >>>> understand, every instance of a RoleAdapter is bound to a KeycloakSession >>>> that can be seen as a transaction/request. Still relying on the delegate >>>> object if the RealmAdapter is being invalidated. >>>> >>>> This is the change that reduced the most the login time to the admin >>>> console or the time to obtain a Bearer token for the admin API: >>>> >>>> https://github.com/glavoie/keycloak/commit/0ab39670525315148 >>>> 761627e7d0f808d4ef9c3c0 >>>> >>>> >>>> 5 - realmRevisions cache size: >>>> In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions >>>> cache >>>> creation was moved from standalone.xml to Java code. With our current >>>> Keycloak cluster we already added the eviction policy mentioned in the >>>> ticket, but had to increase the size to 50000 to support our number of >>>> realms. Having this size hardcoded in the Java code becomes a bit >>>> inconvenient with the latest versions. >>>> >>>> I didn't really fix the code as Stian mentioned that this part is being >>>> heavily reworked. I would guess this has something to do with >>>> http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? >>>> >>>> https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a >>>> 743941980bad3dc83be7965 >>>> >>>> >>>> Thank you! >>>> >>>> Gabriel >>>> >>>> -- >>>> Gabriel Lavoie >>>> glavoie at gmail.com >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev at lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >>> >> >> -- >> Gabriel Lavoie >> glavoie at gmail.com >> > > From mposolda at redhat.com Wed Nov 30 10:22:05 2016 From: mposolda at redhat.com (Marek Posolda) Date: Wed, 30 Nov 2016 16:22:05 +0100 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: On 30/11/16 15:53, Marek Posolda wrote: > On 30/11/16 15:07, Gabriel Lavoie wrote: >> Hi Stian, >> I re-tested for points 4 and 5 on master (2.4.1) and they are still >> both issues. >> >> 4 - The suggested fix still speed-up considerably the admin authentication >> for both the admin console and the admin REST API. For the admin console, >> with the fix I'm getting around 300K hit on the realmRevisions cache to >> display 1000 realms (number increased with all my test on my perf database >> ;). Without that fix, the admin console display attempt ends up with >> expired Bearer token errors after around 238M hits on the realmsRevisions >> cache. >> >> I'll submit the PR for that one right away. > Feel free to submit PR for your composite role fixes. From the 1st look > it seems ok, but it will need a bit deeper review likely. We will see > whether to include it in next release or rather postpone. >> 5 - realmRevisions cache size is still hardcoded and is too small. With the >> 10000 elements limit (from InfinispanConnectionProvider), many operations >> generate an endless loop of eviction/refresh. The 50K value I used >> previously still fixes that with 1000 realms, but that should be >> configurable. I guess I could just submit a PR with the same pattern as the >> users cache where the revisions cache size is based on the cache size or >> the default value if <= 0? > Again, feel free to submit PR. My vote is to include this one soon. > > Just note that "realms" cache doesn't have any limit by default, but > "realmRevisions" cache always must have limit. Otherwise there is risk > of memory leak. So in case that "realms" (or "users" ) cache doesn't > have limit configured, then realmRevisions (or userRevisions) cache > should use some default limit. I guess that's what you were suggesting, > but just doublechecking. I would also use the limit for computing the revision cache size to be like 2*N (in case that "realms" cache has configured size N) as there are always more items in the revisions cache (Revision cache is supposed to contain also items, which were invalidated from realms cache). Same for users (and userRevisions) cache. Marek > > Thanks, > Marek > >> Gabriel >> >> >> 2016-11-28 15:56 GMT-05:00 Gabriel Lavoie : >> >>> Thanks for all the comments. I opened the PRs for 1-3. I will re-test on >>> 2.4.0.Final a soon as possible for the other two points and report back. >>> >>> Gabriel >>> >>> 2016-11-28 10:34 GMT-05:00 Stian Thorgersen : >>> >>>> On a quick glance these changes (1, 2 and 3) seems all to make sense, but >>>> there's a lot of them and we just don't have the capacity to review and >>>> incorporate for 2.x (we're only weeks away from last 2.x release) so I >>>> think it'll have to wait for 3.x. >>>> >>>> 4 and 5 have been significantly reworked so please review 2.4.0.Final and >>>> see if this release fixed issues around caching. >>>> >>>> >>>> On 17 November 2016 at 01:43, Gabriel Lavoie wrote: >>>> >>>>> Hi, >>>>> following >>>>> http://lists.jboss.org/pipermail/keycloak-user/2016-October/008032.html >>>>> that was discussed in the keycloak-user mailing list, I've been working >>>>> in >>>>> the last few weeks on fixing performance issues for Keycloak setup with a >>>>> very large number of realms (tested with 500-600). As suggested by >>>>> Stian, I >>>>> have changes to submit as pull requests that I want to discuss here first >>>>> to make sure they are acceptable. >>>>> >>>>> With this improvements, the following issues were solved in our setup: >>>>> - Startup time reduced from more than 15 minute to under a minute. >>>>> - Login time to the admin console reduced from ~5 minutes to ~20 secs >>>>> when >>>>> cache is being filled, to under 5 when cache is filled. >>>>> - Realm creation time reduced from ~1 minute to under 10 secs. >>>>> >>>>> 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): >>>>> >From what I see, the JPA entity model is very disconnected and many >>>>> entities are retrieved through their ID using NamedQuery. I'm not sure >>>>> why >>>>> the mapping was done this way. >>>>> >>>>> This causes Hibernate to make a autoFlush check for dirty entities on >>>>> every >>>>> query, which is very CPU intensive. There are a few entities that can >>>>> benefit from having a Collection/Set to their child entities and use it >>>>> rather than a query to retrieve the required information. Also, with >>>>> explicit mapping, I foresee multiple explicit em.flush() that could be >>>>> removed when changing the data. >>>>> >>>>> In this changeset I've covered only what was necessary for us, but there >>>>> are still a few areas that could be improved in my opinion: >>>>> >>>>> https://github.com/glavoie/keycloak/commit/93c8056498e38ed07 >>>>> 5523f5b9b78dbde2814d783 >>>>> 2 - Many missing index for FKs: >>>>> We are using an Oracle database and indexes are not created automatically >>>>> for FKs. I've added many but I'm not 100% sure if you want a new >>>>> Liquibase >>>>> changeset for every changes that could go in a version or only one per >>>>> version. I still have one index to fix so don't take this as-is for now: >>>>> >>>>> https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e66 >>>>> 3695218524a0da4ce48a331 >>>>> >>>>> 3 - Using Set rather than Collection with JPA entities: >>>>> When using a Collection, Hibernate will delete all rows and re-insert >>>>> them >>>>> in the join table to update the content. This behavior was very slow and >>>>> visible with the "admin" composite role that points to roles for all >>>>> realms >>>>> (through the different clients). This was slowing down a lot new realm >>>>> creation. When using a Set, Hibernate does a better job at tracking >>>>> changes. >>>>> >>>>> https://github.com/glavoie/keycloak/commit/9eb1bf013e80395af >>>>> feec09625d5ee6157a7084a >>>>> >>>>> >>>>> 4 - "session" cache when testing for "hasRole" with a composite role: >>>>> The recursive search done KeycloakModelUtils.searchFor() can generate >>>>> quite >>>>> a lot of hits on the Infinispan cache when trying to test for role. With >>>>> 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache >>>>> just >>>>> to login to the admin console. I've added a in-memory cache of the >>>>> retrieved composite RoleModel list in the Infinispan RoleAdapter. As I >>>>> understand, every instance of a RoleAdapter is bound to a KeycloakSession >>>>> that can be seen as a transaction/request. Still relying on the delegate >>>>> object if the RealmAdapter is being invalidated. >>>>> >>>>> This is the change that reduced the most the login time to the admin >>>>> console or the time to obtain a Bearer token for the admin API: >>>>> >>>>> https://github.com/glavoie/keycloak/commit/0ab39670525315148 >>>>> 761627e7d0f808d4ef9c3c0 >>>>> >>>>> >>>>> 5 - realmRevisions cache size: >>>>> In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions >>>>> cache >>>>> creation was moved from standalone.xml to Java code. With our current >>>>> Keycloak cluster we already added the eviction policy mentioned in the >>>>> ticket, but had to increase the size to 50000 to support our number of >>>>> realms. Having this size hardcoded in the Java code becomes a bit >>>>> inconvenient with the latest versions. >>>>> >>>>> I didn't really fix the code as Stian mentioned that this part is being >>>>> heavily reworked. I would guess this has something to do with >>>>> http://lists.jboss.org/pipermail/keycloak-dev/2016-November/008397.html? >>>>> >>>>> https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a >>>>> 743941980bad3dc83be7965 >>>>> >>>>> >>>>> Thank you! >>>>> >>>>> Gabriel >>>>> >>>>> -- >>>>> Gabriel Lavoie >>>>> glavoie at gmail.com >>>>> _______________________________________________ >>>>> keycloak-dev mailing list >>>>> keycloak-dev at lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>> >>> -- >>> Gabriel Lavoie >>> glavoie at gmail.com >>> >> > _______________________________________________ > keycloak-dev mailing list > keycloak-dev at lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev From glavoie at gmail.com Wed Nov 30 10:28:08 2016 From: glavoie at gmail.com (Gabriel Lavoie) Date: Wed, 30 Nov 2016 10:28:08 -0500 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: Sure, I'll adjust the computation as you suggest. I'm currently working on the code for this specific PR. Gabriel 2016-11-30 10:22 GMT-05:00 Marek Posolda : > On 30/11/16 15:53, Marek Posolda wrote: > >> On 30/11/16 15:07, Gabriel Lavoie wrote: >> >>> Hi Stian, >>> I re-tested for points 4 and 5 on master (2.4.1) and they are >>> still >>> both issues. >>> >>> 4 - The suggested fix still speed-up considerably the admin >>> authentication >>> for both the admin console and the admin REST API. For the admin console, >>> with the fix I'm getting around 300K hit on the realmRevisions cache to >>> display 1000 realms (number increased with all my test on my perf >>> database >>> ;). Without that fix, the admin console display attempt ends up with >>> expired Bearer token errors after around 238M hits on the realmsRevisions >>> cache. >>> >>> I'll submit the PR for that one right away. >>> >> Feel free to submit PR for your composite role fixes. From the 1st look >> it seems ok, but it will need a bit deeper review likely. We will see >> whether to include it in next release or rather postpone. >> >>> 5 - realmRevisions cache size is still hardcoded and is too small. With >>> the >>> 10000 elements limit (from InfinispanConnectionProvider), many operations >>> generate an endless loop of eviction/refresh. The 50K value I used >>> previously still fixes that with 1000 realms, but that should be >>> configurable. I guess I could just submit a PR with the same pattern as >>> the >>> users cache where the revisions cache size is based on the cache size or >>> the default value if <= 0? >>> >> Again, feel free to submit PR. My vote is to include this one soon. >> >> Just note that "realms" cache doesn't have any limit by default, but >> "realmRevisions" cache always must have limit. Otherwise there is risk >> of memory leak. So in case that "realms" (or "users" ) cache doesn't >> have limit configured, then realmRevisions (or userRevisions) cache >> should use some default limit. I guess that's what you were suggesting, >> but just doublechecking. >> > I would also use the limit for computing the revision cache size to be > like 2*N (in case that "realms" cache has configured size N) as there are > always more items in the revisions cache (Revision cache is supposed to > contain also items, which were invalidated from realms cache). > > Same for users (and userRevisions) cache. > > Marek > > >> Thanks, >> Marek >> >> Gabriel >>> >>> >>> 2016-11-28 15:56 GMT-05:00 Gabriel Lavoie : >>> >>> Thanks for all the comments. I opened the PRs for 1-3. I will re-test on >>>> 2.4.0.Final a soon as possible for the other two points and report back. >>>> >>>> Gabriel >>>> >>>> 2016-11-28 10:34 GMT-05:00 Stian Thorgersen : >>>> >>>> On a quick glance these changes (1, 2 and 3) seems all to make sense, >>>>> but >>>>> there's a lot of them and we just don't have the capacity to review and >>>>> incorporate for 2.x (we're only weeks away from last 2.x release) so I >>>>> think it'll have to wait for 3.x. >>>>> >>>>> 4 and 5 have been significantly reworked so please review 2.4.0.Final >>>>> and >>>>> see if this release fixed issues around caching. >>>>> >>>>> >>>>> On 17 November 2016 at 01:43, Gabriel Lavoie >>>>> wrote: >>>>> >>>>> Hi, >>>>>> following >>>>>> http://lists.jboss.org/pipermail/keycloak-user/2016-October/ >>>>>> 008032.html >>>>>> that was discussed in the keycloak-user mailing list, I've been >>>>>> working >>>>>> in >>>>>> the last few weeks on fixing performance issues for Keycloak setup >>>>>> with a >>>>>> very large number of realms (tested with 500-600). As suggested by >>>>>> Stian, I >>>>>> have changes to submit as pull requests that I want to discuss here >>>>>> first >>>>>> to make sure they are acceptable. >>>>>> >>>>>> With this improvements, the following issues were solved in our setup: >>>>>> - Startup time reduced from more than 15 minute to under a minute. >>>>>> - Login time to the admin console reduced from ~5 minutes to ~20 secs >>>>>> when >>>>>> cache is being filled, to under 5 when cache is filled. >>>>>> - Realm creation time reduced from ~1 minute to under 10 secs. >>>>>> >>>>>> 1 - Too many autoFlush checks by Hibernate and explicit em.flush(): >>>>>> >From what I see, the JPA entity model is very disconnected and many >>>>>> entities are retrieved through their ID using NamedQuery. I'm not sure >>>>>> why >>>>>> the mapping was done this way. >>>>>> >>>>>> This causes Hibernate to make a autoFlush check for dirty entities on >>>>>> every >>>>>> query, which is very CPU intensive. There are a few entities that can >>>>>> benefit from having a Collection/Set to their child entities and use >>>>>> it >>>>>> rather than a query to retrieve the required information. Also, with >>>>>> explicit mapping, I foresee multiple explicit em.flush() that could be >>>>>> removed when changing the data. >>>>>> >>>>>> In this changeset I've covered only what was necessary for us, but >>>>>> there >>>>>> are still a few areas that could be improved in my opinion: >>>>>> >>>>>> https://github.com/glavoie/keycloak/commit/93c8056498e38ed07 >>>>>> 5523f5b9b78dbde2814d783 >>>>>> 2 - Many missing index for FKs: >>>>>> We are using an Oracle database and indexes are not created >>>>>> automatically >>>>>> for FKs. I've added many but I'm not 100% sure if you want a new >>>>>> Liquibase >>>>>> changeset for every changes that could go in a version or only one per >>>>>> version. I still have one index to fix so don't take this as-is for >>>>>> now: >>>>>> >>>>>> https://github.com/glavoie/keycloak/commit/7bb44b7dfeea31e66 >>>>>> 3695218524a0da4ce48a331 >>>>>> >>>>>> 3 - Using Set rather than Collection with JPA entities: >>>>>> When using a Collection, Hibernate will delete all rows and re-insert >>>>>> them >>>>>> in the join table to update the content. This behavior was very slow >>>>>> and >>>>>> visible with the "admin" composite role that points to roles for all >>>>>> realms >>>>>> (through the different clients). This was slowing down a lot new realm >>>>>> creation. When using a Set, Hibernate does a better job at tracking >>>>>> changes. >>>>>> >>>>>> https://github.com/glavoie/keycloak/commit/9eb1bf013e80395af >>>>>> feec09625d5ee6157a7084a >>>>>> >>>>>> >>>>>> 4 - "session" cache when testing for "hasRole" with a composite role: >>>>>> The recursive search done KeycloakModelUtils.searchFor() can generate >>>>>> quite >>>>>> a lot of hits on the Infinispan cache when trying to test for role. >>>>>> With >>>>>> 500 realms, we saw up to 1.5 millions hits on the realmsRevision cache >>>>>> just >>>>>> to login to the admin console. I've added a in-memory cache of the >>>>>> retrieved composite RoleModel list in the Infinispan RoleAdapter. As I >>>>>> understand, every instance of a RoleAdapter is bound to a >>>>>> KeycloakSession >>>>>> that can be seen as a transaction/request. Still relying on the >>>>>> delegate >>>>>> object if the RealmAdapter is being invalidated. >>>>>> >>>>>> This is the change that reduced the most the login time to the admin >>>>>> console or the time to obtain a Bearer token for the admin API: >>>>>> >>>>>> https://github.com/glavoie/keycloak/commit/0ab39670525315148 >>>>>> 761627e7d0f808d4ef9c3c0 >>>>>> >>>>>> >>>>>> 5 - realmRevisions cache size: >>>>>> In https://issues.jboss.org/browse/KEYCLOAK-3202, the realmRevisions >>>>>> cache >>>>>> creation was moved from standalone.xml to Java code. With our current >>>>>> Keycloak cluster we already added the eviction policy mentioned in the >>>>>> ticket, but had to increase the size to 50000 to support our number of >>>>>> realms. Having this size hardcoded in the Java code becomes a bit >>>>>> inconvenient with the latest versions. >>>>>> >>>>>> I didn't really fix the code as Stian mentioned that this part is >>>>>> being >>>>>> heavily reworked. I would guess this has something to do with >>>>>> http://lists.jboss.org/pipermail/keycloak-dev/2016-November/ >>>>>> 008397.html? >>>>>> >>>>>> https://github.com/glavoie/keycloak/commit/6e3e7e8400da3440a >>>>>> 743941980bad3dc83be7965 >>>>>> >>>>>> >>>>>> Thank you! >>>>>> >>>>>> Gabriel >>>>>> >>>>>> -- >>>>>> Gabriel Lavoie >>>>>> glavoie at gmail.com >>>>>> _______________________________________________ >>>>>> keycloak-dev mailing list >>>>>> keycloak-dev at lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >>>>>> -- >>>> Gabriel Lavoie >>>> glavoie at gmail.com >>>> >>>> >>> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev at lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev >> > > > -- Gabriel Lavoie glavoie at gmail.com From glavoie at gmail.com Wed Nov 30 11:15:51 2016 From: glavoie at gmail.com (Gabriel Lavoie) Date: Wed, 30 Nov 2016 11:15:51 -0500 Subject: [keycloak-dev] Performance improvements with large number of realms In-Reply-To: References: Message-ID: The two last pull requests were submitted. Is there a way to re-trigger this build: https://travis-ci.org/keycloak/keycloak/builds/179566864 ? Seems like the failing jobs got stuck in the "mvn install" step for some unknown reasons. I've been trying to reproduce this locally without any success. Gabriel From shmuein+keycloak-dev at gmail.com Wed Nov 30 21:42:01 2016 From: shmuein+keycloak-dev at gmail.com (Muein Muzamil) Date: Wed, 30 Nov 2016 20:42:01 -0600 Subject: [keycloak-dev] Private Keys in plain text in database Message-ID: Hi all, I noticed currently we are storing private keys in plain text in the database, our security team has raised some concerns on that, is there any way to encrypt these private keys before storing them in database? Regards, Muein