AbstractUserAdapterFederatedStorage.setSingleAttribute(, ) causing deadlocks
by David Lustig
Greetings,
I have developed a federated user storage authenticator for Keycloak. Upon each login we set/update at least one federated attribute on the user using AbstractUserAdapterFederatedStorage.setSingleAttribute(String name, String value). During load testing, though, Keycloak starts running into deadlock issues that look to be caused by this method. The error logged is:
2017-05-24 05:15:01,376 WARN [org.keycloak.services] (default task-7) KC-SERVICES0013: Failed authentication: javax.persistence.PersistenceException: org.hibernate.exception.LockAcquisitionException: could not execute statement
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
at org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
at org.hibernate.jpa.spi.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:70)
at org.keycloak.storage.jpa.JpaUserFederatedStorageProvider.deleteAttribute(JpaUserFederatedStorageProvider.java:112)
at org.keycloak.storage.jpa.JpaUserFederatedStorageProvider.setSingleAttribute(JpaUserFederatedStorageProvider.java:129)
at org.keycloak.storage.adapter.AbstractUserAdapterFederatedStorage.setSingleAttribute(AbstractUserAdapterFederatedStorage.java:341)
...
Caused by: org.hibernate.exception.LockAcquisitionException: could not execute statement
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:123)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:95)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:207)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:91)
at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:60)
at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:111)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:429)
at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:374)
at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1348)
at org.hibernate.internal.QueryImpl.executeUpdate(QueryImpl.java:102)
at org.hibernate.jpa.internal.QueryImpl.internalExecuteUpdate(QueryImpl.java:405)
at org.hibernate.jpa.spi.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:61)
... 65 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 111) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:217)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1655)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:440)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2445)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:328)
at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:204)
... 74 more
Eventually, none of the deadlocks are released and the JVM crashes due to an out-of-memory error. Is deadlocking expected behavior? Are there any programming practices that should be followed that would prevent setSingleAttribute(,) from deadlocking the system in all situations, including high usage?
Thanks for your help,
David Lustig
This message is the property of CARBONITE, INC. and may contain confidential or privileged information.
If this message has been delivered to you by mistake, then do not copy or deliver this message to anyone. Instead, destroy it and notify me by reply e-mail
6 years, 11 months
Admin API Authz was Re: [keycloak-user] Keycloak Performance with large number of realms
by Bill Burke
[keycloak-dev] See below thread.
I think that this problem might be solved by the work I'm doing. I'm
changing the admin console to not include roles in the token. The Admin
REST API instead will see that the token was generated for the console
client (by "aud" claim) and look up role mappings directly. I have to
do this anyways because with the new fine grain admin permissions, I
don't want admins to have to change the scope of the admin console
client every time a new fine grain permission policy is specified.
On 5/24/17 11:07 PM, John D. Ament wrote:
> Ok,so I think I have a fix working, but one question I have is whether the
> existing PR for performance fixes will be getting merged in to 3.2? While
> its a different problem it touches a lot of the same areas so it will
> create some conflicts if either gets merged first. LIkewise if I have a
> fix for this, would you consider it part of 3.2?
>
> It also seems to me that there's an inherent problem with how some of the
> authorizations are done via Keycloak. Specifically, it seems that a client
> authenticated to master is getting the roles from all realms, which is
> really what is causing these problems. So while I can fix queries, without
> a fix in that area this type of problem can keep popping up.
>
> On Wed, May 24, 2017 at 9:42 AM Stian Thorgersen <sthorger(a)redhat.com>
> wrote:
>
>> That's used by composite roles. It is probably invoked on all roles in the
>> realm. Could probably be fetched eagerly rather than lazy. Can you create a
>> JIRA please?
>>
>> On 24 May 2017 at 12:11, John D. Ament <john.d.ament(a)gmail.com> wrote:
>>
>>> Stian,
>>>
>>> No, I don't believe its in that PR. This seems to be the table
>>> "CHILD_ROLE" which has a large number of queries being executed against
>>> it. But I'm not sure which entity that maps to in your persistence.xml
>>> https://github.com/keycloak/keycloak/blob/master/model/jpa/src/main/resou...
>>>
>>> John
>>>
>>> On Wed, May 24, 2017 at 3:54 AM Stian Thorgersen <sthorger(a)redhat.com>
>>> wrote:
>>>
>>>> Sure, please create a JIRA and link it to
>>>> https://issues.jboss.org/browse/KEYCLOAK-4593
>>>>
>>>> Does this PR help: https://github.com/keycloak/keycloak/pull/3561?
>>>>
>>>> On 23 May 2017 at 15:04, John D. Ament <john.d.ament(a)gmail.com> wrote:
>>>>
>>>>> Stian,
>>>>>
>>>>> We just got a report of a new issue, not sure if its related to the
>>>>> existing but I can create a ticket on your side if it makes sense.
>>>>>
>>>>> When accessing /auth/realms/master/protocol/openid-connect/token we are
>>>>> seeing 3k SQLs being executed of this format:
>>>>>
>>>>> select compositer0_.COMPOSITE as COMPOSIT1_16_0_,
>>>>> compositer0_.CHILD_ROLE as CHILD_RO2_16_0_, roleentity1_.ID as ID1_38_1_,
>>>>> roleentity1_.CLIENT as CLIENT8_38_1_, roleentity1_.CLIENT_REALM_CONSTRAINT
>>>>> as CLIENT_R2_38_1_, roleentity1_.CLIENT_ROLE as CLIENT_R3_38_1_,
>>>>> roleentity1_.DESCRIPTION as DESCRIPT4_38_1_, roleentity1_.NAME as
>>>>> NAME5_38_1_, roleentity1_.REALM as REALM9_38_1_, roleentity1_.REALM_ID as
>>>>> REALM_ID6_38_1_, roleentity1_.SCOPE_PARAM_REQUIRED as SCOPE_PA7_38_1_ from
>>>>> COMPOSITE_ROLE compositer0_ inner join KEYCLOAK_ROLE roleentity1_ on
>>>>> compositer0_.CHILD_ROLE=roleentity1_.ID where compositer0_.COMPOSITE=?
>>>>>
>>>>> On Wed, May 10, 2017 at 12:40 PM John D. Ament <john.d.ament(a)gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Stian,
>>>>>>
>>>>>> Good news. Glad to see these things get prioritized. So far they
>>>>>> look like they're matching the problems I'm running into, specifically
>>>>>> around the whoami endpoint and overall number of SQLs (2800 queries in one
>>>>>> of my tests) and the total number of DB connections allocated within that
>>>>>> one request (3200+).
>>>>>>
>>>>>> John
>>>>>>
>>>>>>
>>>>>> On Wed, May 10, 2017 at 8:02 AM Stian Thorgersen <sthorger(a)redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> There are a number of issues around having a large number of realms.
>>>>>>> We have a general issue open to support this:
>>>>>>> https://issues.jboss.org/browse/KEYCLOAK-4593
>>>>>>>
>>>>>>> We haven't prioritized this in the past, but that has changed and we
>>>>>>> would like to get this sorted out.
>>>>>>>
>>>>>>> There's a few more related PRs including the one you linked:
>>>>>>> https://github.com/keycloak/keycloak/pull/3557
>>>>>>> https://github.com/keycloak/keycloak/pull/3561
>>>>>>>
>>>>>>> On 10 May 2017 at 12:35, John D. Ament <john.d.ament(a)gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> After enabling Keycloak and starting work on a multi-tenant
>>>>>>>> application, it
>>>>>>>> was noted that the admin console started to get very slow in
>>>>>>>> keycloak.
>>>>>>>> After some searching around, it seemed like this was an already
>>>>>>>> reported
>>>>>>>> issue [1] and a fix underway [2]. I was wondering if this fix would
>>>>>>>> make
>>>>>>>> it into 3.2?
>>>>>>>>
>>>>>>>> If additional testing is needed, I'd be happy to help out. Deleting
>>>>>>>> 161
>>>>>>>> realms with minimal clients and users took me 15 minutes via the
>>>>>>>> REST API.
>>>>>>>>
>>>>>>>> [1]: https://issues.jboss.org/browse/KEYCLOAK-4858
>>>>>>>> [2]: https://github.com/keycloak/keycloak/pull/4095
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>>> keycloak-user mailing list
>>>>>>>> keycloak-user(a)lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>>>>>>>>
>>>>>>>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
6 years, 11 months
Change the binding name in ScriptBasedAuthenticator?
by Marek Posolda
Hi Thomas and all,
In 3.2 we did some refactoring and authenticators are now using
authenticationSession instead of ClientSession. I see if we should do
something for the ScriptBasedAuthenticator as it;s still using
"clientSession" as binding where it puts authentication session. I can
see the possibilities:
1) Keep the binding name "clientSession" for backwards compatibility
2) Change the binding name. Probably to "authenticationSession" ? It
would need to be documented in the migration guide.
My vote is to rather go with 2 as people will likely need to refactor
their scripts anyway. Some method signatures are same for authentication
session like was for client session, but not all. WDYT? Other idea?
Marek
6 years, 11 months
Cross-DC and codeToToken request
by Marek Posolda
Followup on some previous emails I sent this week around sticky sessions
and OIDC backchannel requests.
In shortcut, it would be ideal if we can achieve that backchannel
requests (code-to-token, refresh token, logouts etc) can participate in
same sticky session like the browser request. It may be possible in some
cases (our adapters, some loadbalancers, see previous email I sent this
week) but not everytime. And looks we would need to support the case
when it's not possible.
I can start with code-to-token request as it's slightly more complicated
then the others due to the reasons:
1) code must be single-use per OAuth2 / OIDC specification
2) userSession may not yet be available. In case that we use ASYNC
channel for communication between datacenters for transfer userSession
(which I think should be the default due to performance reasons), then
this example flow can happen:
- user successfully authenticated and userSession was created on DC1.
- code-to-token request is sent by the adapter to DC2. Note that this
request is usually sent very quickly after userSession is created.
- DC2 didn't yet received the message from DC1 about the new
userSession. So this userSession not yet available here.
Questions:
1) Could we remove a need from code-to-token endpoint to lookup
userSession? I see this as an option as long as code itself is JWT
signed with realm HMAC key encapsulating some info about user,
session_state etc. Among other things, this would require some
refactoring of protocolMappers (as userSession won't be available when
tokens are generated). But isn't it bad for security to have some claims
directly to the code? It is query parameter, which may end visible in
browser history. IMO this is not big issue, but not 100% sure..
2) Another option is let the code-to-token endpoint wait until
userSession is available. Then we would need support for asynchronous
requests? I can see blocking undertow workers in waiting (something
based on java.util.concurrent.Future) can be an issue and potential for
DoS? Still even with asynchronous, the request times can be quite long.
3) Can we encourage people to use sticky sessions at least for
code-to-token endpoint? We can add the route directly to the code
itself, so the URL will look like:
http://apphost/app?code=123.node1&state=456 . Many loadbalancers seem to
support sticky session based on URL part. But there is also
response_mode=form_post when the code won't be available in the URI.
4) Is it ok to have option to relax on code one-time use? Otherwise in
cross-DC and without sticky session, the every code exchange may require
SYNC request to another DCs to doublecheck code was not used already.
Not good for performance..
For now, I can see some combination of 1,3,4 as a way to go. WDYT?
Marek
6 years, 11 months
Don't import * please!
by Bill Burke
Please make your IDE import the full classname. This is not allowed:
import java.uti.*;
All imports must be fully qualified. A number of people are making this
mistake lately.
Thanks,
Bill
6 years, 11 months
Sticky sessions in backchannel requests
by Marek Posolda
I was thinking about possibilities how to solve the issue with sticky
sessions in backchannel requests. It seems that we need to support the
scenarios, when backchannel requests won't be able to share the sticky
session cookie with the browser. However in many cases it may be
possible that backchannel requests can participate in the "sticky
session", which will have great advantage for performance. Some thoughts:
- OAuth code (the one used in code-to-token backchannel request) can be
JWT signed by realm HMAC key. One of the claims in the code could be
"route-id"
- Refresh tokens will also contain "route-id" claim
- Keycloak OIDC adapters will be able to read the "route-id" from the
code and they will attach sticky session information to the single
backchannel request. Hence the backchannel request will be able to
participate in sticky session and will be properly routed by
loadbalancer to the correct node. Same for requests using refresh token.
- It seems we will need some flexibility how is the "sticky session"
added to the request to support various loadbalancers (cookie,
path-parameters etc). Maybe we need some SPI on adapter side? Or just
some kind of expressions/tokens, which will help to configure how
exactly will cookie/path parameter look like?
- Keycloak.js adapter seems to be working already. Ajax requests are
re-sending the browser cookies and they are available in backchannel
endpoint on server-side. This is true even if AUTH_SESSION_ID cookie is
httpOnly. I've tested with Firefox, Chrome, my mobile phone. Not sure if
this is different in other browsers/devices (cordova etc)?
- For SAML backchannel requests (backchannel logout), we can add the
same to our own SAML adapters though.
- If people use 3rd party adapters, they won't have this. However we can
also have support for some loadbalancers to route the requests directly
based on the content of the route-id inside code (or refresh token). In
many cases, customers will already have established loadbalancer in
their deployments. However some others might be starting their
deployment from the scratch and we can suggest them to use some of our
preferred loadbalancers though.
I've checked that if wildfly+mod_cluster is used as loadbalancer, we
have the flexibility to inject custom undertow handler, which will be
able to look into JWT and add the route info to the HttpServerExchange,
from where mod_cluster will read it. Maybe we can investigate this
possibility in the other popular loadbalancers (nginx etc) ?
So in summary: Backchannel requests won't be able to participate in
sticky session just if: 3rd party adapter is used AND customer can't use
our preferred loadbalancer.
WDYT?
Marek
6 years, 11 months
Provide a Link to go Back to The Application on a Timeout
by Marek Posolda
We have the issue that after session timeout, the page "An error
occurred, please login again through your application." can be shown.
This is even worse when there is no link to go back to the application
as users might be confused what to do. Details in
https://issues.jboss.org/browse/KEYCLOAK-4016 .
This is already handled in many cases as when authentication session is
expired, it is always restarted from the KC_RESTART cookie.
However there are still cases when this error is shown, which is when
the restart from the cookie failed. This can happen when browser history
(including cookies) was cleared or when user restarted the browser (as
the KC_RESTART cookie is not persistent).
Some possibilities to solve:
1) Make the KC_RESTART cookie persistent. That will handle browser
restart, however it won't handle the case when browser history is deleted
2) Add client-id to every link as Stefan Baust suggested. Then we can
add the link to client base uri on the page. This is more work with the
possibility of error-prone if we miss to add the client-id to some link.
Also we will be able to provide the link just if client has "base-uri"
configured.
3) Add the link to the account management application page. After
successful login will be shown list of applications in account
management and user can click to his favourite application. Message
would need to be changed to something like "An error occurred, please
login again through your application or go to the <link>list of
applications<link> and select your application after login."
My preference is 3, 2, 1. WDYT? Any other ideas?
Thanks,
Marek
6 years, 11 months
questions on Marek/Hynek presentation
by Bill Burke
* Won't the regular case be that the load balancer generates the
affinity cookie or doesn't have a cookie at all? HA-Proxy is quite
popular and they have both options.
* @ 18:25 in bluejeans session, The "You are already logged in" screen.
What happens when the use clicks "proceed"? Does the SAML or OIDC
request continue as normal? Or are you calculating the URI on the
application to redirect to, if so, why?
On Action Tokens:
* What is the relationship between the RequiredAction SPI and
ActionTokenHandler SPI? Does every RequiredAction have to have a
corresponding ActionTokenHandler?
* Why would a app developer need to implement an ActionTokenHandler?
Wouldn't it be better for the Required Action SPI to provide the
appropriate metadata so that the handler could be implemented by us?
i.e. isOneTimeToken, email-template, etc, etc. I guess what I'm saying
is that action tokens should be incorporated into the RequiredAction SPI.
* Related to above. Required actions should be able to specify an
"admin console template" and "login template". These would be the
freemarker template to use to create the email that is sent to the
user. "admin console" would be from an admin generating the action.
"login" would be when user login initiates the action email.
* On the Admin Console "Credential Reset" section. Required Action
emails (now Action tokens) aren't necessarily "Credential Resets".
Verify email is not a credential reset. etc. This need to be renamed and
maybe put in another tab?
* We will need a way to offload action processing to another external
service. keycloak exists to mark that the action was completed, but all
the processing for the action happens in an external application. A lot
of people have existing applications they want to integrate with that
perform action processing. Just something to think about. We need this
for other areas of keycloak (i.e. registration).
6 years, 11 months
healthchek API endpoint for environments like kubernetes
by Hector Fernandez
Hi guys,
We have a keycloak cluster setup in Openshift on which our healthchecks
point to the endpoint `auth/version`. It would be nice if you provide an
endpoint in your API that could tell us more info about healthiness or
readiness of Keycloak (especially in a cluster mode).
Do you have any similar endpoint?
Otherwise, it could be a nice feature for envs such as Kubernetes
--
*<hector/>*
--
*<hector/>*
6 years, 11 months
