[keycloak-user] KEYCLOAK-3202 Creating users causes memory leak

Tue Jul 12 11:16:08 EDT 2016

Hi Stian,

>> adding an eviction policy to the realmVersions cache.
> There's a bit more to it as we're now adding the caches internally +
managing the size of them. This to hide it from users as they shouldn't
really be configurable.

Thank you for the explanation.
We apply the eviction policy in production environment tomorrow morning.
We have changed additionally number of owners to 2 for distributable caches
in our configuration. Would it make sense to set default to this value in
standalone-ha.xml same like for the web or ejb caches in the future
versions of Keycloak?

> Are you redirecting the user to the logout endpoint or just calling it?

Yes, we are redirecting explicit to the logout endpoint. But on Thursday we
will roll out a new version of our web aplication, which will simply call
ServletRequest.logout()
Additionally we will log more information in case of exceptions during
logout.

> It could also be that the session is no longer valid when you are
invoking the logout. Sessions expires on the Keycloak server and are
removed when they are expired so could be that the session you are trying
to logout no longer exist on the server and that causes the bad behavior.
You can try to emulate that in the test environment by changing the max
life for a session in the admin console.

I've simulated such situation in combination with ServletRequest.logout()
call:

Keycloak adapter logs an error:

16:55:53,548 ERROR
[org.keycloak.adapters.RefreshableKeycloakSecurityContext]
(ajp-/0.0.0.0:8009-4)  Refresh token failure status: 400
{"error_description":"Refresh token expired","error":"invalid_grant"}

Keycloak server logs a warning:

2016-07-12 16:55:53,536 WARN  [org.keycloak.events] (default task-11)
type=REFRESH_TOKEN_ERROR, realmId=myTS-DEV, clientId=myts-b2c, userId=null,
ipAddress=10.10.10.20, error=invalid_token, grant_type=refresh_token,
client_auth_method=client-secret

User is redirected as expected to login screen. So I'd say that the
behavior is correct.

As already mentioned we will roll out some changes this week. I will inform
you about the effect of the measures next week.

Thank you for your assistance!

2016-07-11 12:08 GMT+02:00 Stian Thorgersen <sthorger at redhat.com>:

>
>
> On 11 July 2016 at 11:08, Valerij Timofeev <valerij.timofeev at gmail.com>
> wrote:
>
>> Thank you for the prompt response Stian.
>>
>> > adding an eviction policy to the realmVersions cache.
>>
>> This was my impression after reading the ticket too, but I was not sure,
>> because according pull request looks a little bit more complicated.
>> We will give a try to this Keycloak setting in the production environment
>> tomorrow.
>> We are going to enable Infinispan statistics additionally to get more
>> information.
>>
>
> There's a bit more to it as we're now adding the caches internally +
> managing the size of them. This to hide it from users as they shouldn't
> really be configurable.
>
>
>>
>> > Is there any errors in the logs?
>>
>> We could identify only errors duiring the service logout until now:
>>
>> Stack Trace:
>>
>> org.keycloak.adapters.ServerRequest.error(ServerRequest.java:228)
>>
>> org.keycloak.adapters.ServerRequest.invokeLogout(ServerRequest.java:82)
>>
>> com.nhp.ts.b2b.services.auth.KcAdminServiceBean.serviceAccountLogout(KcAdminServiceBean.java:330)
>>
>> com.nhp.ts.b2b.services.auth.KcAdminServiceBean.executeAPIpostMethod(KcAdminServiceBean.java:545)
>>
>> sun.reflect.GeneratedMethodAccessor10512.invoke(Unknown Source)
>>
>> ...
>>
>> > What is the status code returned with the empty page?
>>
>> Our web application unfortunately does not log status code and error
>> message. Exception message is null in case of service account logout. We
>> will roll out a fix for this with the next web application release on
>> Thursday this week.
>>
>> Additionally we are going to switch from the OIDC logout endpint method
>> to the ServletRequest.logout() method because it seems to be a more
>> consistent way for a web application which is already protected by Keycloak
>> EAP 6 adapters, isn't it?
>>
>
> Are you redirecting the user to the logout endpoint or just calling it?
>
> ServletRequest.logout()  redirects to the logout endpoint which will
> invalidate the SSO session, then it redirects back to the application and
> the http session is removed. It's certainly simpler to use this directly as
> it takes care of everything.
>
>
>>
>> Additional details about the experienced behaviour: the empty page is our
>> web application internal page. In Google Chrome webbrowser I see for
>> example that the initiator of the last POST request to this internal page
>> was www.googletagmanager.com/gtm.js?id=... Could be this a problem?
>> If I refresh this empty page, I'm back in the web application (still
>> logged in).
>> But if I call OCID logout endpoint
>> (/realms/${realm}/protocol/openid-connect/logout) in the same browser
>> myself  and then refresh the empty page, then I'm redirected to the KC
>> login screen.
>>
>> Any ideas?
>>
>
> It could also be that the session is no longer valid when you are invoking
> the logout. Sessions expires on the Keycloak server and are removed when
> they are expired so could be that the session you are trying to logout no
> longer exist on the server and that causes the bad behavior. You can try to
> emulate that in the test environment by changing the max life for a session
> in the admin console.
>
>
>>
>> Apart from that I hope that we will get more information after the
>> release on Thursday.
>>
>>
>> 2016-07-11 7:37 GMT+02:00 Stian Thorgersen <sthorger at redhat.com>:
>>
>>> Hi,
>>>
>>> You can relatively easily try though by adding an eviction policy to the
>>> realmVersions cache. I found that with roughly a million users there would
>>> be around 500Mb of memory consumed, which will run you into issues with the
>>> default settings if you have that many users login over a space of a day
>>> and a half.
>>>
>>> Empty page could be due to timeout. Is there any errors in the logs?
>>> What is the status code returned with the empty page?
>>>
>>> On 8 July 2016 at 10:40, Valerij Timofeev <valerij.timofeev at gmail.com>
>>> wrote:
>>>
>>>> Hi Stian,
>>>>
>>>> You are the assignee in KEYCLOAK-3202
>>>> <https://issues.jboss.org/browse/KEYCLOAK-3202>, so I addressed this
>>>> email to you directly.
>>>>
>>>> I guess that this issue could be the cause of trouble in our production
>>>> environment.
>>>>
>>>> There are 4 EAP-6 nodes with Keycloak adapters and 2 Keycloak 1.9.4
>>>> standalone servers running in 2 clusters respectively.
>>>>
>>>> We experience logout failures approximately after one and a half days
>>>> of operation.
>>>> Restarting EAP 6 nodes temporary resolves the logout problem.
>>>>
>>>> Durable load tests in out test environment showed that login and logout
>>>> of existing users don't result in above behaviour.
>>>> We added to the durable load test additional scenario creating new
>>>> users and were able to reproduce logout failure: users are getting empty
>>>> page and not the login screen as expected. Page reload navigates back into
>>>> the protected web application .
>>>>
>>>> Logout is accomplished in a Java web applictaion by calling OIDC logout
>>>> endpoint:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *FacesContext                .getCurrentInstance()
>>>> .getExternalContext()
>>>> .redirect(keycloakDeployment.getLogoutUrl().queryParam("redirect_uri",
>>>> redirectURL).toTemplate());*
>>>>
>>>> Logout is initiated via h:commandLink, so I suppose that the OIDC
>>>> logout endpoint is called via the GET method. Should we use the POST method
>>>> instead?
>>>>
>>>> Has servlet logout any advantages?
>>>>
>>>>
>>>>
>>>> *((HttpServletRequest)
>>>> FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();*
>>>> I'd appreciate quick response*, *because restarting production EAP
>>>> cluster every day is not a pleasant option ;-)
>>>>
>>>> Thank you in advance
>>>>
>>>> Kind regards
>>>> Valerij Timofeev
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160712/198eb237/attachment-0001.html 