[keycloak-user] KEYCLOAK-3202 Creating users causes memory leak

Tue Jul 19 12:07:41 EDT 2016

It looks like our SSO logout problem has been resolved: there are no errors
since our last release day on the 14th of July.
We had to restart the servers at least every 2 days.

Here recap of the rolled out measures:
- logout method changed to *((HttpServletRequest)
FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();*
- added eviction policy to the realmVersions cache
We don't know which of them was the actual solution for our problem, but
the current number of entries equals 10000 (=eviction policy) on both nodes

2016-07-13 7:01 GMT+02:00 Stian Thorgersen <sthorger at redhat.com>:

>
>
> On 12 July 2016 at 17:16, Valerij Timofeev <valerij.timofeev at gmail.com>
> wrote:
>
>> Hi Stian,
>>
>> >> adding an eviction policy to the realmVersions cache.
>> > There's a bit more to it as we're now adding the caches internally +
>> managing the size of them. This to hide it from users as they shouldn't
>> really be configurable.
>>
>> Thank you for the explanation.
>> We apply the eviction policy in production environment tomorrow morning.
>> We have changed additionally number of owners to 2 for distributable
>> caches in our configuration. Would it make sense to set default to this
>> value in standalone-ha.xml same like for the web or ejb caches in the
>> future versions of Keycloak?
>>
>
> We've decided to stick with owners set to 1 by default. It's better for
> performance and in most cases sufficient as users can just re-authenticate
> if a session is lost.
>
>
>>
>>
>> > Are you redirecting the user to the logout endpoint or just calling it?
>>
>> Yes, we are redirecting explicit to the logout endpoint. But on Thursday
>> we will roll out a new version of our web aplication, which will simply
>> call ServletRequest.logout()
>> Additionally we will log more information in case of exceptions during
>> logout.
>>
>> > It could also be that the session is no longer valid when you are
>> invoking the logout. Sessions expires on the Keycloak server and are
>> removed when they are expired so could be that the session you are trying
>> to logout no longer exist on the server and that causes the bad behavior.
>> You can try to emulate that in the test environment by changing the max
>> life for a session in the admin console.
>>
>> I've simulated such situation in combination with ServletRequest.logout()
>> call:
>>
>> Keycloak adapter logs an error:
>>
>> 16:55:53,548 ERROR
>> [org.keycloak.adapters.RefreshableKeycloakSecurityContext]
>> (ajp-/0.0.0.0:8009-4)  Refresh token failure status: 400
>> {"error_description":"Refresh token expired","error":"invalid_grant"}
>>
>> Keycloak server logs a warning:
>>
>> 2016-07-12 16:55:53,536 WARN  [org.keycloak.events] (default task-11)
>> type=REFRESH_TOKEN_ERROR, realmId=myTS-DEV, clientId=myts-b2c, userId=null,
>> ipAddress=10.10.10.20, error=invalid_token, grant_type=refresh_token,
>> client_auth_method=client-secret
>>
>> User is redirected as expected to login screen. So I'd say that the
>> behavior is correct.
>>
>> As already mentioned we will roll out some changes this week. I will
>> inform you about the effect of the measures next week.
>>
>> Thank you for your assistance!
>>
>>
>> 2016-07-11 12:08 GMT+02:00 Stian Thorgersen <sthorger at redhat.com>:
>>
>>>
>>>
>>> On 11 July 2016 at 11:08, Valerij Timofeev <valerij.timofeev at gmail.com>
>>> wrote:
>>>
>>>> Thank you for the prompt response Stian.
>>>>
>>>> > adding an eviction policy to the realmVersions cache.
>>>>
>>>> This was my impression after reading the ticket too, but I was not
>>>> sure, because according pull request looks a little bit more complicated.
>>>> We will give a try to this Keycloak setting in the production
>>>> environment tomorrow.
>>>> We are going to enable Infinispan statistics additionally to get more
>>>> information.
>>>>
>>>
>>> There's a bit more to it as we're now adding the caches internally +
>>> managing the size of them. This to hide it from users as they shouldn't
>>> really be configurable.
>>>
>>>
>>>>
>>>> > Is there any errors in the logs?
>>>>
>>>> We could identify only errors duiring the service logout until now:
>>>>
>>>> Stack Trace:
>>>>
>>>> org.keycloak.adapters.ServerRequest.error(ServerRequest.java:228)
>>>>
>>>> org.keycloak.adapters.ServerRequest.invokeLogout(ServerRequest.java:82)
>>>>
>>>> com.nhp.ts.b2b.services.auth.KcAdminServiceBean.serviceAccountLogout(KcAdminServiceBean.java:330)
>>>>
>>>> com.nhp.ts.b2b.services.auth.KcAdminServiceBean.executeAPIpostMethod(KcAdminServiceBean.java:545)
>>>>
>>>> sun.reflect.GeneratedMethodAccessor10512.invoke(Unknown Source)
>>>>
>>>> ...
>>>>
>>>> > What is the status code returned with the empty page?
>>>>
>>>> Our web application unfortunately does not log status code and error
>>>> message. Exception message is null in case of service account logout. We
>>>> will roll out a fix for this with the next web application release on
>>>> Thursday this week.
>>>>
>>>> Additionally we are going to switch from the OIDC logout endpint method
>>>> to the ServletRequest.logout() method because it seems to be a more
>>>> consistent way for a web application which is already protected by Keycloak
>>>> EAP 6 adapters, isn't it?
>>>>
>>>
>>> Are you redirecting the user to the logout endpoint or just calling it?
>>>
>>> ServletRequest.logout()  redirects to the logout endpoint which will
>>> invalidate the SSO session, then it redirects back to the application and
>>> the http session is removed. It's certainly simpler to use this directly as
>>> it takes care of everything.
>>>
>>>
>>>>
>>>> Additional details about the experienced behaviour: the empty page is
>>>> our web application internal page. In Google Chrome webbrowser I see for
>>>> example that the initiator of the last POST request to this internal page
>>>> was www.googletagmanager.com/gtm.js?id=... Could be this a problem?
>>>> If I refresh this empty page, I'm back in the web application (still
>>>> logged in).
>>>> But if I call OCID logout endpoint
>>>> (/realms/${realm}/protocol/openid-connect/logout) in the same browser
>>>> myself  and then refresh the empty page, then I'm redirected to the KC
>>>> login screen.
>>>>
>>>> Any ideas?
>>>>
>>>
>>> It could also be that the session is no longer valid when you are
>>> invoking the logout. Sessions expires on the Keycloak server and are
>>> removed when they are expired so could be that the session you are trying
>>> to logout no longer exist on the server and that causes the bad behavior.
>>> You can try to emulate that in the test environment by changing the max
>>> life for a session in the admin console.
>>>
>>>
>>>>
>>>> Apart from that I hope that we will get more information after the
>>>> release on Thursday.
>>>>
>>>>
>>>> 2016-07-11 7:37 GMT+02:00 Stian Thorgersen <sthorger at redhat.com>:
>>>>
>>>>> Hi,
>>>>>
>>>>> You can relatively easily try though by adding an eviction policy to
>>>>> the realmVersions cache. I found that with roughly a million users there
>>>>> would be around 500Mb of memory consumed, which will run you into issues
>>>>> with the default settings if you have that many users login over a space of
>>>>> a day and a half.
>>>>>
>>>>> Empty page could be due to timeout. Is there any errors in the logs?
>>>>> What is the status code returned with the empty page?
>>>>>
>>>>> On 8 July 2016 at 10:40, Valerij Timofeev <valerij.timofeev at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Stian,
>>>>>>
>>>>>> You are the assignee in KEYCLOAK-3202
>>>>>> <https://issues.jboss.org/browse/KEYCLOAK-3202>, so I addressed this
>>>>>> email to you directly.
>>>>>>
>>>>>> I guess that this issue could be the cause of trouble in our
>>>>>> production environment.
>>>>>>
>>>>>> There are 4 EAP-6 nodes with Keycloak adapters and 2 Keycloak 1.9.4
>>>>>> standalone servers running in 2 clusters respectively.
>>>>>>
>>>>>> We experience logout failures approximately after one and a half days
>>>>>> of operation.
>>>>>> Restarting EAP 6 nodes temporary resolves the logout problem.
>>>>>>
>>>>>> Durable load tests in out test environment showed that login and
>>>>>> logout of existing users don't result in above behaviour.
>>>>>> We added to the durable load test additional scenario creating new
>>>>>> users and were able to reproduce logout failure: users are getting empty
>>>>>> page and not the login screen as expected. Page reload navigates back into
>>>>>> the protected web application .
>>>>>>
>>>>>> Logout is accomplished in a Java web applictaion by calling OIDC
>>>>>> logout endpoint:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *FacesContext                .getCurrentInstance()
>>>>>> .getExternalContext()
>>>>>> .redirect(keycloakDeployment.getLogoutUrl().queryParam("redirect_uri",
>>>>>> redirectURL).toTemplate());*
>>>>>>
>>>>>> Logout is initiated via h:commandLink, so I suppose that the OIDC
>>>>>> logout endpoint is called via the GET method. Should we use the POST method
>>>>>> instead?
>>>>>>
>>>>>> Has servlet logout any advantages?
>>>>>>
>>>>>>
>>>>>>
>>>>>> *((HttpServletRequest)
>>>>>> FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();*
>>>>>> I'd appreciate quick response*, *because restarting production EAP
>>>>>> cluster every day is not a pleasant option ;-)
>>>>>>
>>>>>> Thank you in advance
>>>>>>
>>>>>> Kind regards
>>>>>> Valerij Timofeev
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160719/82b4bcc0/attachment-0001.html 