adding an eviction policy to the realmVersions cache.

Is there any errors in the logs?

What is the status code returned with the empty page?

Hi, You can relatively easily try though by adding an eviction policy to the realmVersions cache. I found that with roughly a million users there would be around 500Mb of memory consumed, which will run you into issues with the default settings if you have that many users login over a space of a day and a half. Empty page could be due to timeout. Is there any errors in the logs? What is the status code returned with the empty page? On 8 July 2016 at 10:40, Valerij Timofeev <valerij.timofeev(a)gmail.com&gt; wrote: > Hi Stian, > > You are the assignee in KEYCLOAK-3202 > <https://issues.jboss.org/browse/KEYCLOAK-3202>;, so I addressed this > email to you directly. > > I guess that this issue could be the cause of trouble in our production > environment. > > There are 4 EAP-6 nodes with Keycloak adapters and 2 Keycloak 1.9.4 > standalone servers running in 2 clusters respectively. > > We experience logout failures approximately after one and a half days of > operation. > Restarting EAP 6 nodes temporary resolves the logout problem. > > Durable load tests in out test environment showed that login and logout > of existing users don't result in above behaviour. > We added to the durable load test additional scenario creating new users > and were able to reproduce logout failure: users are getting empty page and > not the login screen as expected. Page reload navigates back into the > protected web application . > > Logout is accomplished in a Java web applictaion by calling OIDC logout > endpoint: > > > > > > *FacesContext .getCurrentInstance() > .getExternalContext() > .redirect(keycloakDeployment.getLogoutUrl().queryParam("redirect_uri", > redirectURL).toTemplate());* > > Logout is initiated via h:commandLink, so I suppose that the OIDC logout > endpoint is called via the GET method. Should we use the POST method > instead? > > Has servlet logout any advantages? > > > > *((HttpServletRequest) > FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();* > I'd appreciate quick response*, *because restarting production EAP > cluster every day is not a pleasant option ;-) > > Thank you in advance > > Kind regards > Valerij Timofeev >

> adding an eviction policy to the realmVersions cache. There's a bit more to it as we're now adding the caches internally +

Are you redirecting the user to the logout endpoint or just calling it?

It could also be that the session is no longer valid when you are

Are you redirecting the user to the logout endpoint or just calling it?

On 12 July 2016 at 17:16, Valerij Timofeev <valerij.timofeev(a)gmail.com&gt; wrote: > Hi Stian, > > >> adding an eviction policy to the realmVersions cache. > > There's a bit more to it as we're now adding the caches internally + > managing the size of them. This to hide it from users as they shouldn't > really be configurable. > > Thank you for the explanation. > We apply the eviction policy in production environment tomorrow morning. > We have changed additionally number of owners to 2 for distributable > caches in our configuration. Would it make sense to set default to this > value in standalone-ha.xml same like for the web or ejb caches in the > future versions of Keycloak? > We've decided to stick with owners set to 1 by default. It's better for performance and in most cases sufficient as users can just re-authenticate if a session is lost. > > > > Are you redirecting the user to the logout endpoint or just calling it? > > Yes, we are redirecting explicit to the logout endpoint. But on Thursday > we will roll out a new version of our web aplication, which will simply > call ServletRequest.logout() > Additionally we will log more information in case of exceptions during > logout. > > > It could also be that the session is no longer valid when you are > invoking the logout. Sessions expires on the Keycloak server and are > removed when they are expired so could be that the session you are trying > to logout no longer exist on the server and that causes the bad behavior. > You can try to emulate that in the test environment by changing the max > life for a session in the admin console. > > I've simulated such situation in combination with ServletRequest.logout() > call: > > Keycloak adapter logs an error: > > 16:55:53,548 ERROR > [org.keycloak.adapters.RefreshableKeycloakSecurityContext] > (ajp-/0.0.0.0:8009-4) Refresh token failure status: 400 > {"error_description":"Refresh token expired","error":"invalid_grant"} > > Keycloak server logs a warning: > > 2016-07-12 16:55:53,536 WARN [org.keycloak.events] (default task-11) > type=REFRESH_TOKEN_ERROR, realmId=myTS-DEV, clientId=myts-b2c, userId=null, > ipAddress=10.10.10.20, error=invalid_token, grant_type=refresh_token, > client_auth_method=client-secret > > User is redirected as expected to login screen. So I'd say that the > behavior is correct. > > As already mentioned we will roll out some changes this week. I will > inform you about the effect of the measures next week. > > Thank you for your assistance! > > > 2016-07-11 12:08 GMT+02:00 Stian Thorgersen <sthorger(a)redhat.com&gt;: > >> >> >> On 11 July 2016 at 11:08, Valerij Timofeev <valerij.timofeev(a)gmail.com&gt; >> wrote: >> >>> Thank you for the prompt response Stian. >>> >>> > adding an eviction policy to the realmVersions cache. >>> >>> This was my impression after reading the ticket too, but I was not >>> sure, because according pull request looks a little bit more complicated. >>> We will give a try to this Keycloak setting in the production >>> environment tomorrow. >>> We are going to enable Infinispan statistics additionally to get more >>> information. >>> >> >> There's a bit more to it as we're now adding the caches internally + >> managing the size of them. This to hide it from users as they shouldn't >> really be configurable. >> >> >>> >>> > Is there any errors in the logs? >>> >>> We could identify only errors duiring the service logout until now: >>> >>> Stack Trace: >>> >>> org.keycloak.adapters.ServerRequest.error(ServerRequest.java:228) >>> >>> org.keycloak.adapters.ServerRequest.invokeLogout(ServerRequest.java:82) >>> >>> com.nhp.ts.b2b.services.auth.KcAdminServiceBean.serviceAccountLogout(KcAdminServiceBean.java:330) >>> >>> com.nhp.ts.b2b.services.auth.KcAdminServiceBean.executeAPIpostMethod(KcAdminServiceBean.java:545) >>> >>> sun.reflect.GeneratedMethodAccessor10512.invoke(Unknown Source) >>> >>> ... >>> >>> > What is the status code returned with the empty page? >>> >>> Our web application unfortunately does not log status code and error >>> message. Exception message is null in case of service account logout. We >>> will roll out a fix for this with the next web application release on >>> Thursday this week. >>> >>> Additionally we are going to switch from the OIDC logout endpint method >>> to the ServletRequest.logout() method because it seems to be a more >>> consistent way for a web application which is already protected by Keycloak >>> EAP 6 adapters, isn't it? >>> >> >> Are you redirecting the user to the logout endpoint or just calling it? >> >> ServletRequest.logout() redirects to the logout endpoint which will >> invalidate the SSO session, then it redirects back to the application and >> the http session is removed. It's certainly simpler to use this directly as >> it takes care of everything. >> >> >>> >>> Additional details about the experienced behaviour: the empty page is >>> our web application internal page. In Google Chrome webbrowser I see for >>> example that the initiator of the last POST request to this internal page >>> was www.googletagmanager.com/gtm.js?id=... Could be this a problem? >>> If I refresh this empty page, I'm back in the web application (still >>> logged in). >>> But if I call OCID logout endpoint >>> (/realms/${realm}/protocol/openid-connect/logout) in the same browser >>> myself and then refresh the empty page, then I'm redirected to the KC >>> login screen. >>> >>> Any ideas? >>> >> >> It could also be that the session is no longer valid when you are >> invoking the logout. Sessions expires on the Keycloak server and are >> removed when they are expired so could be that the session you are trying >> to logout no longer exist on the server and that causes the bad behavior. >> You can try to emulate that in the test environment by changing the max >> life for a session in the admin console. >> >> >>> >>> Apart from that I hope that we will get more information after the >>> release on Thursday. >>> >>> >>> 2016-07-11 7:37 GMT+02:00 Stian Thorgersen <sthorger(a)redhat.com&gt;: >>> >>>> Hi, >>>> >>>> You can relatively easily try though by adding an eviction policy to >>>> the realmVersions cache. I found that with roughly a million users there >>>> would be around 500Mb of memory consumed, which will run you into issues >>>> with the default settings if you have that many users login over a space of >>>> a day and a half. >>>> >>>> Empty page could be due to timeout. Is there any errors in the logs? >>>> What is the status code returned with the empty page? >>>> >>>> On 8 July 2016 at 10:40, Valerij Timofeev <valerij.timofeev(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Hi Stian, >>>>> >>>>> You are the assignee in KEYCLOAK-3202 >>>>> <https://issues.jboss.org/browse/KEYCLOAK-3202>;, so I addressed this >>>>> email to you directly. >>>>> >>>>> I guess that this issue could be the cause of trouble in our >>>>> production environment. >>>>> >>>>> There are 4 EAP-6 nodes with Keycloak adapters and 2 Keycloak 1.9.4 >>>>> standalone servers running in 2 clusters respectively. >>>>> >>>>> We experience logout failures approximately after one and a half days >>>>> of operation. >>>>> Restarting EAP 6 nodes temporary resolves the logout problem. >>>>> >>>>> Durable load tests in out test environment showed that login and >>>>> logout of existing users don't result in above behaviour. >>>>> We added to the durable load test additional scenario creating new >>>>> users and were able to reproduce logout failure: users are getting empty >>>>> page and not the login screen as expected. Page reload navigates back into >>>>> the protected web application . >>>>> >>>>> Logout is accomplished in a Java web applictaion by calling OIDC >>>>> logout endpoint: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *FacesContext .getCurrentInstance() >>>>> .getExternalContext() >>>>> .redirect(keycloakDeployment.getLogoutUrl().queryParam("redirect_uri", >>>>> redirectURL).toTemplate());* >>>>> >>>>> Logout is initiated via h:commandLink, so I suppose that the OIDC >>>>> logout endpoint is called via the GET method. Should we use the POST method >>>>> instead? >>>>> >>>>> Has servlet logout any advantages? >>>>> >>>>> >>>>> >>>>> *((HttpServletRequest) >>>>> FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();* >>>>> I'd appreciate quick response*, *because restarting production EAP >>>>> cluster every day is not a pleasant option ;-) >>>>> >>>>> Thank you in advance >>>>> >>>>> Kind regards >>>>> Valerij Timofeev >>>>> >>>> >>>> >>> >> >