<div dir="ltr"><div><div><div>It looks like our SSO logout problem has been resolved: there are no errors since our last release day on the 14th of July.<br></div><div>We had to restart the servers at least every 2 days.<br></div><div><br></div>Here recap of the rolled out measures:<br></div>- logout method changed to <i>((HttpServletRequest) FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();</i><br><span class="im"><span>-<span style="color:rgb(0,0,0)"> added eviction policy to the realmVersions cache<br></span></span></span></div><span class="im"><span><span style="color:rgb(0,0,0)">We don't know which of them was the actual solution for our problem, but the current number of entries equals 10000 (=eviction policy) on both nodes<br></span></span></span><div><br><div><br><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-07-13 7:01 GMT+02:00 Stian Thorgersen <span dir="ltr"><<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On 12 July 2016 at 17:16, Valerij Timofeev <span dir="ltr"><<a href="mailto:valerij.timofeev@gmail.com" target="_blank">valerij.timofeev@gmail.com</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Hi Stian,<span><br><br><span class=""><span><span>>> adding an eviction policy to the realmVersions cache.</span></span><br></span></span><span class=""><span>> There's a bit more to it as we're now adding the caches internally +
managing the size of them. This to hide it from users as they shouldn't
really be configurable.<br><br></span></span></div><span class=""><div>Thank you for the explanation.<br></div>We apply the eviction policy in production environment tomorrow morning. <br>We have changed additionally number of owners to 2 for distributable caches in our configuration. Would it make sense to set default to this value in standalone-ha.xml same like for the web or ejb caches in the future versions of Keycloak?</span></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>We've decided to stick with owners set to 1 by default. It's better for performance and in most cases sufficient as users can just re-authenticate if a session is lost.</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><span><br><br>> Are you redirecting the user to the logout endpoint or just calling it?<br><br></span></div>Yes, we are redirecting explicit to the logout endpoint. But on Thursday we will roll out a new version of our web aplication, which will simply call ServletRequest.logout()<br></div>Additionally we will log more information in case of exceptions during logout.<span><br><br>> It could also be that the session is no longer valid when you are
invoking the logout. Sessions expires on the Keycloak server and are
removed when they are expired so could be that the session you are
trying to logout no longer exist on the server and that causes the bad
behavior. You can try to emulate that in the test environment by
changing the max life for a session in the admin console.<br><br></span></div>I've simulated such situation in combination with ServletRequest.logout() call:<br></div><br>Keycloak adapter logs an error:<br><br>16:55:53,548 ERROR [org.keycloak.adapters.RefreshableKeycloakSecurityContext] (ajp-/0.0.0.0:8009-4) Refresh token failure status: 400 {"error_description":"Refresh token expired","error":"invalid_grant"}<br><br></div>Keycloak server logs a warning:<br><br>2016-07-12 16:55:53,536 WARN [org.keycloak.events] (default task-11) type=REFRESH_TOKEN_ERROR, realmId=myTS-DEV, clientId=myts-b2c, userId=null, ipAddress=10.10.10.20, error=invalid_token, grant_type=refresh_token, client_auth_method=client-secret<br><br></div>User is redirected as expected to login screen. So I'd say that the behavior is correct.<br><br></div><div>As already mentioned we will roll out some changes this week. I will inform you about the effect of the measures next week.<br></div><div><br></div>Thank you for your assistance!<br><div><div><br></div></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">2016-07-11 12:08 GMT+02:00 Stian Thorgersen <span dir="ltr"><<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On 11 July 2016 at 11:08, Valerij Timofeev <span dir="ltr"><<a href="mailto:valerij.timofeev@gmail.com" target="_blank">valerij.timofeev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div>Thank you for the prompt response Stian.<span><br><br>> adding an eviction policy to the realmVersions cache.<br><br></span></div>This was my impression after reading the ticket too, but I was not sure, because according pull request looks a little bit more complicated.<br></div>We will give a try to this Keycloak setting in the production environment tomorrow.<br></div><div>We are going to enable Infinispan statistics additionally to get more information.<br></div></div></blockquote><div><br></div></span><div>There's a bit more to it as we're now adding the caches internally + managing the size of them. This to hide it from users as they shouldn't really be configurable.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><span><div><br>> Is there any errors in the logs? <br><br></div></span>We could identify only errors duiring the service logout until now:<br><br>
<pre style="font-size:1em;margin:0px 10px;width:auto;padding:0px"><span style="color:black;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important">Stack Trace:</span></pre>
<pre style="font-size:1em;margin:0px 10px;width:auto;padding:0px"><span style="color:black;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important">org.keycloak.adapters.ServerRequest.error(ServerRequest.java:228)</span></pre>
<pre style="font-size:1em;margin:0px 10px;width:auto;padding:0px"><span style="color:black;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important">org.keycloak.adapters.ServerRequest.invokeLogout(ServerRequest.java:82)</span></pre>
<pre style="font-size:1em;margin:0px 10px;width:auto;padding:0px"><span style="color:black;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important">com.nhp.ts.b2b.services.auth.KcAdminServiceBean.serviceAccountLogout(KcAdminServiceBean.java:330)</span></pre>
<pre style="font-size:1em;margin:0px 10px;width:auto;padding:0px"><span style="color:black;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important">com.nhp.ts.b2b.services.auth.KcAdminServiceBean.executeAPIpostMethod(KcAdminServiceBean.java:545)</span></pre>
<pre style="font-size:1em;margin:0px 10px;width:auto;padding:0px"><span style="color:black;font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important">sun.reflect.GeneratedMethodAccessor10512.invoke(Unknown Source)</span></pre>
...<span><br><div><br>> What is the status code returned with the empty page?<br><br></div></span><div>Our web application unfortunately does not log status code and error message. Exception message is null in case of service account logout. We will roll out a fix for this with the next web application release on Thursday this week.<br></div><div><br>Additionally we are going to switch from the OIDC logout endpint method to the ServletRequest.logout() method because it seems to be a more consistent way for a web application which is already protected by Keycloak EAP 6 adapters, isn't it?<br></div></div></blockquote><div><br></div></span><div>Are you redirecting the user to the logout endpoint or just calling it?</div><div><br></div><div>ServletRequest.logout() redirects to the logout endpoint which will invalidate the SSO session, then it redirects back to the application and the http session is removed. It's certainly simpler to use this directly as it takes care of everything.<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>Additional details about the experienced behaviour: the empty page is our web application internal page. In Google Chrome webbrowser I see for example that the initiator of the last POST request to this internal page was <a href="http://www.googletagmanager.com/gtm.js?id=." target="_blank">www.googletagmanager.com/gtm.js?id=.</a>.. Could be this a problem?<br></div><div>If I refresh this empty page, I'm back in the web application (still logged in).<br></div><div>But if I call OCID logout endpoint (/realms/${realm}/protocol/openid-connect/logout) in the same browser myself and then refresh the empty page, then I'm redirected to the KC login screen.<br><br></div><div>Any ideas?<br></div></div></blockquote><div><br></div></span><div>It could also be that the session is no longer valid when you are invoking the logout. Sessions expires on the Keycloak server and are removed when they are expired so could be that the session you are trying to logout no longer exist on the server and that causes the bad behavior. You can try to emulate that in the test environment by changing the max life for a session in the admin console.</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>Apart from that I hope that we will get more information after the release on Thursday.<br></div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">2016-07-11 7:37 GMT+02:00 Stian Thorgersen <span dir="ltr"><<a href="mailto:sthorger@redhat.com" target="_blank">sthorger@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>You can relatively easily try though by adding an eviction policy to the realmVersions cache. I found that with roughly a million users there would be around 500Mb of memory consumed, which will run you into issues with the default settings if you have that many users login over a space of a day and a half.</div><div><br></div><div>Empty page could be due to timeout. Is there any errors in the logs? What is the status code returned with the empty page?</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On 8 July 2016 at 10:40, Valerij Timofeev <span dir="ltr"><<a href="mailto:valerij.timofeev@gmail.com" target="_blank">valerij.timofeev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div>Hi Stian,<br><br></div>You are the assignee in <a href="https://issues.jboss.org/browse/KEYCLOAK-3202" rel="12635573" target="_blank">KEYCLOAK-3202</a>, so I addressed this email to you directly.<br><br></div>I guess that this issue could be the cause of trouble in our production environment.<br><br>There are 4 EAP-6 nodes with Keycloak adapters and 2 Keycloak 1.9.4 standalone servers running in 2 clusters respectively.<br><br></div>We experience logout failures approximately after one and a half days of operation. <br>Restarting EAP 6 nodes temporary resolves the logout problem.<br><br></div>Durable load tests in out test environment showed that login and logout of existing users don't result in above behaviour.<br></div>We added to the durable load test additional scenario creating new users and were able to reproduce logout failure: users are getting empty page and not the login screen as expected. Page reload navigates back into the protected web application .<br><br></div>Logout is accomplished in a Java web applictaion by calling OIDC logout endpoint:<br><br><i>FacesContext<br> .getCurrentInstance()<br> .getExternalContext()<br> .redirect(keycloakDeployment.getLogoutUrl().queryParam("redirect_uri", redirectURL).toTemplate());<br></i></div><br>Logout is initiated via h:commandLink, so I suppose that the OIDC logout endpoint is called via the GET method. Should we use the POST method instead?<br></div><br>Has servlet logout any advantages?<br><br><i>((HttpServletRequest) FacesContext.getCurrentInstance().getExternalContext().getRequest()).logout();<br><br></i></div>I'd appreciate quick response<i>, </i>because restarting production EAP cluster every day is not a pleasant option ;-)<span lang="en"><span><br><br></span></span></div><span lang="en"><span>Thank you in advance<br><br></span></span></div><span lang="en"><span>Kind regards<span><font color="#888888"><br></font></span></span></span></div><span><font color="#888888"><span lang="en"><span>Valerij Timofeev<br></span></span></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>