<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.EmailStyle19
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:Calibri;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Marek,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So the issue won’t be fixed by Keycloak 2.3.0?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Sarp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Marek Posolda <mposolda@redhat.com><br>
<b>Date: </b>Thursday, August 4, 2016 at 7:50 PM<br>
<b>To: </b>Abdullah Sarp <akaya@expedia.com>, "keycloak-user@lists.jboss.org" <keycloak-user@lists.jboss.org><br>
<b>Subject: </b>Re: [keycloak-user] Handling SuspectExceptions in Keycloak<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Hmm... so according to <a href="https://docs.jboss.org/author/display/WFLY10/Infinispan+Subsystem">
https://docs.jboss.org/author/display/WFLY10/Infinispan+Subsystem</a> it seems that you're right. It's not easily possible to add the interceptor through infinispan subsystem :/<br>
<br>
As a workaround, you can probably try to do it programatically. You may need to create your own InfinispanConnectionProviderFactory and configure it in keycloak-server.json . It can override DefaultInfinispanConnectionProviderFactory and add the interceptor
programatically to realms and users caches. Sorry, don't have better proposal to avoid this issue right now :( We likely need to wait until
<a href="https://issues.jboss.org/browse/ISPN-6857">https://issues.jboss.org/browse/ISPN-6857</a> is fixed...<br>
<br>
Marek<br>
<br>
On 02/08/16 06:32, Sarp Kaya wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Marek,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">How do I add the </span>StateTransferInterceptor to the standalone.xml? Isn’t that only doable programmatically?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks,<br>
Sarp<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Marek Posolda
<a href="mailto:mposolda@redhat.com"><mposolda@redhat.com></a><br>
<b>Date: </b>Tuesday, August 2, 2016 at 5:32 AM<br>
<b>To: </b>Abdullah Sarp <a href="mailto:akaya@expedia.com"><akaya@expedia.com></a>,
<a href="mailto:keycloak-user@lists.jboss.org">"keycloak-user@lists.jboss.org"</a>
<a href="mailto:keycloak-user@lists.jboss.org"><keycloak-user@lists.jboss.org></a><br>
<b>Subject: </b>Re: [keycloak-user] Handling SuspectExceptions in Keycloak</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""> </span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">See KC issue [1] and related infinispan issue [2] . <br>
<br>
The workaround is to add the StateTransferInterceptor to the proper place in chain to "realms" and "users" caches. See how I did it programatically. I think that based on that, you should be able to add it to infinispan subsystem as well.<br>
<br>
[1] <a href="https://issues.jboss.org/browse/KEYCLOAK-3306">https://issues.jboss.org/browse/KEYCLOAK-3306</a><br>
[2] <a href="https://issues.jboss.org/browse/ISPN-6857">https://issues.jboss.org/browse/ISPN-6857</a><br>
<br>
Marek<br>
<br>
On 28/07/16 11:53, Sarp Kaya wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">There is already an existing bug report for Infinispan here:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://issues.jboss.org/browse/ISPN-6721">https://issues.jboss.org/browse/ISPN-6721</a></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Currently for Keycloak, if this exception is thrown then it sends an Internal Server Error page to the browser. Essentially what would be really good is that it sends the user back to the login page instead
of displaying Internal Server Error.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This happens when I am consistently sending login and logout (around 40 req/s) requests to two Keycloak instances (let’s call them kc1 and kc2), then one new keycloak instance is started kc3. Kc3 connects
to kc1 and 2 in clustering mode.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now kc1 receives a new request (such as login) and while it is processing that, kc3 is gracefully shut including the cache with this log:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28 09:15:53,656 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 61) WFLYCLINF0003: Stopped sessions cache from keycloak container</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Just shortly after that (6 ms) kc1 throws an exception like this:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28 09:15:53,662 ERROR [io.undertow.request] (default task-48) UT005023: Exception handling request to /auth/realms/{realm}/login-actions/authenticate: org.jboss.resteasy.spi.UnhandledException: org.infinispan.statetransfer.OutdatedTopologyException:
Cache topology changed while the command was executing: expected 175, got 176</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">then shortly after(150 ms) kc1 wants to talk to kc3 and fails to do so with this exception:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28 09:15:53,804 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-54) ISPN000136: Error executing command RemoveCommand, writing keys [f9bde276-dd03-41c9-995b-b1aaf64c1489]:
org.infinispan.remoting.transport.jgroups.SuspectException: Cache not running on node kc3</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.blocks.Request.checkCompletion(Request.java:169)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.JChannel.up(JChannel.java:738)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.stack.Protocol.up(Protocol.java:374)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.FORK.up(FORK.java:118)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.ENCRYPT.up(ENCRYPT.java:454)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:140)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.stack.Protocol.up(Protocol.java:412)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.FD.up(FD.java:260)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.Discovery.up(Discovery.java:295)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> at java.lang.Thread.run(Thread.java:745)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The key that it tries to write is the user-id. After this, the browser receives an Internal Server Error page, which looks like this in html:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><html></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <head></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <title></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Error</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </title></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </head></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> <body></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Internal Server Error</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </body></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"></html></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have configured my infinispan cache settings as following (the rest are default):</span><o:p></o:p></p>
<p class="MsoNormal" style="background:#2B2B2B"><span style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A"> <distributed-cache
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="sessions"
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="5"</span><span style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/><br>
<br>
<distributed-cache </span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="offlineSessions"
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="1"</span><span style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/><br>
<br>
<distributed-cache </span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="loginFailures"
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="1"</span><span style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/></span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have tried many things (such as playing with owner amounts or instance amounts etc). It does not seem to fix this exception. I am well aware that this seems more Infinispan issue than Keycloak, but I believe
that Keycloak at least should respond the end user a better error message (perhaps a login again page) rather than an Internal Server Error page. Could you please handle this exception?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Kind Regards,<br>
Sarp Kaya</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""><br>
<br>
<br>
<br>
</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>keycloak-user mailing list<o:p></o:p></pre>
<pre><a href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a><o:p></o:p></pre>
<pre><a href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""> </span><o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal"><span style="font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>