<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">See KC issue [1] and related infinispan
issue [2] . <br>
<br>
The workaround is to add the StateTransferInterceptor to the
proper place in chain to "realms" and "users" caches. See how I
did it programatically. I think that based on that, you should be
able to add it to infinispan subsystem as well.<br>
<br>
[1] <a class="moz-txt-link-freetext" href="https://issues.jboss.org/browse/KEYCLOAK-3306">https://issues.jboss.org/browse/KEYCLOAK-3306</a><br>
[2] <a class="moz-txt-link-freetext" href="https://issues.jboss.org/browse/ISPN-6857">https://issues.jboss.org/browse/ISPN-6857</a><br>
<br>
Marek<br>
<br>
On 28/07/16 11:53, Sarp Kaya wrote:<br>
</div>
<blockquote
cite="mid:035C5A88-0C9C-4D44-A83D-4A227AFF48B6@expedia.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">There is
already an existing bug report for Infinispan here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a
moz-do-not-send="true"
href="https://issues.jboss.org/browse/ISPN-6721"><a class="moz-txt-link-freetext" href="https://issues.jboss.org/browse/ISPN-6721">https://issues.jboss.org/browse/ISPN-6721</a></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Currently
for Keycloak, if this exception is thrown then it sends an
Internal Server Error page to the browser. Essentially what
would be really good is that it sends the user back to the
login page instead of displaying Internal Server Error.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This happens
when I am consistently sending login and logout (around 40
req/s) requests to two Keycloak instances (let’s call them
kc1 and kc2), then one new keycloak instance is started kc3.
Kc3 connects to kc1 and 2 in clustering mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now kc1
receives a new request (such as login) and while it is
processing that, kc3 is gracefully shut including the cache
with this log:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28
09:15:53,656 INFO [org.jboss.as.clustering.infinispan]
(ServerService Thread Pool -- 61) WFLYCLINF0003: Stopped
sessions cache from keycloak container<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Just shortly
after that (6 ms) kc1 throws an exception like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28
09:15:53,662 ERROR [io.undertow.request] (default task-48)
UT005023: Exception handling request to
/auth/realms/{realm}/login-actions/authenticate:
org.jboss.resteasy.spi.UnhandledException:
org.infinispan.statetransfer.OutdatedTopologyException:
Cache topology changed while the command was executing:
expected 175, got 176<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">then shortly
after(150 ms) kc1 wants to talk to kc3 and fails to do so
with this exception:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28
09:15:53,804 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor]
(default task-54) ISPN000136: Error executing command
RemoveCommand, writing keys
[f9bde276-dd03-41c9-995b-b1aaf64c1489]:
org.infinispan.remoting.transport.jgroups.SuspectException:
Cache not running on node kc3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.Request.checkCompletion(Request.java:169)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.JChannel.up(JChannel.java:738)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.stack.Protocol.up(Protocol.java:374)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FORK.up(FORK.java:118)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.FlowControl.up(FlowControl.java:394)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.ENCRYPT.up(ENCRYPT.java:454)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:140)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.stack.Protocol.up(Protocol.java:412)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FD.up(FD.java:260)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.Discovery.up(Discovery.java:295)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at java.lang.Thread.run(Thread.java:745)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The key that
it tries to write is the user-id. After this, the browser
receives an Internal Server Error page, which looks like
this in html:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><html><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
<head><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
<title><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
</title><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
</head><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
<body><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Internal
Server Error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
</body><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"></html><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have
configured my infinispan cache settings as following (the
rest are default):<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#2B2B2B"><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">
<distributed-cache
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="sessions"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="5"</span><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/><br>
<br>
<distributed-cache </span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="offlineSessions"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="1"</span><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/><br>
<br>
<distributed-cache </span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="loginFailures"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="1"</span><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/></span><span
style="font-size:9.0pt;font-family:Menlo;color:#A9B7C6"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have tried
many things (such as playing with owner amounts or instance
amounts etc). It does not seem to fix this exception. I am
well aware that this seems more Infinispan issue than
Keycloak, but I believe that Keycloak at least should
respond the end user a better error message (perhaps a login
again page) rather than an Internal Server Error page. Could
you please handle this exception?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Kind
Regards,<br>
Sarp Kaya<o:p></o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
keycloak-user mailing list
<a class="moz-txt-link-abbreviated" href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
</blockquote>
<br>
</body>
</html>