<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Btv. another possibility to fix is, to
doublecheck if you're not updating user (or other object) on every
login. This may cause that user is always invalidated during each
login. The result is the OutdatedTopologyException, but also the
bad performance.<br>
<br>
Marek<br>
<br>
On 01/08/16 21:32, Marek Posolda wrote:<br>
</div>
<blockquote cite="mid:579FA3E7.4000809@redhat.com" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div class="moz-cite-prefix">See KC issue [1] and related
infinispan issue [2] . <br>
<br>
The workaround is to add the StateTransferInterceptor to the
proper place in chain to "realms" and "users" caches. See how I
did it programatically. I think that based on that, you should
be able to add it to infinispan subsystem as well.<br>
<br>
[1] <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://issues.jboss.org/browse/KEYCLOAK-3306">https://issues.jboss.org/browse/KEYCLOAK-3306</a><br>
[2] <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://issues.jboss.org/browse/ISPN-6857">https://issues.jboss.org/browse/ISPN-6857</a><br>
<br>
Marek<br>
<br>
On 28/07/16 11:53, Sarp Kaya wrote:<br>
</div>
<blockquote
cite="mid:035C5A88-0C9C-4D44-A83D-4A227AFF48B6@expedia.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Courier New";
panose-1:2 7 3 9 2 2 5 2 4 4;}
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">There is
already an existing bug report for Infinispan here:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://issues.jboss.org/browse/ISPN-6721"><a class="moz-txt-link-freetext" href="https://issues.jboss.org/browse/ISPN-6721">https://issues.jboss.org/browse/ISPN-6721</a></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Currently
for Keycloak, if this exception is thrown then it sends an
Internal Server Error page to the browser. Essentially
what would be really good is that it sends the user back
to the login page instead of displaying Internal Server
Error.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This
happens when I am consistently sending login and logout
(around 40 req/s) requests to two Keycloak instances
(let’s call them kc1 and kc2), then one new keycloak
instance is started kc3. Kc3 connects to kc1 and 2 in
clustering mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Now kc1
receives a new request (such as login) and while it is
processing that, kc3 is gracefully shut including the
cache with this log:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28
09:15:53,656 INFO [org.jboss.as.clustering.infinispan]
(ServerService Thread Pool -- 61) WFLYCLINF0003: Stopped
sessions cache from keycloak container<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Just
shortly after that (6 ms) kc1 throws an exception like
this:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28
09:15:53,662 ERROR [io.undertow.request] (default task-48)
UT005023: Exception handling request to
/auth/realms/{realm}/login-actions/authenticate:
org.jboss.resteasy.spi.UnhandledException:
org.infinispan.statetransfer.OutdatedTopologyException:
Cache topology changed while the command was executing:
expected 175, got 176<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">then
shortly after(150 ms) kc1 wants to talk to kc3 and fails
to do so with this exception:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">2016-07-28
09:15:53,804 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor]
(default task-54) ISPN000136: Error executing command
RemoveCommand, writing keys
[f9bde276-dd03-41c9-995b-b1aaf64c1489]:
org.infinispan.remoting.transport.jgroups.SuspectException:
Cache not running on node kc3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.Request.checkCompletion(Request.java:169)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.JChannel.up(JChannel.java:738)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.stack.Protocol.up(Protocol.java:374)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FORK.up(FORK.java:118)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.FlowControl.up(FlowControl.java:394)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.ENCRYPT.up(ENCRYPT.java:454)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:140)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.stack.Protocol.up(Protocol.java:412)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FD.up(FD.java:260)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.Discovery.up(Discovery.java:295)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
at java.lang.Thread.run(Thread.java:745)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The key
that it tries to write is the user-id. After this, the
browser receives an Internal Server Error page, which
looks like this in html:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><html><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
<head><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
<title><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
Error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
</title><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
</head><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
<body><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
Internal Server Error<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">
</body><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"></html><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have
configured my infinispan cache settings as following (the
rest are default):<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#2B2B2B"><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">
<distributed-cache </span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="sessions"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="5"</span><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/><br>
<br>
<distributed-cache </span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="offlineSessions"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="1"</span><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/><br>
<br>
<distributed-cache </span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">name</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="loginFailures"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">mode</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="SYNC"
</span><span
style="font-size:9.0pt;font-family:Menlo;color:#BABABA">owners</span><span
style="font-size:9.0pt;font-family:Menlo;color:#6A8759">="1"</span><span
style="font-size:9.0pt;font-family:Menlo;color:#E8BF6A">/></span><span
style="font-size:9.0pt;font-family:Menlo;color:#A9B7C6"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have
tried many things (such as playing with owner amounts or
instance amounts etc). It does not seem to fix this
exception. I am well aware that this seems more Infinispan
issue than Keycloak, but I believe that Keycloak at least
should respond the end user a better error message
(perhaps a login again page) rather than an Internal
Server Error page. Could you please handle this exception?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Kind
Regards,<br>
Sarp Kaya<o:p></o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
keycloak-user mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:keycloak-user@lists.jboss.org">keycloak-user@lists.jboss.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/keycloak-user">https://lists.jboss.org/mailman/listinfo/keycloak-user</a></pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>