[keycloak-user] Handling SuspectExceptions in Keycloak

Mon Aug 1 15:32:55 EDT 2016

See KC issue [1] and related infinispan issue [2] .

The workaround is to add the StateTransferInterceptor to the proper 
place in chain to "realms" and "users" caches. See how I did it 
programatically. I think that based on that, you should be able to add 
it to infinispan subsystem as well.

[1] https://issues.jboss.org/browse/KEYCLOAK-3306
[2] https://issues.jboss.org/browse/ISPN-6857

Marek

On 28/07/16 11:53, Sarp Kaya wrote:
>
> Hello,
>
> There is already an existing bug report for Infinispan here:
>
> https://issues.jboss.org/browse/ISPN-6721
>
> Currently for Keycloak, if this exception is thrown then it sends an 
> Internal Server Error page to the browser. Essentially what would be 
> really good is that it sends the user back to the login page instead 
> of displaying Internal Server Error.
>
> This happens when I am consistently sending login and logout (around 
> 40 req/s) requests to two Keycloak instances (let’s call them kc1 and 
> kc2), then one new keycloak instance is started kc3. Kc3 connects to 
> kc1 and 2 in clustering mode.
>
> Now kc1 receives a new request (such as login) and while it is 
> processing that, kc3 is gracefully shut including the cache with this log:
>
> 2016-07-28 09:15:53,656 INFO  [org.jboss.as.clustering.infinispan] 
> (ServerService Thread Pool -- 61) WFLYCLINF0003: Stopped sessions 
> cache from keycloak container
>
> Just shortly after that (6 ms) kc1 throws an exception like this:
>
> 2016-07-28 09:15:53,662 ERROR [io.undertow.request] (default task-48) 
> UT005023: Exception handling request to 
> /auth/realms/{realm}/login-actions/authenticate: 
> org.jboss.resteasy.spi.UnhandledException: 
> org.infinispan.statetransfer.OutdatedTopologyException: Cache topology 
> changed while the command was executing: expected 175, got 176
>
> at 
> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)
>
> at 
> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
>
> at 
> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)
>
> at 
> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>
> then shortly after(150 ms) kc1 wants to talk to kc3 and fails to do so 
> with this exception:
>
> 2016-07-28 09:15:53,804 ERROR 
> [org.infinispan.interceptors.InvocationContextInterceptor] (default 
> task-54) ISPN000136: Error executing command RemoveCommand, writing 
> keys [f9bde276-dd03-41c9-995b-b1aaf64c1489]: 
> org.infinispan.remoting.transport.jgroups.SuspectException: Cache not 
> running on node kc3
>
> at 
> org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
>
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)
>
> at 
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>
> at 
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>
> at 
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>
> at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>
> at 
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>
> at 
> org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
>
> at org.jgroups.blocks.Request.checkCompletion(Request.java:169)
>
> at org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261)
>
> at 
> org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331)
>
> at 
> org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242)
>
> at 
> org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
>
> at org.jgroups.JChannel.up(JChannel.java:738)
>
> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123)
>
> at org.jgroups.stack.Protocol.up(Protocol.java:374)
>
> at org.jgroups.protocols.FORK.up(FORK.java:118)
>
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
>
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
>
> at org.jgroups.protocols.ENCRYPT.up(ENCRYPT.java:454)
>
> at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735)
>
> at 
> org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:140)
>
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)
>
> at org.jgroups.stack.Protocol.up(Protocol.java:412)
>
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
>
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
>
> at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)
>
> at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)
>
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)
>
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
>
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
>
> at org.jgroups.protocols.FD.up(FD.java:260)
>
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
>
> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
>
> at org.jgroups.protocols.Discovery.up(Discovery.java:295)
>
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
>
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
>
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> The key that it tries to write is the user-id. After this, the browser 
> receives an Internal Server Error page, which looks like this in html:
>
> <html>
>
> <head>
>
> <title>
>
>       Error
>
> </title>
>
> </head>
>
> <body>
>
>     Internal Server Error
>
> </body>
>
> </html>
>
> I have configured my infinispan cache settings as following (the rest 
> are default):
>
> <distributed-cache name="sessions" mode="SYNC" owners="5"/>
>
>     <distributed-cache name="offlineSessions" mode="SYNC" owners="1"/>
>
>     <distributed-cache name="loginFailures" mode="SYNC" owners="1"/>
>
> I have tried many things (such as playing with owner amounts or 
> instance amounts etc). It does not seem to fix this exception. I am 
> well aware that this seems more Infinispan issue than Keycloak, but I 
> believe that Keycloak at least should respond the end user a better 
> error message (perhaps a login again page) rather than an Internal 
> Server Error page. Could you please handle this exception?
>
> Kind Regards,
> Sarp Kaya
>
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160801/b734e4f7/attachment.html 