[keycloak-user] Handling SuspectExceptions in Keycloak

Mon Aug 1 15:35:53 EDT 2016

Btv. another possibility to fix is, to doublecheck if you're not 
updating user (or other object) on every login. This may cause that user 
is always invalidated during each login. The result is the 
OutdatedTopologyException, but also the bad performance.

Marek

On 01/08/16 21:32, Marek Posolda wrote:
> See KC issue [1] and related infinispan issue [2] .
>
> The workaround is to add the StateTransferInterceptor to the proper 
> place in chain to "realms" and "users" caches. See how I did it 
> programatically. I think that based on that, you should be able to add 
> it to infinispan subsystem as well.
>
> [1] https://issues.jboss.org/browse/KEYCLOAK-3306
> [2] https://issues.jboss.org/browse/ISPN-6857
>
> Marek
>
> On 28/07/16 11:53, Sarp Kaya wrote:
>>
>> Hello,
>>
>> There is already an existing bug report for Infinispan here:
>>
>> https://issues.jboss.org/browse/ISPN-6721
>>
>> Currently for Keycloak, if this exception is thrown then it sends an 
>> Internal Server Error page to the browser. Essentially what would be 
>> really good is that it sends the user back to the login page instead 
>> of displaying Internal Server Error.
>>
>> This happens when I am consistently sending login and logout (around 
>> 40 req/s) requests to two Keycloak instances (let’s call them kc1 and 
>> kc2), then one new keycloak instance is started kc3. Kc3 connects to 
>> kc1 and 2 in clustering mode.
>>
>> Now kc1 receives a new request (such as login) and while it is 
>> processing that, kc3 is gracefully shut including the cache with this 
>> log:
>>
>> 2016-07-28 09:15:53,656 INFO  [org.jboss.as.clustering.infinispan] 
>> (ServerService Thread Pool -- 61) WFLYCLINF0003: Stopped sessions 
>> cache from keycloak container
>>
>> Just shortly after that (6 ms) kc1 throws an exception like this:
>>
>> 2016-07-28 09:15:53,662 ERROR [io.undertow.request] (default task-48) 
>> UT005023: Exception handling request to 
>> /auth/realms/{realm}/login-actions/authenticate: 
>> org.jboss.resteasy.spi.UnhandledException: 
>> org.infinispan.statetransfer.OutdatedTopologyException: Cache 
>> topology changed while the command was executing: expected 175, got 176
>>
>> at 
>> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:247)
>>
>> at 
>> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
>>
>> at 
>> org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:471)
>>
>> at 
>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:415)
>>
>> then shortly after(150 ms) kc1 wants to talk to kc3 and fails to do 
>> so with this exception:
>>
>> 2016-07-28 09:15:53,804 ERROR 
>> [org.infinispan.interceptors.InvocationContextInterceptor] (default 
>> task-54) ISPN000136: Error executing command RemoveCommand, writing 
>> keys [f9bde276-dd03-41c9-995b-b1aaf64c1489]: 
>> org.infinispan.remoting.transport.jgroups.SuspectException: Cache not 
>> running on node kc3
>>
>> at 
>> org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
>>
>> at 
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:763)
>>
>> at 
>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
>>
>> at 
>> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>>
>> at 
>> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>>
>> at 
>> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>>
>> at 
>> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>>
>> at 
>> org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
>>
>> at org.jgroups.blocks.Request.checkCompletion(Request.java:169)
>>
>> at org.jgroups.blocks.GroupRequest.viewChange(GroupRequest.java:261)
>>
>> at 
>> org.jgroups.blocks.RequestCorrelator.receiveView(RequestCorrelator.java:331)
>>
>> at 
>> org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:242)
>>
>> at 
>> org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
>>
>> at org.jgroups.JChannel.up(JChannel.java:738)
>>
>> at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:123)
>>
>> at org.jgroups.stack.Protocol.up(Protocol.java:374)
>>
>> at org.jgroups.protocols.FORK.up(FORK.java:118)
>>
>> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
>>
>> at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
>>
>> at org.jgroups.protocols.ENCRYPT.up(ENCRYPT.java:454)
>>
>> at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:735)
>>
>> at 
>> org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:140)
>>
>> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:922)
>>
>> at org.jgroups.stack.Protocol.up(Protocol.java:412)
>>
>> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
>>
>> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
>>
>> at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:982)
>>
>> at org.jgroups.protocols.pbcast.NAKACK2.removeAndPassUp(NAKACK2.java:912)
>>
>> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:846)
>>
>> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
>>
>> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
>>
>> at org.jgroups.protocols.FD.up(FD.java:260)
>>
>> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
>>
>> at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
>>
>> at org.jgroups.protocols.Discovery.up(Discovery.java:295)
>>
>> at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
>>
>> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
>>
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> The key that it tries to write is the user-id. After this, the 
>> browser receives an Internal Server Error page, which looks like this 
>> in html:
>>
>> <html>
>>
>> <head>
>>
>> <title>
>>
>> Error
>>
>> </title>
>>
>> </head>
>>
>> <body>
>>
>> Internal Server Error
>>
>> </body>
>>
>> </html>
>>
>> I have configured my infinispan cache settings as following (the rest 
>> are default):
>>
>> <distributed-cache name="sessions" mode="SYNC" owners="5"/>
>>
>>     <distributed-cache name="offlineSessions" mode="SYNC" owners="1"/>
>>
>>     <distributed-cache name="loginFailures" mode="SYNC" owners="1"/>
>>
>> I have tried many things (such as playing with owner amounts or 
>> instance amounts etc). It does not seem to fix this exception. I am 
>> well aware that this seems more Infinispan issue than Keycloak, but I 
>> believe that Keycloak at least should respond the end user a better 
>> error message (perhaps a login again page) rather than an Internal 
>> Server Error page. Could you please handle this exception?
>>
>> Kind Regards,
>> Sarp Kaya
>>
>>
>>
>> _______________________________________________
>> keycloak-user mailing list
>> keycloak-user at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-user
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160801/556939cc/attachment-0001.html 