[keycloak-user] Logout 100k users produces an Exception

[Fabricio Milone]

Thanks Marek,

I was looking into that method yesterday as well, with a huge amount of
sessions the Sessions screen performs very slowly as well, so I will create
a Jira ticket to track this.

Regards,
Fab

On 11 July 2016 at 16:45, Marek Posolda <mposolda at redhat.com> wrote:

> It seems that bulk removal of userSessions is not very effective. It loads
> all the sessions locally from the remote nodes and then remove them one by
> one
> https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/InfinispanUserSessionProvider.java#L286-290
> . Hopefully infinispan has a way that you will trigger task for "visit"
> sessions, which should be removed. Then you will directly remove the
> sessions on the node, where they live, in single step.
>
> Maybe there are more such bulk operations, which can be improved in
> similar way. Could you please create JIRA for it?
>
> Thanks,
> Marek
>
>
> On 11/07/16 01:31, Fabricio Milone wrote:
>
> Hi all,
>
> I have this issue with my current configuration on Keycloak 2.0.0 where
> after creating more than 100k sessions (or much less than that, but I don't
> know the exact number), when I try to log out all of them from the admin
> console I got an UI error after some seconds and the exception at the end
> of my email. The process does not finish as expected and the sessions are
> still there.
>
> I have 12 nodes running in the network with all my users going through a
> federator and everything is working as expected so far in terms of
> functionality.
>
> Is there any server side configuration I can tune to avoid this?
>
> Thanks in advance!
>
> Regads,
> Fab
>
> 2016-07-11 08:15:34,334 ERROR [org.keycloak.services] (Timer-2)
>>> KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions:
>>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
>>> identityapp06l3
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown
>>> Source)
>>
>> at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)
>>
>> at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
>>
>> at java.util.concurrent.CompletableFuture.complete(Unknown Source)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>>
>> at java.util.concurrent.FutureTask.run(Unknown Source)
>>
>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>>> Source)
>>
>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> Source)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>
>> at java.lang.Thread.run(Unknown Source)
>>
>>
>>> 2016-07-11 08:28:52,374 WARN
>>>  [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor]
>>> (remote-thread--p7-t1116) ISPN000135: Could not lock key
>>> 43bcf8ef-c40b-48de-9f20-a118843cfc30 in order to invalidate from L1 at node
>>> identityapp01l3, skipping....
>>
>> 2016-07-11 08:30:34,385 ERROR [org.keycloak.services] (Timer-2)
>>> KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions:
>>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
>>> identityapp06l3
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown
>>> Source)
>>
>> at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)
>>
>> at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
>>
>> at java.util.concurrent.CompletableFuture.complete(Unknown Source)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>>
>> at java.util.concurrent.FutureTask.run(Unknown Source)
>>
>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>>> Source)
>>
>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> Source)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>
>> at java.lang.Thread.run(Unknown Source)
>>
>>
>>> 2016-07-11 08:35:41,792 ERROR [io.undertow.request] (default task-60)
>>> UT005023: Exception handling request to
>>> /auth/admin/realms/electricsheep/logout-all:
>>> org.jboss.resteasy.spi.UnhandledException:
>>> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
>>> identityapp06l3
>>
>> at
>>> org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
>>
>> at
>>> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
>>
>> at
>>> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)
>>
>> at
>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411)
>>
>> at
>>> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
>>
>> at
>>> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
>>
>> at
>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>>
>> at
>>> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>>
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>>
>> at
>>> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
>>
>> at
>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
>>
>> at
>>> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:107)
>>
>> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
>>
>> at
>>> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
>>
>> at
>>> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
>>
>> at
>>> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
>>
>> at
>>> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
>>
>> at
>>> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
>>
>> at
>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>
>> at
>>> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
>>
>> at
>>> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
>>
>> at
>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>
>> at
>>> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
>>
>> at
>>> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
>>
>> at
>>> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
>>
>> at
>>> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
>>
>> at
>>> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
>>
>> at
>>> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
>>
>> at
>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>
>> at
>>> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
>>
>> at
>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>
>> at
>>> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
>>
>> at
>>> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
>>
>> at
>>> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
>>
>> at
>>> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
>>
>> at
>>> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
>>
>> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
>>
>> at
>>> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
>>
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>>
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>>
>> at java.lang.Thread.run(Unknown Source)
>>
>> Caused by: org.infinispan.util.concurrent.TimeoutException: Replication
>>> timeout for identityapp06l3
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown
>>> Source)
>>
>> at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)
>>
>> at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
>>
>> at java.util.concurrent.CompletableFuture.complete(Unknown Source)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>>
>> at
>>> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>>
>> at java.util.concurrent.FutureTask.run(Unknown Source)
>>
>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>>> Source)
>>
>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>> Source)
>>
>> ... 3 more
>>
>>
> --
> *Fabricio Milone*
> Developer
>
> * Shine Consulting *
>
> 30/600 Bourke Street
>
> Melbourne VIC 3000
>
> T: 03 8488 9939
>
> M: 04 3200 4006
>
>
> www.shinetech.com  *a* passion for excellence
>
>
> _______________________________________________
> keycloak-user mailing listkeycloak-user at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/keycloak-user
>
>
>


-- 
*Fabricio Milone*
Developer

*Shine Consulting *

30/600 Bourke Street

Melbourne VIC 3000

T: 03 8488 9939

M: 04 3200 4006


www.shinetech.com  *a* passion for excellence
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/keycloak-user/attachments/20160712/b094c785/attachment-0001.html