Re: [keycloak-user] Logout 100k users produces an Exception

It seems that bulk removal of userSessions is not very effective. It loads all the sessions locally from the remote nodes and then remove them one by one https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/mai... . Hopefully infinispan has a way that you will trigger task for "visit" sessions, which should be removed. Then you will directly remove the sessions on the node, where they live, in single step. Maybe there are more such bulk operations, which can be improved in similar way. Could you please create JIRA for it? Thanks, Marek On 11/07/16 01:31, Fabricio Milone wrote: Hi all, I have this issue with my current configuration on Keycloak 2.0.0 where after creating more than 100k sessions (or much less than that, but I don't know the exact number), when I try to log out all of them from the admin console I got an UI error after some seconds and the exception at the end of my email. The process does not finish as expected and the sessions are still there. I have 12 nodes running in the network with all my users going through a federator and everything is working as expected so far in terms of functionality. Is there any server side configuration I can tune to avoid this? Thanks in advance! Regads, Fab 2016-07-11 08:15:34,334 ERROR [org.keycloak.services] (Timer-2) >> KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions: >> org.infinispan.util.concurrent.TimeoutException: Replication timeout for >> identityapp06l3 > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599) > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown >> Source) > > at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source) > > at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) > > at java.util.concurrent.CompletableFuture.complete(Unknown Source) > > at >> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) > > at >> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown >> Source) > > at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > >> 2016-07-11 08:28:52,374 WARN >> [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] >> (remote-thread--p7-t1116) ISPN000135: Could not lock key >> 43bcf8ef-c40b-48de-9f20-a118843cfc30 in order to invalidate from L1 at node >> identityapp01l3, skipping.... > > 2016-07-11 08:30:34,385 ERROR [org.keycloak.services] (Timer-2) >> KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions: >> org.infinispan.util.concurrent.TimeoutException: Replication timeout for >> identityapp06l3 > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599) > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown >> Source) > > at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source) > > at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) > > at java.util.concurrent.CompletableFuture.complete(Unknown Source) > > at >> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) > > at >> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown >> Source) > > at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> Source) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > >> 2016-07-11 08:35:41,792 ERROR [io.undertow.request] (default task-60) >> UT005023: Exception handling request to >> /auth/admin/realms/electricsheep/logout-all: >> org.jboss.resteasy.spi.UnhandledException: >> org.infinispan.util.concurrent.TimeoutException: Replication timeout for >> identityapp06l3 > > at >> org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76) > > at >> org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212) > > at >> org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168) > > at >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411) > > at >> org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202) > > at >> org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221) > > at >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) > > at >> org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > > at >> io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) > > at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) > > at >> org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:107) > > at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) > > at >> io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) > > at >> io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) > > at >> io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) > > at >> io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) > > at >> org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) > > at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at >> io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) > > at >> io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) > > at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at >> io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) > > at >> io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) > > at >> io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) > > at >> io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) > > at >> io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) > > at >> io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) > > at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at >> org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) > > at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at >> io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) > > at >> io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) > > at >> io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) > > at >> io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) > > at >> io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) > > at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) > > at >> io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > Caused by: org.infinispan.util.concurrent.TimeoutException: Replication >> timeout for identityapp06l3 > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765) > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599) > > at >> org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown >> Source) > > at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source) > > at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) > > at java.util.concurrent.CompletableFuture.complete(Unknown Source) > > at >> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) > > at >> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) > > at java.util.concurrent.FutureTask.run(Unknown Source) > > at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown >> Source) > > at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown >> Source) > > ... 3 more > > -- *Fabricio Milone* Developer * Shine Consulting * 30/600 Bourke Street Melbourne VIC 3000 T: 03 8488 9939 M: 04 3200 4006 www.shinetech.com *a* passion for excellence _______________________________________________ keycloak-user mailing listkeycloak-user@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/keycloak-user