It seems that bulk removal of userSessions is not very effective. It loads all the sessions locally from the remote nodes and then remove them one by one https://github.com/keycloak/keycloak/blob/master/model/infinispan/src/main/java/org/keycloak/models/sessions/infinispan/InfinispanUserSessionProvider.java#L286-290 . Hopefully infinispan has a way that you will trigger task for "visit" sessions, which should be removed. Then you will directly remove the sessions on the node, where they live, in single step.

Maybe there are more such bulk operations, which can be improved in similar way. Could you please create JIRA for it?

Thanks,
Marek

On 11/07/16 01:31, Fabricio Milone wrote:

Hi all,

I have this issue with my current configuration on Keycloak 2.0.0 where after creating more than 100k sessions (or much less than that, but I don't know the exact number), when I try to log out all of them from the admin console I got an UI error after some seconds and the exception at the end of my email. The process does not finish as expected and the sessions are still there.

I have 12 nodes running in the network with all my users going through a federator and everything is working as expected so far in terms of functionality.

Is there any server side configuration I can tune to avoid this?

Thanks in advance!

Regads,

Fab

2016-07-11 08:15:34,334 ERROR [org.keycloak.services] (Timer-2) KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions: org.infinispan.util.concurrent.TimeoutException: Replication timeout for identityapp06l3

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown Source)

at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)

at java.util.concurrent.CompletableFuture.complete(Unknown Source)

at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)

at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2016-07-11 08:28:52,374 WARN [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] (remote-thread--p7-t1116) ISPN000135: Could not lock key 43bcf8ef-c40b-48de-9f20-a118843cfc30 in order to invalidate from L1 at node identityapp01l3, skipping....

2016-07-11 08:30:34,385 ERROR [org.keycloak.services] (Timer-2) KC-SERVICES0089: Failed to run scheduled task ClearExpiredUserSessions: org.infinispan.util.concurrent.TimeoutException: Replication timeout for identityapp06l3

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown Source)

at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)

at java.util.concurrent.CompletableFuture.complete(Unknown Source)

at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)

at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

2016-07-11 08:35:41,792 ERROR [io.undertow.request] (default task-60) UT005023: Exception handling request to /auth/admin/realms/electricsheep/logout-all: org.jboss.resteasy.spi.UnhandledException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for identityapp06l3

at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)

at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)

at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:168)

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:411)

at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)

at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)

at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)

at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)

at org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:107)

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for identityapp06l3

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$72(JGroupsTransport.java:599)

at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$106/1829642022.apply(Unknown Source)

at java.util.concurrent.CompletableFuture$ThenApply.run(Unknown Source)

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source)

at java.util.concurrent.CompletableFuture.complete(Unknown Source)

at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)

at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

... 3 more

--

Fabricio Milone

Developer

Shine Consulting
30/600 Bourke Street

Melbourne VIC 3000

T: 03 8488 9939

M: 04 3200 4006

www.shinetech.com a passion for excellence
_______________________________________________
keycloak-user mailing list
keycloak-user@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user