[keycloak-user] cache replication problems?

[Marek Posolda]

What is your Keycloak version? If you used 2.X and you will migrate to 
latest 3.4.3, there is some chance that issues might be solved as we did 
some performance improvements.

Overally, it depends on number of sessions, network connection between 
cluster servers etc. If network can't be tweaked, then maybe it's 
possible to increase replication timeout? See infinispan and Wildfly 
Infinispan Subsystem docs for how to do it.

Marek

On 06/02/18 12:09, Angel Abella wrote:
> Hello,
>
> We have a 2 server standalone-ha installation. When the number of sessions
> alive increases we get this errors:
>
>
> 2018-02-06 11:42:07,161 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (default
> task-22) ISPN000136: Error executing command PutKeyValueCommand, writing
> keys [f75b436f-d316-4442-8d9b-c7313647c5b8]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> sson2
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
>          at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>          at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>          at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>          at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 2018-02-06 11:42:07,162 ERROR
> [org.keycloak.services.error.KeycloakErrorHandler] (default task-22)
> Uncaught server error: org.infinispan.util.concurrent.TimeoutException:
> Replication timeout for sson2
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
>          at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>          at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>          at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>          at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 2018-02-06 11:42:07,166 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (default
> task-17) ISPN000136: Error executing command RemoveCommand, writing keys
> [0d8d4c5c-7971-46dd-b414-cb5f16862085]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> sson2
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
>          at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>          at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>          at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>          at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 2018-02-06 11:42:07,171 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (default
> task-11) ISPN000136: Error executing command PutKeyValueCommand, writing
> keys [dfd69644-e241-465c-8a92-ef84e76caf62]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> sson2
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
>          at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>          at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>          at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>          at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 2018-02-06 11:42:07,173 ERROR
> [org.keycloak.services.error.KeycloakErrorHandler] (default task-11)
> Uncaught server error: org.infinispan.util.concurrent.TimeoutException:
> Replication timeout for sson2
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
>          at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>          at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>          at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>          at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
> 2018-02-06 11:42:07,205 ERROR
> [org.keycloak.services.error.KeycloakErrorHandler] (default task-17)
> Uncaught server error: org.infinispan.util.concurrent.TimeoutException:
> Replication timeout for sson2
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
>          at
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
>          at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
>          at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
>          at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>          at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
>          at
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>          at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>          at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>          at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>          at java.lang.Thread.run(Thread.java:745)
>
>
>
> Any idea of what's going on?
>