Re: [keycloak-user] cache replication problems?

I'd start checking your resources - are the boxes under heavy load? What about the Infinispan caches? We saw some increased timeouts for a spell as a function of load and had to do some JVM + Cache tuning to make sure they were sized properly for our workload. Josh Cain Senior Software Applications Engineer, RHCE Red Hat North America jcain(a)redhat.com IRC: jcain On 02/06/2018 05:09 AM, Angel Abella wrote: > Hello, > > We have a 2 server standalone-ha installation. When the number of sessions > alive increases we get this errors: > > > 2018-02-06 11:42:07,161 ERROR > [org.infinispan.interceptors.InvocationContextInterceptor] (default > task-22) ISPN000136: Error executing command PutKeyValueCommand, writing > keys [f75b436f-d316-4442-8d9b-c7313647c5b8]: > org.infinispan.util.concurrent.TimeoutException: Replication timeout for > sson2 > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp( JGroupsTransport.java:827) > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$ invokeRemotelyAsync$0(JGroupsTransport.java:628) > at > java.util.concurrent.CompletableFuture.uniApply( CompletableFuture.java:602) > at > java.util.concurrent.CompletableFuture$UniApply. tryFire(CompletableFuture.java:577) > at > java.util.concurrent.CompletableFuture.postComplete( CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete( CompletableFuture.java:1962) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:46) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:17) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 2018-02-06 11:42:07,162 ERROR > [org.keycloak.services.error.KeycloakErrorHandler] (default task-22) > Uncaught server error: org.infinispan.util.concurrent.TimeoutException: > Replication timeout for sson2 > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp( JGroupsTransport.java:827) > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$ invokeRemotelyAsync$0(JGroupsTransport.java:628) > at > java.util.concurrent.CompletableFuture.uniApply( CompletableFuture.java:602) > at > java.util.concurrent.CompletableFuture$UniApply. tryFire(CompletableFuture.java:577) > at > java.util.concurrent.CompletableFuture.postComplete( CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete( CompletableFuture.java:1962) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:46) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:17) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 2018-02-06 11:42:07,166 ERROR > [org.infinispan.interceptors.InvocationContextInterceptor] (default > task-17) ISPN000136: Error executing command RemoveCommand, writing keys > [0d8d4c5c-7971-46dd-b414-cb5f16862085]: > org.infinispan.util.concurrent.TimeoutException: Replication timeout for > sson2 > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp( JGroupsTransport.java:827) > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$ invokeRemotelyAsync$0(JGroupsTransport.java:628) > at > java.util.concurrent.CompletableFuture.uniApply( CompletableFuture.java:602) > at > java.util.concurrent.CompletableFuture$UniApply. tryFire(CompletableFuture.java:577) > at > java.util.concurrent.CompletableFuture.postComplete( CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete( CompletableFuture.java:1962) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:46) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:17) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 2018-02-06 11:42:07,171 ERROR > [org.infinispan.interceptors.InvocationContextInterceptor] (default > task-11) ISPN000136: Error executing command PutKeyValueCommand, writing > keys [dfd69644-e241-465c-8a92-ef84e76caf62]: > org.infinispan.util.concurrent.TimeoutException: Replication timeout for > sson2 > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp( JGroupsTransport.java:827) > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$ invokeRemotelyAsync$0(JGroupsTransport.java:628) > at > java.util.concurrent.CompletableFuture.uniApply( CompletableFuture.java:602) > at > java.util.concurrent.CompletableFuture$UniApply. tryFire(CompletableFuture.java:577) > at > java.util.concurrent.CompletableFuture.postComplete( CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete( CompletableFuture.java:1962) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:46) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:17) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 2018-02-06 11:42:07,173 ERROR > [org.keycloak.services.error.KeycloakErrorHandler] (default task-11) > Uncaught server error: org.infinispan.util.concurrent.TimeoutException: > Replication timeout for sson2 > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp( JGroupsTransport.java:827) > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$ invokeRemotelyAsync$0(JGroupsTransport.java:628) > at > java.util.concurrent.CompletableFuture.uniApply( CompletableFuture.java:602) > at > java.util.concurrent.CompletableFuture$UniApply. tryFire(CompletableFuture.java:577) > at > java.util.concurrent.CompletableFuture.postComplete( CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete( CompletableFuture.java:1962) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:46) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:17) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > 2018-02-06 11:42:07,205 ERROR > [org.keycloak.services.error.KeycloakErrorHandler] (default task-17) > Uncaught server error: org.infinispan.util.concurrent.TimeoutException: > Replication timeout for sson2 > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp( JGroupsTransport.java:827) > at > org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$ invokeRemotelyAsync$0(JGroupsTransport.java:628) > at > java.util.concurrent.CompletableFuture.uniApply( CompletableFuture.java:602) > at > java.util.concurrent.CompletableFuture$UniApply. tryFire(CompletableFuture.java:577) > at > java.util.concurrent.CompletableFuture.postComplete( CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.complete( CompletableFuture.java:1962) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:46) > at > org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call( SingleResponseFuture.java:17) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > > Any idea of what's going on? > _______________________________________________ keycloak-user mailing list keycloak-user(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-user