Re: [keycloak-user] Keycloak standalone-ha cluster replication Timeout issue

Monday, 8 October 2018

Replication Timeout happens when a node can not replicate data to other
cluster members withing a certain amount of time. There might be many
different causes including:
- long GC pause on one of the nodes
- network delays
- firewall problems
- problems with physical network interfaces

I highly encourage you to try to look up some solution in the Internet.
When it comes to the replication timeouts, here's my personal list of
things to do:
- Turn the GC logs on, so that you can correlate GC pauses with the
exceptions
- Increase timeout in the XML configuration
- Observe JGroups "view" messages from the server logs. Make sure it's
stable. If not, tune FD_* protocol settings

Good luck and cheers,
Sebastian

On Fri, Oct 5, 2018 at 9:38 AM Upananda Singha <
upananda.singha(a)motorolasolutions.com&gt; wrote:

...
 Hi All,

 I have been running Keycloak (4.0.0 Final) in standalone-ha mode with 2
 instances in a cluster.

 I have been running the cluster of 2 nodes for more 3/4 months now. But
 suddenly
 I am facing a strange issue now with cache replication between the 2 nodes
 in the same cluster itself.
 Anybody faced this kind of issue or anyone can guide me why this is
 happening and how to resolve the issue.

 2018-10-04 17:51:52.793 INFO  [org.jboss.as.server] (ServerService Thread
 Pool -- 49) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name :
 "keycloak-server.war")
 2018-10-04 17:51:52.932 INFO  [org.jboss.as.server] (Controller Boot
 Thread) WFLYSRV0212: Resuming server
 2018-10-04 17:51:52.934 INFO  [org.jboss.as] (Controller Boot Thread)
 WFLYSRV0060: Http management interface listening on
 http://192.168.190.77:10110/management
 2018-10-04 17:51:52.935 INFO  [org.jboss.as] (Controller Boot Thread)
 WFLYSRV0051: Admin console listening on http://192.168.190.77:10110
 2018-10-04 17:51:52.935 INFO  [org.jboss.as] (Controller Boot Thread)
 WFLYSRV0025: Keycloak 4.0.0.Final (WildFly Core 3.0.8.Final) started in
 21492ms - Started 645 of 996 services (707 services are lazy, passive or
 on-demand)
 2018-10-04 18:07:02.670 ERROR
 [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
 ISPN000136: Error executing command PutKeyValueCommand, writing keys
 [task::ClearExpiredEvents]:
 org.infinispan.util.concurrent.TimeoutException: Replication timeout for
 keycloak-216
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at
 java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
 [rt.jar:1.8.0_162]
 at

org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at

org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 [rt.jar:1.8.0_162]
 at

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 [rt.jar:1.8.0_162]
 at

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 [rt.jar:1.8.0_162]
 at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]

 2018-10-04 18:07:12.694 ERROR
 [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
 ISPN000136: Error executing command PutKeyValueCommand, writing keys
 [task::ClearExpiredEvents]:
 org.infinispan.util.concurrent.TimeoutException: Replication timeout for
 keycloak-216
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at
 java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
 [rt.jar:1.8.0_162]
 at

org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at

org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 [rt.jar:1.8.0_162]
 at

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 [rt.jar:1.8.0_162]
 at

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 [rt.jar:1.8.0_162]
 at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]

 2018-10-04 18:07:22.712 ERROR
 [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
 ISPN000136: Error executing command PutKeyValueCommand, writing keys
 [task::ClearExpiredEvents]:
 org.infinispan.util.concurrent.TimeoutException: Replication timeout for
 keycloak-216
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at
 java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
 [rt.jar:1.8.0_162]
 at

org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at

org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 [rt.jar:1.8.0_162]
 at

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 [rt.jar:1.8.0_162]
 at

java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 [rt.jar:1.8.0_162]
 at

 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 [rt.jar:1.8.0_162]
 at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]

 2018-10-04 18:07:32.723 ERROR
 [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
 ISPN000136: Error executing command PutKeyValueCommand, writing keys
 [task::ClearExpiredEvents]:
 org.infinispan.util.concurrent.TimeoutException: Replication timeout for
 keycloak-216
 at

org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
 [infinispan-core-8.2.8.Final.jar:8.2.8.Final]

 My cache configurations looks like:

 standalone-ha.xml
 ------------------------
 <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
             <cache-container name="keycloak"
 jndi-name="infinispan/Keycloak">
                 <transport lock-timeout="60000"/>
                 <local-cache name="realms">
                     <eviction max-entries="10000"
strategy="LRU"/>
                 </local-cache>
                 <local-cache name="users">
                     <eviction max-entries="10000"
strategy="LRU"/>
                 </local-cache>
                 <distributed-cache name="sessions" mode="SYNC"
owners="2"/>
                 <distributed-cache name="authenticationSessions"
 mode="SYNC" owners="2"/>
                 <distributed-cache name="offlineSessions"
mode="SYNC"
 owners="2">
                     <eviction max-entries="10000"
strategy="LRU"/>
                 </distributed-cache>
                 <distributed-cache name="clientSessions"
mode="SYNC"
 owners="2">
                     <eviction max-entries="1000000"
strategy="LRU"/>
                 </distributed-cache>
                 <distributed-cache name="offlineClientSessions"
mode="SYNC"
 owners="2">
                     <eviction max-entries="10000"
strategy="LRU"/>
                 </distributed-cache>
                 <distributed-cache name="loginFailures"
mode="SYNC"
 owners="2">
                     <eviction max-entries="10000"
strategy="LRU"/>
                 </distributed-cache>
                 <local-cache name="authorization">
                     <eviction max-entries="10000"
strategy="LRU"/>
                 </local-cache>
                 <replicated-cache name="work" mode="SYNC"/>
                 <local-cache name="keys">
                     <eviction max-entries="1000"
strategy="LRU"/>
                     <expiration max-idle="3600000"/>
                 </local-cache>
                 <distributed-cache name="actionTokens"
mode="SYNC"
 owners="2">
                     <eviction max-entries="-1"
strategy="NONE"/>
                     <expiration max-idle="-1"
interval="300000"/>
                 </distributed-cache>
             </cache-container>

        <cache-container name="server" aliases="singleton cluster"
 default-cache="default" module="org.wildfly.clustering.server">
                 <transport lock-timeout="60000"/>
                 <replicated-cache name="default">
                     <transaction mode="BATCH"/>
                 </replicated-cache>
             </cache-container>
             <cache-container name="web" default-cache="dist"
 module="org.wildfly.clustering.web.infinispan">
                 <transport lock-timeout="60000"/>
                 <distributed-cache name="dist">
                     <locking isolation="REPEATABLE_READ"/>
                     <transaction mode="BATCH"/>
                     <file-store/>
                 </distributed-cache>
             </cache-container>
             <cache-container name="ejb" aliases="sfsb"
default-cache="dist"
 module="org.wildfly.clustering.ejb.infinispan">
                 <transport lock-timeout="60000"/>
                 <distributed-cache name="dist">
                     <locking isolation="REPEATABLE_READ"/>
                     <transaction mode="BATCH"/>
                     <file-store/>
                 </distributed-cache>
             </cache-container>

            <cache-container name="hibernate"
default-cache="local-query"
 module="org.hibernate.infinispan">
                 <transport lock-timeout="60000"/>
                 <local-cache name="local-query">
                     <eviction strategy="LRU"
max-entries="10000"/>
                     <expiration max-idle="100000"/>
                 </local-cache>
                 <invalidation-cache name="entity">
                     <transaction mode="NON_XA"/>
                     <eviction strategy="LRU"
max-entries="10000"/>
                     <expiration max-idle="100000"/>
                 </invalidation-cache>
                 <replicated-cache name="timestamps"
mode="ASYNC"/>
             </cache-container>
         </subsystem>
 ------------------------------------------

 Thanks & Regds,

 Upananda Singha
 _______________________________________________
 keycloak-user mailing list
 keycloak-user(a)lists.jboss.org
 https://lists.jboss.org/mailman/listinfo/keycloak-user

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

Re: [keycloak-user] Keycloak standalone-ha cluster replication Timeout issue