Replication Timeout happens when a node can not replicate data to other
cluster members withing a certain amount of time. There might be many
different causes including:
- long GC pause on one of the nodes
- network delays
- firewall problems
- problems with physical network interfaces
I highly encourage you to try to look up some solution in the Internet.
When it comes to the replication timeouts, here's my personal list of
things to do:
- Turn the GC logs on, so that you can correlate GC pauses with the
exceptions
- Increase timeout in the XML configuration
- Observe JGroups "view" messages from the server logs. Make sure it's
stable. If not, tune FD_* protocol settings
Good luck and cheers,
Sebastian
On Fri, Oct 5, 2018 at 9:38 AM Upananda Singha <
upananda.singha(a)motorolasolutions.com> wrote:
Hi All,
I have been running Keycloak (4.0.0 Final) in standalone-ha mode with 2
instances in a cluster.
I have been running the cluster of 2 nodes for more 3/4 months now. But
suddenly
I am facing a strange issue now with cache replication between the 2 nodes
in the same cluster itself.
Anybody faced this kind of issue or anyone can guide me why this is
happening and how to resolve the issue.
2018-10-04 17:51:52.793 INFO [org.jboss.as.server] (ServerService Thread
Pool -- 49) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name :
"keycloak-server.war")
2018-10-04 17:51:52.932 INFO [org.jboss.as.server] (Controller Boot
Thread) WFLYSRV0212: Resuming server
2018-10-04 17:51:52.934 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0060: Http management interface listening on
http://192.168.190.77:10110/management
2018-10-04 17:51:52.935 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0051: Admin console listening on
http://192.168.190.77:10110
2018-10-04 17:51:52.935 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0025: Keycloak 4.0.0.Final (WildFly Core 3.0.8.Final) started in
21492ms - Started 645 of 996 services (707 services are lazy, passive or
on-demand)
2018-10-04 18:07:02.670 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[rt.jar:1.8.0_162]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
2018-10-04 18:07:12.694 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[rt.jar:1.8.0_162]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
2018-10-04 18:07:22.712 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[rt.jar:1.8.0_162]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
2018-10-04 18:07:32.723 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
My cache configurations looks like:
standalone-ha.xml
------------------------
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak">
<transport lock-timeout="60000"/>
<local-cache name="realms">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<local-cache name="users">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<distributed-cache name="sessions" mode="SYNC"
owners="2"/>
<distributed-cache name="authenticationSessions"
mode="SYNC" owners="2"/>
<distributed-cache name="offlineSessions"
mode="SYNC"
owners="2">
<eviction max-entries="10000"
strategy="LRU"/>
</distributed-cache>
<distributed-cache name="clientSessions"
mode="SYNC"
owners="2">
<eviction max-entries="1000000"
strategy="LRU"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions"
mode="SYNC"
owners="2">
<eviction max-entries="10000"
strategy="LRU"/>
</distributed-cache>
<distributed-cache name="loginFailures"
mode="SYNC"
owners="2">
<eviction max-entries="10000"
strategy="LRU"/>
</distributed-cache>
<local-cache name="authorization">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<replicated-cache name="work" mode="SYNC"/>
<local-cache name="keys">
<eviction max-entries="1000"
strategy="LRU"/>
<expiration max-idle="3600000"/>
</local-cache>
<distributed-cache name="actionTokens"
mode="SYNC"
owners="2">
<eviction max-entries="-1"
strategy="NONE"/>
<expiration max-idle="-1"
interval="300000"/>
</distributed-cache>
</cache-container>
<cache-container name="server" aliases="singleton cluster"
default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="web" default-cache="dist"
module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb"
default-cache="dist"
module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate"
default-cache="local-query"
module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU"
max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity">
<transaction mode="NON_XA"/>
<eviction strategy="LRU"
max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps"
mode="ASYNC"/>
</cache-container>
</subsystem>
------------------------------------------
Thanks & Regds,
Upananda Singha
_______________________________________________
keycloak-user mailing list
keycloak-user(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-user