[keycloak-user] Keycloak standalone-ha cluster replication Timeout issue
Sebastian Laskawiec
slaskawi at redhat.com
Mon Oct 8 07:12:30 EDT 2018
Replication Timeout happens when a node can not replicate data to other
cluster members withing a certain amount of time. There might be many
different causes including:
- long GC pause on one of the nodes
- network delays
- firewall problems
- problems with physical network interfaces
I highly encourage you to try to look up some solution in the Internet.
When it comes to the replication timeouts, here's my personal list of
things to do:
- Turn the GC logs on, so that you can correlate GC pauses with the
exceptions
- Increase timeout in the XML configuration
- Observe JGroups "view" messages from the server logs. Make sure it's
stable. If not, tune FD_* protocol settings
Good luck and cheers,
Sebastian
On Fri, Oct 5, 2018 at 9:38 AM Upananda Singha <
upananda.singha at motorolasolutions.com> wrote:
> Hi All,
>
> I have been running Keycloak (4.0.0 Final) in standalone-ha mode with 2
> instances in a cluster.
>
> I have been running the cluster of 2 nodes for more 3/4 months now. But
> suddenly
> I am facing a strange issue now with cache replication between the 2 nodes
> in the same cluster itself.
> Anybody faced this kind of issue or anyone can guide me why this is
> happening and how to resolve the issue.
>
>
> 2018-10-04 17:51:52.793 INFO [org.jboss.as.server] (ServerService Thread
> Pool -- 49) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name :
> "keycloak-server.war")
> 2018-10-04 17:51:52.932 INFO [org.jboss.as.server] (Controller Boot
> Thread) WFLYSRV0212: Resuming server
> 2018-10-04 17:51:52.934 INFO [org.jboss.as] (Controller Boot Thread)
> WFLYSRV0060: Http management interface listening on
> http://192.168.190.77:10110/management
> 2018-10-04 17:51:52.935 INFO [org.jboss.as] (Controller Boot Thread)
> WFLYSRV0051: Admin console listening on http://192.168.190.77:10110
> 2018-10-04 17:51:52.935 INFO [org.jboss.as] (Controller Boot Thread)
> WFLYSRV0025: Keycloak 4.0.0.Final (WildFly Core 3.0.8.Final) started in
> 21492ms - Started 645 of 996 services (707 services are lazy, passive or
> on-demand)
> 2018-10-04 18:07:02.670 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
> ISPN000136: Error executing command PutKeyValueCommand, writing keys
> [task::ClearExpiredEvents]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> keycloak-216
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [rt.jar:1.8.0_162]
> at
>
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
>
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [rt.jar:1.8.0_162]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
>
> 2018-10-04 18:07:12.694 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
> ISPN000136: Error executing command PutKeyValueCommand, writing keys
> [task::ClearExpiredEvents]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> keycloak-216
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [rt.jar:1.8.0_162]
> at
>
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
>
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [rt.jar:1.8.0_162]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
>
> 2018-10-04 18:07:22.712 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
> ISPN000136: Error executing command PutKeyValueCommand, writing keys
> [task::ClearExpiredEvents]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> keycloak-216
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [rt.jar:1.8.0_162]
> at
>
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at
>
> org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [rt.jar:1.8.0_162]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [rt.jar:1.8.0_162]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
>
> 2018-10-04 18:07:32.723 ERROR
> [org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
> ISPN000136: Error executing command PutKeyValueCommand, writing keys
> [task::ClearExpiredEvents]:
> org.infinispan.util.concurrent.TimeoutException: Replication timeout for
> keycloak-216
> at
>
> org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
> [infinispan-core-8.2.8.Final.jar:8.2.8.Final]
>
>
> My cache configurations looks like:
>
> standalone-ha.xml
> ------------------------
> <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
> <cache-container name="keycloak"
> jndi-name="infinispan/Keycloak">
> <transport lock-timeout="60000"/>
> <local-cache name="realms">
> <eviction max-entries="10000" strategy="LRU"/>
> </local-cache>
> <local-cache name="users">
> <eviction max-entries="10000" strategy="LRU"/>
> </local-cache>
> <distributed-cache name="sessions" mode="SYNC" owners="2"/>
> <distributed-cache name="authenticationSessions"
> mode="SYNC" owners="2"/>
> <distributed-cache name="offlineSessions" mode="SYNC"
> owners="2">
> <eviction max-entries="10000" strategy="LRU"/>
> </distributed-cache>
> <distributed-cache name="clientSessions" mode="SYNC"
> owners="2">
> <eviction max-entries="1000000" strategy="LRU"/>
> </distributed-cache>
> <distributed-cache name="offlineClientSessions" mode="SYNC"
> owners="2">
> <eviction max-entries="10000" strategy="LRU"/>
> </distributed-cache>
> <distributed-cache name="loginFailures" mode="SYNC"
> owners="2">
> <eviction max-entries="10000" strategy="LRU"/>
> </distributed-cache>
> <local-cache name="authorization">
> <eviction max-entries="10000" strategy="LRU"/>
> </local-cache>
> <replicated-cache name="work" mode="SYNC"/>
> <local-cache name="keys">
> <eviction max-entries="1000" strategy="LRU"/>
> <expiration max-idle="3600000"/>
> </local-cache>
> <distributed-cache name="actionTokens" mode="SYNC"
> owners="2">
> <eviction max-entries="-1" strategy="NONE"/>
> <expiration max-idle="-1" interval="300000"/>
> </distributed-cache>
> </cache-container>
>
> <cache-container name="server" aliases="singleton cluster"
> default-cache="default" module="org.wildfly.clustering.server">
> <transport lock-timeout="60000"/>
> <replicated-cache name="default">
> <transaction mode="BATCH"/>
> </replicated-cache>
> </cache-container>
> <cache-container name="web" default-cache="dist"
> module="org.wildfly.clustering.web.infinispan">
> <transport lock-timeout="60000"/>
> <distributed-cache name="dist">
> <locking isolation="REPEATABLE_READ"/>
> <transaction mode="BATCH"/>
> <file-store/>
> </distributed-cache>
> </cache-container>
> <cache-container name="ejb" aliases="sfsb" default-cache="dist"
> module="org.wildfly.clustering.ejb.infinispan">
> <transport lock-timeout="60000"/>
> <distributed-cache name="dist">
> <locking isolation="REPEATABLE_READ"/>
> <transaction mode="BATCH"/>
> <file-store/>
> </distributed-cache>
> </cache-container>
>
> <cache-container name="hibernate" default-cache="local-query"
> module="org.hibernate.infinispan">
> <transport lock-timeout="60000"/>
> <local-cache name="local-query">
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration max-idle="100000"/>
> </local-cache>
> <invalidation-cache name="entity">
> <transaction mode="NON_XA"/>
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration max-idle="100000"/>
> </invalidation-cache>
> <replicated-cache name="timestamps" mode="ASYNC"/>
> </cache-container>
> </subsystem>
> ------------------------------------------
>
>
>
> Thanks & Regds,
>
> Upananda Singha
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
More information about the keycloak-user
mailing list