[keycloak-user] Keycloak standalone-ha cluster replication Timeout issue
Upananda Singha
upananda.singha at motorolasolutions.com
Fri Oct 5 03:36:12 EDT 2018
Hi All,
I have been running Keycloak (4.0.0 Final) in standalone-ha mode with 2
instances in a cluster.
I have been running the cluster of 2 nodes for more 3/4 months now. But
suddenly
I am facing a strange issue now with cache replication between the 2 nodes
in the same cluster itself.
Anybody faced this kind of issue or anyone can guide me why this is
happening and how to resolve the issue.
2018-10-04 17:51:52.793 INFO [org.jboss.as.server] (ServerService Thread
Pool -- 49) WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name :
"keycloak-server.war")
2018-10-04 17:51:52.932 INFO [org.jboss.as.server] (Controller Boot
Thread) WFLYSRV0212: Resuming server
2018-10-04 17:51:52.934 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0060: Http management interface listening on
http://192.168.190.77:10110/management
2018-10-04 17:51:52.935 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0051: Admin console listening on http://192.168.190.77:10110
2018-10-04 17:51:52.935 INFO [org.jboss.as] (Controller Boot Thread)
WFLYSRV0025: Keycloak 4.0.0.Final (WildFly Core 3.0.8.Final) started in
21492ms - Started 645 of 996 services (707 services are lazy, passive or
on-demand)
2018-10-04 18:07:02.670 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[rt.jar:1.8.0_162]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
2018-10-04 18:07:12.694 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[rt.jar:1.8.0_162]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
2018-10-04 18:07:22.712 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:628)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[rt.jar:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[rt.jar:1.8.0_162]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at
org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
2018-10-04 18:07:32.723 ERROR
[org.infinispan.interceptors.InvocationContextInterceptor] (Timer-2)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[task::ClearExpiredEvents]:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for
keycloak-216
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:827)
[infinispan-core-8.2.8.Final.jar:8.2.8.Final]
My cache configurations looks like:
standalone-ha.xml
------------------------
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak">
<transport lock-timeout="60000"/>
<local-cache name="realms">
<eviction max-entries="10000" strategy="LRU"/>
</local-cache>
<local-cache name="users">
<eviction max-entries="10000" strategy="LRU"/>
</local-cache>
<distributed-cache name="sessions" mode="SYNC" owners="2"/>
<distributed-cache name="authenticationSessions"
mode="SYNC" owners="2"/>
<distributed-cache name="offlineSessions" mode="SYNC"
owners="2">
<eviction max-entries="10000" strategy="LRU"/>
</distributed-cache>
<distributed-cache name="clientSessions" mode="SYNC"
owners="2">
<eviction max-entries="1000000" strategy="LRU"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" mode="SYNC"
owners="2">
<eviction max-entries="10000" strategy="LRU"/>
</distributed-cache>
<distributed-cache name="loginFailures" mode="SYNC"
owners="2">
<eviction max-entries="10000" strategy="LRU"/>
</distributed-cache>
<local-cache name="authorization">
<eviction max-entries="10000" strategy="LRU"/>
</local-cache>
<replicated-cache name="work" mode="SYNC"/>
<local-cache name="keys">
<eviction max-entries="1000" strategy="LRU"/>
<expiration max-idle="3600000"/>
</local-cache>
<distributed-cache name="actionTokens" mode="SYNC"
owners="2">
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1" interval="300000"/>
</distributed-cache>
</cache-container>
<cache-container name="server" aliases="singleton cluster"
default-cache="default" module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="web" default-cache="dist"
module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb" default-cache="dist"
module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate" default-cache="local-query"
module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps" mode="ASYNC"/>
</cache-container>
</subsystem>
------------------------------------------
Thanks & Regds,
Upananda Singha
More information about the keycloak-user
mailing list