12:09 a.m.
We have a 3 node cluster of Keycloak 3.1.0.Final (WildFly Core 2.0.10.Final) running in
Docker AWS ECS.
It all has been running smoothly for many months, through various versions, deployments
and recycles.
Just recently, when one of the containers was terminated, the new container was run by ECS
but could not start Keycloak and join the cluster.
Error is: Failed to start service jboss.infinispan.keycloak.loginFailures:
org.jboss.msc.service.StartException in service jboss.infinispan.keycloak.loginFailures,
Initial state transfer timed out for cache loginFailures
We also see some of the following, which may be related?
2017-06-27 05:54:01,914 DEBUG
[org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler]
(remote-thread--p10-t3) ISPN000311: Received a command from an outdated topology,
returning the exception to caller: org.infinispan.statetransfer.OutdatedTopologyException:
Cache topology changed while the command was executing: expected 1962, got 1963
We enabled various Debug log levels but haven’t yet found anything conclusive.
We tried clearing caches via Infinispan JMX tooling etc, but still could not start the new
node.
The two remaining instances were still clustered OK.
We provisioned additional cluster and it's working as expected, so don’t believe at
this stage it’s a network communication issue.
Found this report for Redhat SSO, seems similar? I can’t see solution.
https://access.redhat.com/solutions/2841711
Any info on possible cause or how to debug/investigate further would be a great help.
Configs and logs below..
Thanks
Dan
Our cache config is:
<subsystem xmlns="urn:jboss:domain:infinispan:4.0">
<cache-container name="keycloak"
jndi-name="infinispan/Keycloak">
<transport lock-timeout="60000"/>
<local-cache name="realms">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<local-cache name="users">
<eviction max-entries="10000"
strategy="LRU"/>
</local-cache>
<distributed-cache name="sessions" mode="SYNC"
owners="5" />
<distributed-cache name="offlineSessions"
mode="SYNC" owners="1" />
<distributed-cache name="loginFailures" mode="SYNC"
owners="1"/>
<distributed-cache name="authorization" mode="SYNC"
owners="1"/>
<replicated-cache name="work" mode="SYNC"/>
<local-cache name="keys">
<eviction max-entries="1000"
strategy="LRU"/>
<expiration max-idle="3600000"/>
</local-cache>
</cache-container>
<cache-container name="server" aliases="singleton
cluster" default-cache="default"
module="org.wildfly.clustering.server">
<transport lock-timeout="60000"/>
<replicated-cache name="default" mode="SYNC">
<transaction mode="BATCH"/>
</replicated-cache>
</cache-container>
<cache-container name="web" default-cache="dist"
module="org.wildfly.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC"
l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="ejb" aliases="sfsb"
default-cache="dist"
module="org.wildfly.clustering.ejb.infinispan">
<transport lock-timeout="60000"/>
<distributed-cache name="dist" mode="ASYNC"
l1-lifespan="0" owners="2">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<file-store/>
</distributed-cache>
</cache-container>
<cache-container name="hibernate"
default-cache="local-query" module="org.hibernate.infinispan">
<transport lock-timeout="60000"/>
<local-cache name="local-query">
<eviction strategy="LRU"
max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<invalidation-cache name="entity" mode="SYNC">
<transaction mode="NON_XA"/>
<eviction strategy="LRU"
max-entries="10000"/>
<expiration max-idle="100000"/>
</invalidation-cache>
<replicated-cache name="timestamps"
mode="ASYNC"/>
</cache-container>
</subsystem>
Snip of error log below.
02:49:49,248 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60)
WFLYCLINF0002: Started sessions cache from keycloak container
02:53:48,262 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 53)
MSC000001: Failed to start service jboss.infinispan.keycloak.loginFailures:
org.jboss.msc.service.StartException in service jboss.infinispan.keycloak.loginFailures:
org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
at
org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:870)
at
org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:628)
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:531)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:222)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849)
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:621)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:572)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at
org.jboss.as.clustering.infinispan.DefaultCacheContainer.lambda$getCache$6(DefaultCacheContainer.java:119)
at
org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120)
at
org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:114)
at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)
at
org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)
... 4 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d
at
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:224)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 18 more
02:53:48,275 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,276 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "backup-for")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,276 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "backups")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,277 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "eviction")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,277 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "expiration")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,277 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "locking")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,278 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "partition-handling")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,278 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "state-transfer")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,279 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("component" => "transaction")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,279 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread)
WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "infinispan"),
("cache-container" => "keycloak"),
("distributed-cache" => "loginFailures"),
("store" => "none")
]) - failure description: {"WFLYCTL0080: Failed services" =>
{"jboss.infinispan.keycloak.loginFailures" =>
"org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for
cache loginFailures on 9c8ea5960a4d"}}
02:53:48,350 INFO [org.jboss.as.server] (ServerService Thread Pool -- 51) WFLYSRV0010:
Deployed "keycloak-server.war" (runtime-name : "keycloak-server.war")
02:53:48,369 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service
status report
WFLYCTL0186: Services which failed to start: service
jboss.infinispan.keycloak.loginFailures: org.jboss.msc.service.StartException in service
jboss.infinispan.keycloak.loginFailures: org.infinispan.commons.CacheException: Unable to
invoke method public void
org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
throws java.lang.Exception on object of type StateTransferManagerImpl