[keycloak-user] Initial state transfer times out for cache loginFailures
Stian Thorgersen
sthorger at redhat.com
Wed Jun 28 01:36:31 EDT 2017
Should be fixed with https://issues.jboss.org/browse/ISPN-6806, but that is
not included in Keycloak until we upgrade to WildFly 11. That'll be another
couple months probably. It's fixed in RH-SSO 7.1 as it's fixed in JBoss EAP.
There's a potential workaround:
As a workaround for RH SSO 7.0.0 users, disable L1 caching by setting
l1-lifespan="0" on the distributed-caches.
You could maybe also upgrade the Infinispan version yourself. Or get the
supported version of Keycloak (RH-SSO). Or wait....
On 28 June 2017 at 00:09, Dan Corbett <dcorbett at expedia.com> wrote:
> We have a 3 node cluster of Keycloak 3.1.0.Final (WildFly Core
> 2.0.10.Final) running in Docker AWS ECS.
> It all has been running smoothly for many months, through various
> versions, deployments and recycles.
>
> Just recently, when one of the containers was terminated, the new
> container was run by ECS but could not start Keycloak and join the cluster.
>
> Error is: Failed to start service jboss.infinispan.keycloak.loginFailures:
> org.jboss.msc.service.StartException in service jboss.infinispan.keycloak.loginFailures,
> Initial state transfer timed out for cache loginFailures
>
> We also see some of the following, which may be related?
> 2017-06-27 05:54:01,914 DEBUG [org.infinispan.remoting.inboundhandler.
> NonTotalOrderPerCacheInboundInvocationHandler] (remote-thread--p10-t3)
> ISPN000311: Received a command from an outdated topology, returning the
> exception to caller: org.infinispan.statetransfer.OutdatedTopologyException:
> Cache topology changed while the command was executing: expected 1962, got
> 1963
>
> We enabled various Debug log levels but haven’t yet found anything
> conclusive.
> We tried clearing caches via Infinispan JMX tooling etc, but still could
> not start the new node.
> The two remaining instances were still clustered OK.
> We provisioned additional cluster and it's working as expected, so don’t
> believe at this stage it’s a network communication issue.
>
> Found this report for Redhat SSO, seems similar? I can’t see solution.
> https://access.redhat.com/solutions/2841711
>
> Any info on possible cause or how to debug/investigate further would be a
> great help.
> Configs and logs below..
>
> Thanks
> Dan
>
> Our cache config is:
>
> <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
> <cache-container name="keycloak" jndi-name="infinispan/
> Keycloak">
> <transport lock-timeout="60000"/>
> <local-cache name="realms">
> <eviction max-entries="10000" strategy="LRU"/>
> </local-cache>
> <local-cache name="users">
> <eviction max-entries="10000" strategy="LRU"/>
> </local-cache>
> <distributed-cache name="sessions" mode="SYNC" owners="5"
> />
> <distributed-cache name="offlineSessions" mode="SYNC"
> owners="1" />
> <distributed-cache name="loginFailures" mode="SYNC"
> owners="1"/>
> <distributed-cache name="authorization" mode="SYNC"
> owners="1"/>
> <replicated-cache name="work" mode="SYNC"/>
> <local-cache name="keys">
> <eviction max-entries="1000" strategy="LRU"/>
> <expiration max-idle="3600000"/>
> </local-cache>
> </cache-container>
> <cache-container name="server" aliases="singleton cluster"
> default-cache="default" module="org.wildfly.clustering.server">
> <transport lock-timeout="60000"/>
> <replicated-cache name="default" mode="SYNC">
> <transaction mode="BATCH"/>
> </replicated-cache>
> </cache-container>
> <cache-container name="web" default-cache="dist"
> module="org.wildfly.clustering.web.infinispan">
> <transport lock-timeout="60000"/>
> <distributed-cache name="dist" mode="ASYNC"
> l1-lifespan="0" owners="2">
> <locking isolation="REPEATABLE_READ"/>
> <transaction mode="BATCH"/>
> <file-store/>
> </distributed-cache>
> </cache-container>
> <cache-container name="ejb" aliases="sfsb"
> default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
> <transport lock-timeout="60000"/>
> <distributed-cache name="dist" mode="ASYNC"
> l1-lifespan="0" owners="2">
> <locking isolation="REPEATABLE_READ"/>
> <transaction mode="BATCH"/>
> <file-store/>
> </distributed-cache>
> </cache-container>
> <cache-container name="hibernate" default-cache="local-query"
> module="org.hibernate.infinispan">
> <transport lock-timeout="60000"/>
> <local-cache name="local-query">
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration max-idle="100000"/>
> </local-cache>
> <invalidation-cache name="entity" mode="SYNC">
> <transaction mode="NON_XA"/>
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration max-idle="100000"/>
> </invalidation-cache>
> <replicated-cache name="timestamps" mode="ASYNC"/>
> </cache-container>
> </subsystem>
>
> Snip of error log below.
>
> 02:49:49,248 INFO [org.jboss.as.clustering.infinispan] (ServerService
> Thread Pool -- 60) WFLYCLINF0002: Started sessions cache from keycloak
> container
> 02:53:48,262 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool
> -- 53) MSC000001: Failed to start service jboss.infinispan.keycloak.loginFailures:
> org.jboss.msc.service.StartException in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(
> AsynchronousServiceBuilder.java:107)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method
> public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(
> ReflectionUtil.java:172)
> at org.infinispan.factories.AbstractComponentRegistry$
> PrioritizedMethod.invoke(AbstractComponentRegistry.java:870)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(
> AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(
> AbstractComponentRegistry.java:628)
> at org.infinispan.factories.AbstractComponentRegistry.start(
> AbstractComponentRegistry.java:531)
> at org.infinispan.factories.ComponentRegistry.start(
> ComponentRegistry.java:222)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(
> DefaultCacheManager.java:621)
> at org.infinispan.manager.DefaultCacheManager.createCache(
> DefaultCacheManager.java:572)
> at org.infinispan.manager.DefaultCacheManager.getCache(
> DefaultCacheManager.java:440)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.lambda$
> getCache$6(DefaultCacheContainer.java:119)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(
> DefaultCacheContainer.java:120)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(
> DefaultCacheContainer.java:114)
> at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(
> CacheBuilder.java:80)
> at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(
> AsynchronousServiceBuilder.java:102)
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: Initial state transfer
> timed out for cache loginFailures on 9c8ea5960a4d
> at org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:224)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(
> ReflectionUtil.java:168)
> ... 18 more
> 02:53:48,275 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,276 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "backup-for")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,276 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "backups")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,277 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "eviction")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,277 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "expiration")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,277 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "locking")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,278 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "partition-handling")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,278 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "state-transfer")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,279 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("component" => "transaction")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,279 ERROR [org.jboss.as.controller.management-operation]
> (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
> ("subsystem" => "infinispan"),
> ("cache-container" => "keycloak"),
> ("distributed-cache" => "loginFailures"),
> ("store" => "none")
> ]) - failure description: {"WFLYCTL0080: Failed services" =>
> {"jboss.infinispan.keycloak.loginFailures" => "org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Unable to invoke
> method public void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
> Caused by: org.infinispan.commons.CacheException: Initial state
> transfer timed out for cache loginFailures on 9c8ea5960a4d"}}
> 02:53:48,350 INFO [org.jboss.as.server] (ServerService Thread Pool -- 51)
> WFLYSRV0010: Deployed "keycloak-server.war" (runtime-name :
> "keycloak-server.war")
> 02:53:48,369 INFO [org.jboss.as.controller] (Controller Boot Thread)
> WFLYCTL0183: Service status report
> WFLYCTL0186: Services which failed to start: service
> jboss.infinispan.keycloak.loginFailures: org.jboss.msc.service.StartException
> in service jboss.infinispan.keycloak.loginFailures:
> org.infinispan.commons.CacheException: Unable to invoke method public
> void org.infinispan.statetransfer.StateTransferManagerImpl.
> waitForInitialStateTransferToComplete() throws java.lang.Exception on
> object of type StateTransferManagerImpl
>
>
> _______________________________________________
> keycloak-user mailing list
> keycloak-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-user
>
More information about the keycloak-user
mailing list