CacheException: Initial state transfer timed out for cache ---------------------------------------------------------- Key: WFLY-5307 URL: https://issues.jboss.org/browse/WFLY-5307 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 10.0.0.Beta1, 10.0.0.CR2 Reporter: Michal Vinkler Assignee: Paul Ferraro Priority: Critical Labels: clean_undeploy, clustering_revalidate EAP 7.0.0.DR9 Scenario: failover-ejb-ejbremote-undeploy-dist-async During application deployment (after failover), perf18 logged this error (besides the Replication timeout WARN message): {code} [JBossINF] [0m[0m15:29:55,511 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 85) WFLYCLINF0002: Started dist cache from ejb container [JBossINF] [0m[0m15:29:55,513 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 86) WFLYCLINF0002: Started clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container [JBossINF] [0m[0m15:29:55,516 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0002: Started clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war cache from web container [JBossINF] [0m[0m15:29:55,538 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 87) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [JBossINF] [0m[0m15:29:55,539 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 87) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. [JBossINF] [0m[0m15:29:56,499 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0002: Started clusterbench-ee7.ear/clusterbench-ee7-ejb.jar cache from ejb container [JBossINF] [0m[0m15:30:00,507 INFO [org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl] (EJB default - 1) New SFSB created: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl@5293c097. [JBossINF] [0m[0m15:30:01,323 INFO [org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl] (EJB default - 2) New SFSB created: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl@4dfb0a03. [JBossINF] [0m[33m15:30:55,461 WARN [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) ISPN000286: Issue when retrieving cluster listeners from perf19: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf19 [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589) [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$22/1199733093.apply(Unknown Source) [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954) [JBossINF] at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46) [JBossINF] at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] [JBossINF] [0m[31m15:31:55,467 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 84) MSC000001: Failed to start service jboss.infinispan.web.dist: org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320) [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:839) [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:629) [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:580) [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:445) [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:464) [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455) [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120) [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:111) [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80) [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102) [JBossINF] ... 4 more [JBossINF] Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache dist on perf18 [JBossINF] at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:225) [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [JBossINF] at java.lang.reflect.Method.invoke(Method.java:497) [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) [JBossINF] ... 19 more [JBossINF] [JBossINF] [0m[31m15:31:55,476 ERROR [org.jboss.as.server] (management-handler-thread - 2) WFLYSRV0021: Deploy of deployment "clusterbench-ee7.ear" was rolled back with the following failure message: undefined {code} Which ends up with WFLYSRV0021: *Deploy of deployment "clusterbench-ee7.ear" was rolled back*. In this very same testrun, there was another issue with application deployment on perf20, may be related: JBEAP-1043 Server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e... This issue is similar to this EAP6 bug: [BZ927948|https://bugzilla.redhat.com/show_bug.cgi?id=927948]