[jboss-jira] [JBoss JIRA] (WFLY-5123) Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology

Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology ------------------------------------------------------------------------------------------------ Key: WFLY-5123 URL: https://issues.jboss.org/browse/WFLY-5123 Project: WildFly Issue Type: Bug Components: Clustering Affects Versions: 10.0.0.Alpha6, 10.0.0.Beta1 Reporter: Michal Vinkler Assignee: Paul Ferraro Fix For: 10.0.0.CR1 Seen regularly in basically every failover test (no matter what failover type/cache used). Setup: 4 node cluster, one node at time is shutdown, while standalone clients keep calling the application. After failing one node(failover type: jvmkill/shutdown/undeploy) - perf20 in this case, perf21 logged the following error message: {code} [JBossINF] [0m[31m12:57:35,029 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p7-t10) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20 [JBossINF] at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92) [JBossINF] at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:274) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:243) [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:119) [JBossINF] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209) [JBossINF] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [JBossINF] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) [JBossINF] at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) [JBossINF] at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49) [JBossINF] at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [JBossINF] at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1613) [JBossINF] at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579) [JBossINF] at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572) [JBossINF] at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297) [JBossINF] at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152) [JBossINF] at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [JBossINF] at java.lang.reflect.Method.invoke(Method.java:497) [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286) [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22) [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309) [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212) [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1170) [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1135) [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:590) [JBossINF] at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:203) [JBossINF] at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:47) [JBossINF] at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:115) [JBossINF] at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:282) [JBossINF] at org.infinispan.topology.LocalTopologyManagerImpl$1.run(LocalTopologyManagerImpl.java:217) [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173) [JBossINF] at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745) [JBossINF] {code} Server log: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e... This error sometimes occur just before the server shuts down, as can be seen here: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e... See also: [JBEAP-505#comment-13098103|https://issues.jboss.org/browse/JBEAP-505?focu...] This issue is also related to: [WFLY-4697#comment-13083024|https://issues.jboss.org/browse/WFLY-4697?focu...]