[jboss-jira] [JBoss JIRA] (WFLY-5123) Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology

[Michal Vinkler (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100171#comment-13100171 ] 

Michal Vinkler commented on WFLY-5123:
--------------------------------------

Retested against EAP7.0.0.DR8 (Infinispan 8.0.0.Beta1) and the error is still there, although the number of occurrences is now only about 5-10 per the whole test. 

> Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology
> ------------------------------------------------------------------------------------------------
>
>                 Key: WFLY-5123
>                 URL: https://issues.jboss.org/browse/WFLY-5123
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 10.0.0.Alpha6, 10.0.0.Beta1
>            Reporter: Michal Vinkler
>            Assignee: Paul Ferraro
>
> Seen regularly in basically every failover test (no matter what failover type/cache used). 
> Setup: 4 node cluster, one node at time is shutdown, while standalone clients keep calling the application.
> After failing one node(failover type: jvmkill/shutdown/undeploy) - perf20 in this case, perf21 logged the following error message:
> {code}
> [JBossINF] [0m[31m12:57:35,029 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p7-t10) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
> [JBossINF] 	at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:274)
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:243)
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:119)
> [JBossINF] 	at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
> [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] 	at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209)
> [JBossINF] 	at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
> [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
> [JBossINF] 	at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
> [JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1613)
> [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579)
> [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572)
> [JBossINF] 	at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442)
> [JBossINF] 	at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297)
> [JBossINF] 	at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152)
> [JBossINF] 	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> [JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:497)
> [JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
> [JBossINF] 	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
> [JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
> [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212)
> [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1170)
> [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1135)
> [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:590)
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:203)
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:47)
> [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:115)
> [JBossINF] 	at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:282)
> [JBossINF] 	at org.infinispan.topology.LocalTopologyManagerImpl$1.run(LocalTopologyManagerImpl.java:217)
> [JBossINF] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] 	at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173)
> [JBossINF] 	at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
> [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
> [JBossINF] 
> {code}
> Server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-shutdown-dist-async/3/console-perf21/
> This error sometimes occur just before the server shuts down, as can be seen here:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-shutdown-dist-async/3/console-perf19/
> See also: [JBEAP-505#comment-13098103|https://issues.jboss.org/browse/JBEAP-505?focusedCommentId=13098103&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13098103]
> This issue is also related to: [WFLY-4697#comment-13083024|https://issues.jboss.org/browse/WFLY-4697?focusedCommentId=13083024&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13083024]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)