[infinispan-issues] [JBoss JIRA] (ISPN-2572) "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740283#comment-12740283 ] 

Dan Berindei commented on ISPN-2572:
------------------------------------

There is a with the way we process JGroups view updates in ClusterTopologyManagerImpl. We update the members list for each cache in order, but if one of the caches' members list doesn't need any changes we skip processing all the remaining caches. That's why we can see in the log {{Updating cluster members for all the caches}} after node-1/ejb leaves, but no {{Cache repl topology updated}} messages follow it.

@Erik, I think the problem in your case is slightly different: you to have an async listener that takes a very long time (I think it tries to bring a site online). Because we only have one notification thread by default, this delays ClusterTopologyManagerImpl's own listener a lot. The coordinator can't handle cache joins for view id x until the listener executes for view x, so the joiner times out. Try to increase the number of async notification threads, this should work:

{noformat}
      globalConfigurationBuilder.asyncTransportExecutor().addProperty("maxThreads", "3");
{noformat}

                
> "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
> --------------------------------------------------------------------------------------
>
>                 Key: ISPN-2572
>                 URL: https://issues.jboss.org/browse/ISPN-2572
>             Project: Infinispan
>          Issue Type: Bug
>          Components: State transfer
>    Affects Versions: 5.2.0.Beta4
>            Reporter: Radoslav Husar
>            Assignee: Dan Berindei
>            Priority: Blocker
>             Fix For: 5.2.0.Beta6
>
>
> While running AS7 testsuite with speedups implemented in my branch (https://github.com/jbossas/jboss-as/pull/3381) we are contantly seeing (log below) on Windows 2008.
> Run:
> http://teamcity.cafe-babe.org/viewLog.html?buildId=1689&tab=buildResultsDiv&buildTypeId=bt2
> {code}
> 16:34:46,092 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 13) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
>     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
>     at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
>     at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
>     at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
>     at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
>     at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
>     at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
>     at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
>     at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)
>     at org.infinispan.CacheImpl.start(CacheImpl.java:520)
>     at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)
>     at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
>     at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
>     at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
>     at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
>     at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
>     at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
>     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
>     ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-connector-client-mappings on node-1/ejb
>     at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
>     at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
>     at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
>     ... 18 more
> {code}
> Affected version -- current master (say 7dc531002539b078e429418d8ef204e401beafd1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira