[infinispan-issues] [JBoss JIRA] (ISPN-2572) "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite

Tue Dec 4 11:19:21 EST 2012

    [ https://issues.jboss.org/browse/ISPN-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739328#comment-12739328 ] 

Radoslav Husar commented on ISPN-2572:
--------------------------------------

Hi Dan,

lets look at the logs. Looks to me that ClusterTopologyManager picks up the topology, so that doesnt seem to be an issue here.

{noformat}
15:46:57,442 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (notification-thread-0) Received new cluster view: 11
15:46:57,442 TRACE [org.infinispan.topology.ClusterTopologyManagerImpl] (notification-thread-0) Updating cluster members for all the caches. New list is [node-0/ejb, node-1/ejb]
{noformat}

Whats interesting, node1 says

{noformat}15:46:57,476 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (ServerService Thread Pool -- 15) Waiting for initial state transfer to finish for cache repl on node-1/ejb
{noformat}

node0 says

{noformat}
15:46:57,474 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-13,null) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=repl, type=JOIN, sender=node-1/ejb, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.ReplicatedConsistentHashFactory at 43e3ac, hashFunction=org.infinispan.commons.hash.MurmurHash3 at 118618d, numSegments=60, numOwners=2, timeout=60000}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=11} [sender=node-1/ejb]
15:46:57,474 TRACE [org.infinispan.topology.DefaultRebalancePolicy] (OOB-13,null) Cache repl status changed: joiners=[node-1/ejb], topology=CacheTopology{id=7, currentCH=ReplicatedConsistentHash{members=[node-0/ejb, node-1/ejb]}, pendingCH=ReplicatedConsistentHash{members=[node-0/ejb, node-1/ejb]}}
15:46:57,475 TRACE [org.infinispan.topology.DefaultRebalancePolicy] (OOB-13,null) Not triggering rebalance for cache %s, a rebalance is already in progress
{noformat}

So we end up with  "Not triggering rebalance for cache %s, a rebalance is already in progress" and then timeout -- Not sure which cache so I submitted: https://github.com/infinispan/infinispan/pull/1506

 Could it be the problem is the rebalance?

Full logs are here
http://teamcity.cafe-babe.org/repository/download/bt2/1755:id/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-0/standalone/log/server.log
http://teamcity.cafe-babe.org/repository/download/bt2/1755:id/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-1/standalone/log/server.log
http://teamcity.cafe-babe.org/viewLog.html?buildId=1755&buildTypeId=bt2

> "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
> --------------------------------------------------------------------------------------
>
>                 Key: ISPN-2572
>                 URL: https://issues.jboss.org/browse/ISPN-2572
>             Project: Infinispan
>          Issue Type: Bug
>          Components: State transfer
>    Affects Versions: 5.2.0.Beta4
>            Reporter: Radoslav Husar
>            Assignee: Dan Berindei
>            Priority: Blocker
>             Fix For: 5.2.0.Beta6
>
>
> While running AS7 testsuite with speedups implemented in my branch (https://github.com/jbossas/jboss-as/pull/3381) we are contantly seeing (log below) on Windows 2008.
> Run:
> http://teamcity.cafe-babe.org/viewLog.html?buildId=1689&tab=buildResultsDiv&buildTypeId=bt2
> {code}
> 16:34:46,092 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 13) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
>     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
>     at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
>     at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
>     at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
>     at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
>     at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
>     at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
>     at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
>     at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)
>     at org.infinispan.CacheImpl.start(CacheImpl.java:520)
>     at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)
>     at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
>     at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
>     at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
>     at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
>     at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
>     at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
>     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
>     ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-connector-client-mappings on node-1/ejb
>     at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
>     at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
>     at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
>     ... 18 more
> {code}
> Affected version -- current master (say 7dc531002539b078e429418d8ef204e401beafd1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira