[infinispan-issues] [JBoss JIRA] (ISPN-2572) "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
Dan Berindei (JIRA)
jira-events at lists.jboss.org
Mon Dec 3 13:46:22 EST 2012
[ https://issues.jboss.org/browse/ISPN-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739027#comment-12739027 ]
Dan Berindei commented on ISPN-2572:
------------------------------------
I looked at the remote-connector-client-mappings cache history and it looks like the ClusterTopologyManager component on the coordinator missed a notification of node-1/ejb leaving the cluster. When it joined again (with a different JGroups address), the old node-1 obviously wouldn't confirm the rebalance, and so the new node-1 couldn't finish the join.
{noformat}
20:12:34,554 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-19,null) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=remote-connector-client-mappings, type=JOIN, sender=node-1/ejb, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.ReplicatedConsistentHashFactory at f35bd, hashFunction=org.infinispan.commons.hash.MurmurHash3 at 911387, numSegments=60, numOwners=2, timeout=60000}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=5} [sender=node-1/ejb]
20:12:34,580 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (OOB-13,null) Installing new cache topology CacheTopology{id=6, currentCH=ReplicatedConsistentHash{members=[node-0/ejb, node-1/ejb]}, pendingCH=null} on cache remote-connector-client-mappings
20:12:34,580 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (transport-thread-1) Replication task sending CacheTopologyControlCommand{cache=remote-connector-client-mappings, type=CH_UPDATE, sender=node-0/ejb, joinInfo=null, topologyId=6, currentCH=ReplicatedConsistentHash{members=[node-0/ejb, node-1/ejb]}, pendingCH=null, throwable=null, viewId=5} to addresses null with response mode GET_ALL
20:12:38,042 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-16,null) ISPN000094: Received new cluster view: [node-0/ejb|6] [node-0/ejb]
20:12:42,758 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,null) ISPN000094: Received new cluster view: [node-0/ejb|7] [node-0/ejb, node-1/ejb]
20:12:42,933 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-13,null) Attempting to execute non-CacheRpcCommand command: CacheTopologyControlCommand{cache=remote-connector-client-mappings, type=JOIN, sender=node-1/ejb, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.ReplicatedConsistentHashFactory at 12cdb01, hashFunction=org.infinispan.commons.hash.MurmurHash3 at 1de2ea4, numSegments=60, numOwners=2, timeout=60000}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=7} [sender=node-1/ejb]
20:12:42,933 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (transport-thread-22) Installing new cache topology CacheTopology{id=7, currentCH=ReplicatedConsistentHash{members=[node-0/ejb, node-1/ejb]}, pendingCH=ReplicatedConsistentHash{members=[node-1/ejb, node-0/ejb, node-1/ejb]}} on cache remote-connector-client-mappings
{noformat}
[~rhusar], could you get another round of logs with TRACE enabled for org.infinispan.topology as well? JGroupsTransport does install view 6 with a single member, so ClusterTopologyManager should update the cache topology as well. There is at least one test that removes a node without it sending a cache LEAVE command in the Infinispan test suite - RehashAfterPartitionMergeTest - and it works fine.
> "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
> --------------------------------------------------------------------------------------
>
> Key: ISPN-2572
> URL: https://issues.jboss.org/browse/ISPN-2572
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta4
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Priority: Blocker
> Fix For: 5.2.0.CR1
>
>
> While running AS7 testsuite with speedups implemented in my branch (https://github.com/jbossas/jboss-as/pull/3381) we are contantly seeing (log below) on Windows 2008.
> Run:
> http://teamcity.cafe-babe.org/viewLog.html?buildId=1689&tab=buildResultsDiv&buildTypeId=bt2
> {code}
> 16:34:46,092 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 13) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)
> at org.infinispan.CacheImpl.start(CacheImpl.java:520)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
> ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-connector-client-mappings on node-1/ejb
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 18 more
> {code}
> Affected version -- current master (say 7dc531002539b078e429418d8ef204e401beafd1).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the infinispan-issues
mailing list