[infinispan-issues] [JBoss JIRA] (ISPN-3395) ISPN000196: Failed to recover cluster state after the current node became the coordinator
Ernest Bursa (JIRA)
jira-events at lists.jboss.org
Sun Nov 17 17:49:06 EST 2013
[ https://issues.jboss.org/browse/ISPN-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924264#comment-12924264 ]
Ernest Bursa commented on ISPN-3395:
------------------------------------
+1, I also found this error in my log of #TorqueBox server
{code}
22:44:07,531 INFO [org.projectodd.polyglot.hasingleton] (AsynchViewChangeHandler Thread) Becoming HASingleton master (BeardedHipster-knob.yml-hasingleton-global)
22:44:32,643 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 50) MSC00001: Failed to start service jboss.infinispan.web.repl: org.jboss.msc.service.StartException in service jboss.infinispan.web.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) [jboss-as-clustering-common-7.2.x.slim.incremental.9.jar:7.2.x.slim.incremental.9]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final.jar:2.1.0.Final]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:207)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:658)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:647)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:550)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:221)
at org.infinispan.CacheImpl.start(CacheImpl.java:691)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:685)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:648)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:544)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:558)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) [jboss-as-clustering-common-7.2.x.slim.incremental.9.jar:7.2.x.slim.incremental.9]
... 4 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Node torquebox-app03/web timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:296)
at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:110)
at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_04]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_04]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_04]
at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_04]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
... 18 more
Caused by: org.jgroups.TimeoutException: TimeoutException
at org.jgroups.util.Promise._getResultWithTimeout(Promise.java:145)
at org.jgroups.util.Promise.getResultWithTimeout(Promise.java:40)
at org.jgroups.util.AckCollector.waitForAllAcks(AckCollector.java:93)
at org.jgroups.protocols.RSVP$Entry.block(RSVP.java:321)
at org.jgroups.protocols.RSVP.down(RSVP.java:117)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1022)
at org.jgroups.JChannel.down(JChannel.java:767)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683)
at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202)
at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43)
at org.jgroups.blocks.Request.execute(Request.java:83)
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
... 27 more
22:44:33,138 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 52) MSC00001: Failed to start service jboss.infinispan.web.default-host/ROOT: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/ROOT: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final.jar:2.1.0.Final]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:207)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:658)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:647)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:550)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:221)
at org.infinispan.CacheImpl.start(CacheImpl.java:691)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:685)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:648)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:544)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:558)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
... 4 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Node torquebox-app03/web timed out
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:296)
at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:110)
at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_04]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_04]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_04]
at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_04]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
... 18 more
Caused by: org.jgroups.TimeoutException: TimeoutException
at org.jgroups.util.Promise._getResultWithTimeout(Promise.java:145) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.util.Promise.getResultWithTimeout(Promise.java:40) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.util.AckCollector.waitForAllAcks(AckCollector.java:93) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.protocols.RSVP$Entry.block(RSVP.java:321) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.protocols.RSVP.down(RSVP.java:117) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1022) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.JChannel.down(JChannel.java:767) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:683) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:202) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:43) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.blocks.Request.execute(Request.java:83) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:399) [jgroups-3.3.1.Final.jar:3.3.1.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
... 27 more
22:44:45,888 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-9,shared=tcp) ISPN000094: Received new cluster view: [torquebox-app01/web|3] [torquebox-app01/web]
22:44:45,909 ERROR [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread-0) ISPN000196: Failed to recover cluster state after the current node became the coordinator: java.lang.NullPointerException
at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:455)
at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:235)
at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:647)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_04]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_04]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_04]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
{code}
This error is stopping deploy of TorqueBox application to Jboss.
> ISPN000196: Failed to recover cluster state after the current node became the coordinator
> -----------------------------------------------------------------------------------------
>
> Key: ISPN-3395
> URL: https://issues.jboss.org/browse/ISPN-3395
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.3.0.Final
> Reporter: Mayank Agarwal
> Assignee: Mircea Markus
>
> We are using infinispan 5.3.0.Final in our distributed application. we are testing infinispan in HA scenarios and getting following exception when new node becomes co-ordinator.
> ISPN000196: Failed to recover cluster state after the current node became the coordinator
> java.lang.NullPointerException: null
> at org.infinispan.topology.ClusterTopologyManagerImpl.recoverClusterStatus(ClusterTopologyManagerImpl.java:455) ~[infinispan-core-5.3.0.1.Final.jar:5.3.0.1.Final]
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleNewView(ClusterTopologyManagerImpl.java:235) ~[infinispan-core-5.3.0.1.Final.jar:5.3.0.1.Final]
> at org.infinispan.topology.ClusterTopologyManagerImpl$ClusterViewListener$1.run(ClusterTopologyManagerImpl.java:647) ~[infinispan-core-5.3.0.1.Final.jar:5.3.0.1.Final]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_25]
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_25]
> at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.6.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [na:1.6.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.6.0_25]
> at java.lang.Thread.run(Unknown Source) [na:1.6.0_25]
> This is happening because cacheTopology is null at ClusterTopologyManagerImpl.java:455
> at 449: code is checking cacheTopology for null that for loop which is updating cacheStatusMap at 457 should be in that check itself.
> Fix:
> --- a/core/src/main/java/org/infinispan/topology/ClusterTopologyManagerImpl.java
> +++ b/core/src/main/java/org/infinispan/topology/ClusterTopologyManagerImpl.java
> @@ -448,7 +448,7 @@ public class ClusterTopologyManagerImpl implements ClusterTopologyManager {
> // but didn't get a response back yet
> if (cacheTopology != null) {
> topologyList.add(cacheTopology);
> - }
> +
>
> // Add all the members of the topology that have sent responses first
> // If we only added the sender, we could end up with a different member order
> @@ -457,6 +457,7 @@ public class ClusterTopologyManagerImpl implements ClusterTopologyManager {
> cacheStatusMap.get(cacheName).addMember(member);
> }
> }
> + }
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the infinispan-issues
mailing list