[jbosscache-issues] [JBoss JIRA] Assigned: (JBCACHE-1538) State transfer failures during semi-concurrent startup
Mircea Markus (JIRA)
jira-events at lists.jboss.org
Thu Sep 17 09:51:49 EDT 2009
[ https://jira.jboss.org/jira/browse/JBCACHE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mircea Markus reassigned JBCACHE-1538:
--------------------------------------
Assignee: Mircea Markus (was: Manik Surtani)
> State transfer failures during semi-concurrent startup
> ------------------------------------------------------
>
> Key: JBCACHE-1538
> URL: https://jira.jboss.org/jira/browse/JBCACHE-1538
> Project: JBoss Cache
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Clustering
> Affects Versions: 3.2.0.GA
> Reporter: Brian Stansberry
> Assignee: Mircea Markus
> Fix For: 3.3.0.GA
>
>
> Saw this twice today when starting 2 AS instances at roughly the same time, i.e. w/in 5 or so seconds:
> From the node providing state:
> ---------------------------------------------------------
> GMS: address is 127.0.0.1:45892 (cluster=DefaultPartition-SessionCache)
> ---------------------------------------------------------
> 2009-08-27 22:03:57,397 INFO [org.jboss.cache.RPCManagerImpl] (main) Received new cluster view: [127.0.0.1:45892|0] [127.0.0.1:45892]
> 2009-08-27 22:03:57,896 INFO [org.jboss.cache.RPCManagerImpl] (Incoming-19,127.0.0.1:45892) Received new cluster view: [127.0.0.1:45892|1] [127.0.0.1:45892, 192.168.201.145:60963]
> 2009-08-27 22:03:58,210 INFO [org.jboss.cache.RPCManagerImpl] (Incoming-6,127.0.0.1:45892) Received new cluster view: [127.0.0.1:45892|1] [127.0.0.1:45892, 192.168.201.145:60963]
> 2009-08-27 22:03:59,767 INFO [org.jboss.cache.RPCManagerImpl] (Incoming-16,127.0.0.1:45892) Received new cluster view: [127.0.0.1:45892|1] [127.0.0.1:45892, 192.168.201.145:60963]
> 2009-08-27 22:04:00,965 INFO [org.jboss.cache.RPCManagerImpl] (Incoming-7,127.0.0.1:45892) Received new cluster view: [127.0.0.1:45892|1] [127.0.0.1:45892, 192.168.201.145:60963]
> 2009-08-27 22:04:00,978 ERROR [org.jboss.cache.remoting.jgroups.ChannelMessageListener] (Incoming-8,127.0.0.1:45892) Caught java.lang.NullPointerException while responding to state transfer request
> java.lang.NullPointerException
> at org.jboss.cache.statetransfer.LegacyStateTransferGenerator.generateNodeDataList(LegacyStateTransferGenerator.java:175)
> at org.jboss.cache.statetransfer.LegacyStateTransferGenerator.marshallTransientState(LegacyStateTransferGenerator.java:169)
> at org.jboss.cache.statetransfer.LegacyStateTransferGenerator.generateState(LegacyStateTransferGenerator.java:82)
> at org.jboss.cache.statetransfer.LegacyStateTransferManager.getState(LegacyStateTransferManager.java:83)
> at org.jboss.cache.remoting.jgroups.ChannelMessageListener.getState(ChannelMessageListener.java:163)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:658)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:776)
> at org.jgroups.JChannel.up(JChannel.java:1338)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
> at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:486)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.requestApplicationStates(STATE_TRANSFER.java:241)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.handleStateReq(STATE_TRANSFER.java:374)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:117)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
> at org.jgroups.protocols.FC.up(FC.java:473)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)
> at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
> at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
> at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:758)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
> at org.jgroups.protocols.FD.up(FD.java:284)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:309)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
> at org.jgroups.protocols.Discovery.up(Discovery.java:264)
> at org.jgroups.protocols.PING.up(PING.java:273)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2327)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1261)
> at org.jgroups.protocols.TP.access$100(TP.java:49)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1838)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> 2009-08-27 22:04:00,991 INFO [org.jboss.cache.RPCManagerImpl] (main) Cache local address is 127.0.0.1:45892
> 2009-08-27 22:04:01,022 INFO [org.jboss.cache.RPCManagerImpl] (Incoming-11,127.0.0.1:45892) Received new cluster view: [127.0.0.1:45892|2] [127.0.0.1:45892]
> The node receiving the state doesn't deal with the failure cleanly:
> ---------------------------------------------------------
> GMS: address is 192.168.201.145:60963 (cluster=DefaultPartition-SessionCache)
> ---------------------------------------------------------
> 2009-08-27 22:04:00,968 INFO [org.jboss.cache.RPCManagerImpl] (main) Received new cluster view: [127.0.0.1:45892|1] [127.0.0.1:45892, 192.168.201.145:60963]
> 2009-08-27 22:04:00,981 INFO [org.jboss.cache.statetransfer.LegacyStateTransferIntegrator] (Incoming-7,192.168.201.145:60963) Using version 4096
> 2009-08-27 22:04:00,989 INFO [org.jboss.cache.RPCManagerImpl] (main) Cache local address is 192.168.201.145:60963
> 2009-08-27 22:04:00,989 INFO [org.jboss.cache.RPCManagerImpl] (main) Disconnecting and closing the Channel
> 2009-08-27 22:04:01,030 ERROR [org.jboss.web.tomcat.service.session.JBossCacheManager.Order] (main) Unable to start manager.
> org.jboss.cache.CacheException: java.lang.reflect.InvocationTargetException
> at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:148)
> at org.jboss.cache.factories.ComponentRegistry$PrioritizedMethod.invoke(ComponentRegistry.java:1012)
> at org.jboss.cache.factories.ComponentRegistry.internalStart(ComponentRegistry.java:778)
> at org.jboss.cache.factories.ComponentRegistry.start(ComponentRegistry.java:632)
> at org.jboss.cache.invocation.CacheInvocationDelegate.start(CacheInvocationDelegate.java:345)
> at org.jboss.ha.cachemanager.CacheManagerManagedCache.start(CacheManagerManagedCache.java:98)
> at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.AbstractJBossCacheService.start(AbstractJBossCacheService.java:206)
> at org.jboss.web.tomcat.service.session.JBossCacheManager.startEmbedded(JBossCacheManager.java:2584)
> at org.jboss.web.tomcat.service.session.JBossCacheManager.start(JBossCacheManager.java:789)
> at org.apache.catalina.core.ContainerBase.setManager(ContainerBase.java:447)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.tomcat.util.modeler.BaseModelMBean.setAttribute(BaseModelMBean.java:414)
> at org.jboss.mx.server.RawDynamicInvoker.setAttribute(RawDynamicInvoker.java:70)
> at org.jboss.mx.server.MBeanServerImpl.setAttribute(MBeanServerImpl.java:617)
> at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeployInternal(TomcatDeployment.java:348)
> at org.jboss.web.tomcat.service.deployers.TomcatDeployment.performDeploy(TomcatDeployment.java:142)
> at org.jboss.web.deployers.AbstractWarDeployment.start(AbstractWarDeployment.java:461)
> at org.jboss.web.deployers.WebModule.startModule(WebModule.java:118)
> at org.jboss.web.deployers.WebModule.start(WebModule.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
> at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
> at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
> at $Proxy38.start(Unknown Source)
> at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
> at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
> at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
> at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
> at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
> at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
> at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1632)
> at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
> at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
> at org.jboss.system.ServiceController.start(ServiceController.java:460)
> at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
> at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
> at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
> at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
> at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
> at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1440)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1158)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1179)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1211)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1099)
> at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1632)
> at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
> at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
> at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:782)
> at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
> at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
> at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
> at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
> at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
> at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1632)
> at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
> at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:775)
> at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
> at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
> at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
> at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
> at org.jboss.Main.boot(Main.java:221)
> at org.jboss.Main$1.run(Main.java:556)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.jboss.cache.util.reflect.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:144)
> ... 84 more
> Caused by: org.jboss.cache.CacheException: Unable to fetch state on startup
> at org.jboss.cache.RPCManagerImpl.start(RPCManagerImpl.java:396)
> ... 89 more
> Caused by: org.jboss.cache.CacheException: java.lang.ClassCastException: org.jboss.cache.marshall.NodeDataExceptionMarker cannot be cast to java.util.List
> at org.jboss.cache.statetransfer.LegacyStateTransferIntegrator.integrateTransientState(LegacyStateTransferIntegrator.java:118)
> at org.jboss.cache.statetransfer.LegacyStateTransferIntegrator.integrateState(LegacyStateTransferIntegrator.java:88)
> at org.jboss.cache.statetransfer.LegacyStateTransferManager.setState(LegacyStateTransferManager.java:155)
> at org.jboss.cache.statetransfer.DefaultStateTransferManager.setState(DefaultStateTransferManager.java:163)
> at org.jboss.cache.remoting.jgroups.ChannelMessageListener.setState(ChannelMessageListener.java:190)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:676)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:776)
> at org.jgroups.JChannel.up(JChannel.java:1251)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:462)
> at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:486)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.handleStateRsp(STATE_TRANSFER.java:430)
> at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:124)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
> at org.jgroups.protocols.FC.up(FC.java:473)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)
> at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
> at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:616)
> at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
> at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:758)
> at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
> at org.jgroups.protocols.FD.up(FD.java:284)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:309)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
> at org.jgroups.protocols.Discovery.up(Discovery.java:264)
> at org.jgroups.protocols.PING.up(PING.java:273)
> at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2327)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1261)
> at org.jgroups.protocols.TP.access$100(TP.java:49)
> at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1838)
> at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1817)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> Caused by: java.lang.ClassCastException: org.jboss.cache.marshall.NodeDataExceptionMarker cannot be cast to java.util.List
> at org.jboss.cache.statetransfer.LegacyStateTransferIntegrator.readNodesAsList(LegacyStateTransferIntegrator.java:278)
> at org.jboss.cache.statetransfer.LegacyStateTransferIntegrator.integrateTransientState(LegacyStateTransferIntegrator.java:217)
> at org.jboss.cache.statetransfer.LegacyStateTransferIntegrator.integrateTransientState(LegacyStateTransferIntegrator.java:105)
> ... 34 more
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jbosscache-issues
mailing list