[JBoss JIRA] (ISPN-5330) Various issues with stopping a cache/cachemanager
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5330?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5330:
--------------------------------
Description:
While running stress test that constantly starts and stops a new cache manager I found several issues where a server would respond with incorrect values or exceptions.
# The Marshaller objectTable can be nulled out causing a NPE.
{code}
Caused by: java.lang.NullPointerException
at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:84)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:434)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:226)
at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:434)
at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:226)
at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:245)
... 22 more
{code}
# A command can be ran while a cache is STOPPING causing possibly invalid values to be returned. We need to most likely tweak the DistributionResponseGenerator for various commands to drop results that are null for GetKeyValueCommand for example (instead return UnsureResponse)
# If a cache has gone to TERMINATING state an exception is thrown back to the caller which is majority of cases not handled
{code}
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeIJ-43170, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:69)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:656)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:346)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSources(BaseDistributionInterceptor.java:223)
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitGetAllCommand(NonTxDistributionInterceptor.java:142)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetAllCommand(EntryWrappingInterceptor.java:150)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetAllCommand(NonTransactionalLockingInterceptor.java:50)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.statetransfer.StateTransferInterceptor.visitGetAllCommand(StateTransferInterceptor.java:177)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetAllCommand(CacheMgmtInterceptor.java:125)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:448)
at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:442)
at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:117)
at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:1)
at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'org.infinispan.commands.GetAllCommandStressTest' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.remote.ClusteredGetManyCommand.perform(ClusteredGetManyCommand.java:83)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:84)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:31)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:124)
at org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler.handle(NonTotalOrderPerCacheInboundInvocationHandler.java:53)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:326)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:297)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
at org.jgroups.JChannel.up(JChannel.java:739)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
at org.jgroups.protocols.RSVP.up(RSVP.java:201)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
at org.jgroups.protocols.Discovery.up(Discovery.java:291)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
... 3 more
{code}
was:
While running stress test that constantly starts and stops a new cache manager I found several issues where a server would respond with incorrect values or exceptions.
# The Marshaller objectTable can be nulled out causing a NPE. I don't seem to have the stack trace, hopefully will update later.
# A command can be ran while a cache is STOPPING causing possibly invalid values to be returned. We need to most likely tweak the DistributionResponseGenerator for various commands to drop results that are null for GetKeyValueCommand for example (instead return UnsureResponse)
# If a cache has gone to TERMINATING state an exception is thrown back to the caller which is majority of cases not handled
{code}
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeIJ-43170, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:69)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:656)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:346)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSources(BaseDistributionInterceptor.java:223)
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitGetAllCommand(NonTxDistributionInterceptor.java:142)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetAllCommand(EntryWrappingInterceptor.java:150)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetAllCommand(NonTransactionalLockingInterceptor.java:50)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.statetransfer.StateTransferInterceptor.visitGetAllCommand(StateTransferInterceptor.java:177)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetAllCommand(CacheMgmtInterceptor.java:125)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:448)
at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:442)
at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:117)
at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:1)
at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'org.infinispan.commands.GetAllCommandStressTest' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.remote.ClusteredGetManyCommand.perform(ClusteredGetManyCommand.java:83)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:84)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:31)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:124)
at org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler.handle(NonTotalOrderPerCacheInboundInvocationHandler.java:53)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:326)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:297)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
at org.jgroups.JChannel.up(JChannel.java:739)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
at org.jgroups.protocols.RSVP.up(RSVP.java:201)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
at org.jgroups.protocols.Discovery.up(Discovery.java:291)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
... 3 more
{code}
> Various issues with stopping a cache/cachemanager
> -------------------------------------------------
>
> Key: ISPN-5330
> URL: https://issues.jboss.org/browse/ISPN-5330
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.1.Final
> Reporter: William Burns
>
> While running stress test that constantly starts and stops a new cache manager I found several issues where a server would respond with incorrect values or exceptions.
> # The Marshaller objectTable can be nulled out causing a NPE.
> {code}
> Caused by: java.lang.NullPointerException
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:84)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:434)
> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:226)
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:434)
> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:226)
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:245)
> ... 22 more
> {code}
> # A command can be ran while a cache is STOPPING causing possibly invalid values to be returned. We need to most likely tweak the DistributionResponseGenerator for various commands to drop results that are null for GetKeyValueCommand for example (instead return UnsureResponse)
> # If a cache has gone to TERMINATING state an exception is thrown back to the caller which is majority of cases not handled
> {code}
> org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeIJ-43170, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:69)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:656)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:346)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSources(BaseDistributionInterceptor.java:223)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitGetAllCommand(NonTxDistributionInterceptor.java:142)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetAllCommand(EntryWrappingInterceptor.java:150)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetAllCommand(NonTransactionalLockingInterceptor.java:50)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
> at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitGetAllCommand(StateTransferInterceptor.java:177)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetAllCommand(CacheMgmtInterceptor.java:125)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:448)
> at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:442)
> at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:117)
> at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:1)
> at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'org.infinispan.commands.GetAllCommandStressTest' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
> at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.commands.remote.ClusteredGetManyCommand.perform(ClusteredGetManyCommand.java:83)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:84)
> at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:31)
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:124)
> at org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler.handle(NonTotalOrderPerCacheInboundInvocationHandler.java:53)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
> at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:326)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:297)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
> at org.jgroups.JChannel.up(JChannel.java:739)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
> at org.jgroups.protocols.RSVP.up(RSVP.java:201)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
> at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
> at org.jgroups.protocols.Discovery.up(Discovery.java:291)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
> ... 3 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months
[JBoss JIRA] (ISPN-5242) Adjust inconsistencies between local-cache and clustered-cache statistics
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-5242?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic updated ISPN-5242:
--------------------------------------
Summary: Adjust inconsistencies between local-cache and clustered-cache statistics (was: Minor inconsistencies between local-cache and clustered-cache statistics)
> Adjust inconsistencies between local-cache and clustered-cache statistics
> -------------------------------------------------------------------------
>
> Key: ISPN-5242
> URL: https://issues.jboss.org/browse/ISPN-5242
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Reporter: Martin Gencur
> Priority: Minor
>
> There are some inconsistencies among MBeans for local and clustered cache:
> 1)
> * local-cache stats: timeSinceReset, elapsedTime
> * cluster-cache stats name: timeSinceStart (NO timeSinceReset or elapsedTime stats here)
> 2) cluster cache now has some statistics that are not available for local cache: CacheLoaderLoads, CacheLoaderMisses, etc. I'm not sure but maybe it would be better to have this consistent.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months
[JBoss JIRA] (ISPN-5242) Minor inconsistencies between local-cache and clustered-cache statistics
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-5242?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic commented on ISPN-5242:
-------------------------------------------
Apologies. I was not right about elapsedTime. We have elapsed time in cache stats - timeSinceStart :-)
> Minor inconsistencies between local-cache and clustered-cache statistics
> ------------------------------------------------------------------------
>
> Key: ISPN-5242
> URL: https://issues.jboss.org/browse/ISPN-5242
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Reporter: Martin Gencur
> Priority: Minor
>
> There are some inconsistencies among MBeans for local and clustered cache:
> 1)
> * local-cache stats: timeSinceReset, elapsedTime
> * cluster-cache stats name: timeSinceStart (NO timeSinceReset or elapsedTime stats here)
> 2) cluster cache now has some statistics that are not available for local cache: CacheLoaderLoads, CacheLoaderMisses, etc. I'm not sure but maybe it would be better to have this consistent.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months
[JBoss JIRA] (ISPN-5242) Minor inconsistencies between local-cache and clustered-cache statistics
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-5242?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic commented on ISPN-5242:
-------------------------------------------
Martin,
You are right about 1). We could add timeSinceReset to cluster cache stats i.e org.infinispan.stats.Stats.java API. elapsedTime does not make sense in the context of cache stats as it is related to the time since cache has started. I'll make a PR with this change and will see what others say.
Regarding 2) these are here by design requirements. See org.infinispan.stats.ClusterCacheStats.java
> Minor inconsistencies between local-cache and clustered-cache statistics
> ------------------------------------------------------------------------
>
> Key: ISPN-5242
> URL: https://issues.jboss.org/browse/ISPN-5242
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Reporter: Martin Gencur
> Priority: Minor
>
> There are some inconsistencies among MBeans for local and clustered cache:
> 1)
> * local-cache stats: timeSinceReset, elapsedTime
> * cluster-cache stats name: timeSinceStart (NO timeSinceReset or elapsedTime stats here)
> 2) cluster cache now has some statistics that are not available for local cache: CacheLoaderLoads, CacheLoaderMisses, etc. I'm not sure but maybe it would be better to have this consistent.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months
[JBoss JIRA] (ISPN-5242) Minor inconsistencies between local-cache and clustered-cache statistics
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-5242?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic edited comment on ISPN-5242 at 3/31/15 4:05 PM:
--------------------------------------------------------------------
Martin,
You are right about 1). We could add timeSinceReset to cluster cache stats i.e org.infinispan.stats.Stats.java API. elapsedTime does not make sense in the context of cache stats as it is related to the elapsed time since cache has started. I'll make a PR with this change and will see what others say.
Regarding 2) these are here by design requirements. See org.infinispan.stats.ClusterCacheStats.java
was (Author: vblagojevic):
Martin,
You are right about 1). We could add timeSinceReset to cluster cache stats i.e org.infinispan.stats.Stats.java API. elapsedTime does not make sense in the context of cache stats as it is related to the time since cache has started. I'll make a PR with this change and will see what others say.
Regarding 2) these are here by design requirements. See org.infinispan.stats.ClusterCacheStats.java
> Minor inconsistencies between local-cache and clustered-cache statistics
> ------------------------------------------------------------------------
>
> Key: ISPN-5242
> URL: https://issues.jboss.org/browse/ISPN-5242
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Reporter: Martin Gencur
> Priority: Minor
>
> There are some inconsistencies among MBeans for local and clustered cache:
> 1)
> * local-cache stats: timeSinceReset, elapsedTime
> * cluster-cache stats name: timeSinceStart (NO timeSinceReset or elapsedTime stats here)
> 2) cluster cache now has some statistics that are not available for local cache: CacheLoaderLoads, CacheLoaderMisses, etc. I'm not sure but maybe it would be better to have this consistent.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months
[JBoss JIRA] (ISPN-5266) Optimize PutAll operations (Hot Rod)
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5266?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5266:
--------------------------------
Description:
- PutAll: to ensure that users can load massive amount of data into the grid quickly over Hot Rod.
- Performance improvement over large data sets.
was:
- PutAll: to ensure that users can load massive amount of data into the grid quickly, both in Library mode and over Hot Rod.
- Performance improvement over large data sets.
> Optimize PutAll operations (Hot Rod)
> ------------------------------------
>
> Key: ISPN-5266
> URL: https://issues.jboss.org/browse/ISPN-5266
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Pedro Zapata
>
> - PutAll: to ensure that users can load massive amount of data into the grid quickly over Hot Rod.
> - Performance improvement over large data sets.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months
[JBoss JIRA] (ISPN-5264) Optimize PutAll operations (library mode)
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5264?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5264:
--------------------------------
Description:
- PutAll: to ensure that users can load massive amount of data into the grid quickly in Library mode
- Performance improvement over large data sets.
was:
- PutAll: to ensure that users can load massive amount of data into the grid quickly, both in Library mode and over Hot Rod.
- Performance improvement over large data sets.
> Optimize PutAll operations (library mode)
> -----------------------------------------
>
> Key: ISPN-5264
> URL: https://issues.jboss.org/browse/ISPN-5264
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Pedro Zapata
> Assignee: William Burns
>
> - PutAll: to ensure that users can load massive amount of data into the grid quickly in Library mode
> - Performance improvement over large data sets.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
9 years, 8 months