[JBoss JIRA] (ISPN-5330) Various issues with stopping a cache/cachemanager
by William Burns (JIRA)
William Burns created ISPN-5330:
-----------------------------------
Summary: Various issues with stopping a cache/cachemanager
Key: ISPN-5330
URL: https://issues.jboss.org/browse/ISPN-5330
Project: Infinispan
Issue Type: Bug
Affects Versions: 7.1.1.Final
Reporter: William Burns
While running stress test that constantly starts and stops a new cache manager I found several issues where a server would respond with incorrect values or exceptions.
# The Marshaller objectTable can be nulled out causing a NPE. I don't seem to have the stack trace, hopefully will update later.
# A command can be ran while a cache is STOPPING causing possibly invalid values to be returned. We need to most likely tweak the DistributionResponseGenerator for various commands to drop results that are null for GetKeyValueCommand for example (instead return UnsureResponse)
# If a cache has gone to TERMINATING state an exception is thrown back to the caller which is majority of cases not handled
{code}
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeIJ-43170, see cause for remote stack trace
at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:69)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:656)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:346)
at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSources(BaseDistributionInterceptor.java:223)
at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitGetAllCommand(NonTxDistributionInterceptor.java:142)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetAllCommand(EntryWrappingInterceptor.java:150)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetAllCommand(NonTransactionalLockingInterceptor.java:50)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.statetransfer.StateTransferInterceptor.visitGetAllCommand(StateTransferInterceptor.java:177)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetAllCommand(CacheMgmtInterceptor.java:125)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:448)
at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:442)
at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:117)
at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:1)
at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'org.infinispan.commands.GetAllCommandStressTest' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.commands.remote.ClusteredGetManyCommand.perform(ClusteredGetManyCommand.java:83)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:84)
at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:31)
at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:124)
at org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler.handle(NonTotalOrderPerCacheInboundInvocationHandler.java:53)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:326)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:297)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
at org.jgroups.JChannel.up(JChannel.java:739)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
at org.jgroups.protocols.RSVP.up(RSVP.java:201)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
at org.jgroups.protocols.Discovery.up(Discovery.java:291)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
... 3 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5324) Create extended RemoteCacheManager for testing purposes
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5324?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5324:
-----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 7.2.0.Beta2
Resolution: Done
> Create extended RemoteCacheManager for testing purposes
> -------------------------------------------------------
>
> Key: ISPN-5324
> URL: https://issues.jboss.org/browse/ISPN-5324
> Project: Infinispan
> Issue Type: Feature Request
> Components: Test Suite - Core
> Reporter: Martin Gencur
> Assignee: Martin Gencur
> Fix For: 7.2.0.Beta2
>
>
> The extended RemoteCacheManager should expose details such as the following:
> * transportFactory
> The implementation should be internal and NOT part of public API.
> This way we can get rid of reflection in tests for HotRod client and these tests can be reused for C++ and C# HotRod clients. Any tests that use reflection are currently unusable for those implementations.
> The ultimate goal is to extend the test suite for C++ and C# clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5329) Reduce number of allocations
by Radim Vansa (JIRA)
Radim Vansa created ISPN-5329:
---------------------------------
Summary: Reduce number of allocations
Key: ISPN-5329
URL: https://issues.jboss.org/browse/ISPN-5329
Project: Infinispan
Issue Type: Enhancement
Components: Core
Affects Versions: 7.2.0.Beta1
Reporter: Radim Vansa
Assignee: Radim Vansa
Common cache operations generate a lot of temporary objects, which increases GC pressure. Let's try to analyze & reduce this.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5243) Configuration attribute holders
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5243?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5243:
-----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 7.2.0.Beta2
Resolution: Done
> Configuration attribute holders
> -------------------------------
>
> Key: ISPN-5243
> URL: https://issues.jboss.org/browse/ISPN-5243
> Project: Infinispan
> Issue Type: Enhancement
> Components: Configuration
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Fix For: 7.2.0.Beta2, 7.2.0.Final
>
>
> Configuration attributes are stored in simple variables, which makes it impossible to determine if they have been user-set or they have default values. The purpose of this issue is to introduce attribute wrappers which track modifications. Also we want to be able to react to changes by adding attribute listeners
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5208) Avoid invalid topology
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5208?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5208:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 7.2.0.Beta2
Resolution: Done
> Avoid invalid topology
> ----------------------
>
> Key: ISPN-5208
> URL: https://issues.jboss.org/browse/ISPN-5208
> Project: Infinispan
> Issue Type: Enhancement
> Components: Server
> Reporter: Takayoshi Kimura
> Assignee: Galder Zamarreño
> Labels: jdg641
> Fix For: 7.2.0.Beta2, 7.2.0.Final
>
> Attachments: ISPN_5208.java
>
>
> We've seen some invalid topology propagated to client and it causes ArrayIndexOutOfBoundsException:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.getServerByIndex(RoundRobinBalancingStrategy.java:68) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy.nextServer(RoundRobinBalancingStrategy.java:44) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.nextServer(TcpTransportFactory.java:220) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:194) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:27) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:535) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:635) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:616) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:527) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:523) [infinispan-client-hotrod-6.1.0.Final-redhat-4.jar:6.1.0.Final-redhat-4]
> {noformat}
> {noformat}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
> at org.infinispan.client.hotrod.impl.consistenthash.SegmentConsistentHash.getServer(SegmentConsistentHash.java:33)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:204)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:40)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:48)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:237)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at sample.Main.main(Main.java:16)
> {noformat}
> It happens on both Hot Rod 2 and 1.3 clients.
> It's really hard to reproduce this state and we don't have a consistent way to reproduce it. However when this happens there is always view change happening so it's related to view change.
> Judging from the stack trace, the client receives numOwners=0 or numSegments=0 topology from the server.
> Also we are unable to find to recover this situation. Rebooting random nodes don't help and keep getting this exceptions on client side.
> Until we can find the root cause, I think it's better to add a guard to avoid this kind invalid topology stored in the server side and propagated to the clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5324) Create extended RemoteCacheManager for testing purposes
by Martin Gencur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5324?page=com.atlassian.jira.plugin.... ]
Martin Gencur updated ISPN-5324:
--------------------------------
Description:
The extended RemoteCacheManager should expose details such as the following:
* transportFactory
The implementation should be internal and NOT part of public API.
This way we can get rid of reflection in tests for HotRod client and these tests can be reused for C++ and C# HotRod clients. Any tests that use reflection are currently unusable for those implementations.
The ultimate goal is to extend the test suite for C++ and C# clients.
was:
The extended RemoteCacheManager should expose details such as the following:
* transportFactory
* defaultCacheTopologyId
* connectionPool
* balancers
The implementation should be internal and NOT part of public API.
This way we can get rid of reflection in tests for HotRod client and these tests can be reused for C++ and C# HotRod clients. Any tests that use reflection are currently unusable for those implementations.
The ultimate goal is to extend the test suite for C++ and C# clients.
> Create extended RemoteCacheManager for testing purposes
> -------------------------------------------------------
>
> Key: ISPN-5324
> URL: https://issues.jboss.org/browse/ISPN-5324
> Project: Infinispan
> Issue Type: Feature Request
> Components: Test Suite - Core
> Reporter: Martin Gencur
> Assignee: Martin Gencur
>
> The extended RemoteCacheManager should expose details such as the following:
> * transportFactory
> The implementation should be internal and NOT part of public API.
> This way we can get rid of reflection in tests for HotRod client and these tests can be reused for C++ and C# HotRod clients. Any tests that use reflection are currently unusable for those implementations.
> The ultimate goal is to extend the test suite for C++ and C# clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5324) Create extended RemoteCacheManager for testing purposes
by Martin Gencur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5324?page=com.atlassian.jira.plugin.... ]
Martin Gencur updated ISPN-5324:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3338
> Create extended RemoteCacheManager for testing purposes
> -------------------------------------------------------
>
> Key: ISPN-5324
> URL: https://issues.jboss.org/browse/ISPN-5324
> Project: Infinispan
> Issue Type: Feature Request
> Components: Test Suite - Core
> Reporter: Martin Gencur
> Assignee: Martin Gencur
>
> The extended RemoteCacheManager should expose details such as the following:
> * transportFactory
> * defaultCacheTopologyId
> * connectionPool
> * balancers
> The implementation should be internal and NOT part of public API.
> This way we can get rid of reflection in tests for HotRod client and these tests can be reused for C++ and C# HotRod clients. Any tests that use reflection are currently unusable for those implementations.
> The ultimate goal is to extend the test suite for C++ and C# clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months
[JBoss JIRA] (ISPN-5324) Create extended RemoteCacheManager for testing purposes
by Martin Gencur (JIRA)
[ https://issues.jboss.org/browse/ISPN-5324?page=com.atlassian.jira.plugin.... ]
Martin Gencur updated ISPN-5324:
--------------------------------
Component/s: Test Suite - Core
> Create extended RemoteCacheManager for testing purposes
> -------------------------------------------------------
>
> Key: ISPN-5324
> URL: https://issues.jboss.org/browse/ISPN-5324
> Project: Infinispan
> Issue Type: Feature Request
> Components: Test Suite - Core
> Reporter: Martin Gencur
> Assignee: Martin Gencur
>
> The extended RemoteCacheManager should expose details such as the following:
> * transportFactory
> * defaultCacheTopologyId
> * connectionPool
> * balancers
> The implementation should be internal and NOT part of public API.
> This way we can get rid of reflection in tests for HotRod client and these tests can be reused for C++ and C# HotRod clients. Any tests that use reflection are currently unusable for those implementations.
> The ultimate goal is to extend the test suite for C++ and C# clients.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
10 years, 6 months