[JBoss JIRA] (ISPN-3617) Inconsistent L1 in non-tx distributed cache
by Radim Vansa (JIRA)
Radim Vansa created ISPN-3617:
---------------------------------
Summary: Inconsistent L1 in non-tx distributed cache
Key: ISPN-3617
URL: https://issues.jboss.org/browse/ISPN-3617
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 6.0.0.CR1
Reporter: Radim Vansa
Assignee: William Burns
When the change is replicated to backup owner, it sends the InvalidateL1Command to backup owners before committing the entry in EntryWrappingInterceptor (it performs the WriteCommand in parallel with sending the invalidation commmand, but then it waits until the invalidation request gets acked. If a GET is executed between the invalidation and committing the entry, the response contains outdated result and the L1 will not be invalidated until next write operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-3604) Provide JMX operation which shows the numberOfEntries in the entire dist-cache
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3604?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-3604:
-------------------------------------
makes sense to me. the M/R code from the server should be reused for this operation as well.
> Provide JMX operation which shows the numberOfEntries in the entire dist-cache
> ------------------------------------------------------------------------------
>
> Key: ISPN-3604
> URL: https://issues.jboss.org/browse/ISPN-3604
> Project: Infinispan
> Issue Type: Feature Request
> Components: JMX, reporting and management
> Affects Versions: 6.0.0.CR1
> Environment: Max OS-X 10.8.5, Oracle Hotspot 1.7.0_40, Infinispan 5.2.4.Final
> Reporter: Duncan Doyle
> Assignee: Mircea Markus
>
> The Infinispan statistics component only shows the 'numberOfEntries' in a particular node, including entries of which the node is not the primary owner (basically, doing a sum of this value of all the cluster members gives you (numberOfEntries * numOwners). The 'cache.keySet()' method does return the entire keyset, and thus you can calculate the numberOfEntries in the cache, but this requires writing custom code in a cache client. It would be nice if Infinispan would provide a JMX operation which is able to show the number of entries in the entire dist-cache cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-3604) Provide JMX operation which shows the numberOfEntries in the entire dist-cache
by Duncan Doyle (JIRA)
[ https://issues.jboss.org/browse/ISPN-3604?page=com.atlassian.jira.plugin.... ]
Duncan Doyle commented on ISPN-3604:
------------------------------------
Also, cache.keySet() only returns the entire keySet when using the HotRod client. When using an embedded cache, it only returns the entries in memory of that specific node. Furthermore, it does not return the keys of the entries stored in the CacheStore.
So, it seems we need to implement the JMX operation that same way as HotRod does it: use the ISPN Map/Reduce functionality.
> Provide JMX operation which shows the numberOfEntries in the entire dist-cache
> ------------------------------------------------------------------------------
>
> Key: ISPN-3604
> URL: https://issues.jboss.org/browse/ISPN-3604
> Project: Infinispan
> Issue Type: Feature Request
> Components: JMX, reporting and management
> Affects Versions: 6.0.0.CR1
> Environment: Max OS-X 10.8.5, Oracle Hotspot 1.7.0_40, Infinispan 5.2.4.Final
> Reporter: Duncan Doyle
> Assignee: Mircea Markus
>
> The Infinispan statistics component only shows the 'numberOfEntries' in a particular node, including entries of which the node is not the primary owner (basically, doing a sum of this value of all the cluster members gives you (numberOfEntries * numOwners). The 'cache.keySet()' method does return the entire keyset, and thus you can calculate the numberOfEntries in the cache, but this requires writing custom code in a cache client. It would be nice if Infinispan would provide a JMX operation which is able to show the number of entries in the entire dist-cache cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-3375) Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-3375?page=com.atlassian.jira.plugin.... ]
Alan Field commented on ISPN-3375:
----------------------------------
[~dan.berindei] Yeah, I could try a run with the previous log4j settings, but I have another bug I need to investigate in the short term.
> Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
> --------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3375
> URL: https://issues.jboss.org/browse/ISPN-3375
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.3.0.Final
> Reporter: Alan Field
> Assignee: Dan Berindei
> Priority: Critical
> Labels: jdg62GAblocker
> Fix For: 6.0.0.Final
>
> Attachments: jgroups-udp-mr.xml, stdout.zip
>
>
> The following exception occurs on the primary node running a Map/Reduce job with 1KB values:
> {noformat}
> 12:57:32,755 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:369)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:113)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:207)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:149)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:867)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:461)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:363)
> ... 10 more
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:340)
> ... 5 more
> Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> ... 3 more
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> And this exception is occurring on the other node in the cluster:
> {noformat}
> 12:57:32,484 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error
> org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> 12:57:32,494 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-0) Exception executing command
> org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> ISPN-2836 added a timeout for the entire MapReduceTask, but this timeout is happening when the intermediate keys are put into the cache by the map phase. This error does not occur when the value size is 8KB or larger.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-2478) PutMapCommand not correctly removing previous types from indexes
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-2478?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-2478:
---------------------------------------
Sorry for me to not be clear, I think I rushed this description as part of an overall design review, and the case isn't entirely clear to me either. Adrian it seems you have a test now? Let me know if you'd like some more brainstorming together about this.
We mentioned this briefly on our Query triage phone call, AFAIR we said that if no better solution could be found, a valid fix (although ugly) would be to convert the put map in a sequence of simple put operations.
As further consideration, this problem might be resolved at least partially by ISPN-2143 : the tricky aspect is that we need to do cleanup on indexes for the case of entries which are overwritten. An overwritten entry means that an entry could be replaced by a non-indexed type, while the removed type is indexed but potentially using a different index or different options so it's possible the the key of the two entries - while logically the same in terms of hashmap - isn't matching their indexed counterparts.
I concede the problem is probably far fetched, and it feels like it takes to apply multiple bad coding practices to trigger the scenario.. this leads me to think that I would consider it acceptable if we can detect the tricky mapping (from annotations?) and log a warning about the problem, and document it as an unsupported scenario. If you have a test it would be a great help to document it, as it's quite unclear to me as well how it would behave.
By bad coding practices I specifically mean that a Value is being replaced by a Value of a different type. Different types being stored under a same-typed index seems quite fishy to me.
> PutMapCommand not correctly removing previous types from indexes
> ----------------------------------------------------------------
>
> Key: ISPN-2478
> URL: https://issues.jboss.org/browse/ISPN-2478
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Affects Versions: 5.1.8.Final
> Reporter: Sanne Grinovero
> Assignee: Adrian Nistor
> Labels: stable_embedded_query
> Fix For: 6.0.0.Final
>
>
> We need to perform certain cleanup operations - potentially - on the return types of commands such as put and remove. In case of a putAll command, we're ignoring this requirement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-2478) PutMapCommand not correctly removing previous types from indexes
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2478?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2478:
--------------------------------
Summary: PutMapCommand not correctly removing previous types from indexes (was: PutMapCommand not correctly removing previous values from indexes)
Description: We need to perform certain cleanup operations - potentially - on the return types of commands such as put and remove. In case of a putAll command, we're ignoring this requirement. (was: We need to perform certain cleanup operations - potentially - on the return values of commands such as put and remove. In case of a putAll command, we're ignoring this requirement.)
> PutMapCommand not correctly removing previous types from indexes
> ----------------------------------------------------------------
>
> Key: ISPN-2478
> URL: https://issues.jboss.org/browse/ISPN-2478
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Affects Versions: 5.1.8.Final
> Reporter: Sanne Grinovero
> Assignee: Adrian Nistor
> Labels: stable_embedded_query
> Fix For: 6.0.0.Final
>
>
> We need to perform certain cleanup operations - potentially - on the return types of commands such as put and remove. In case of a putAll command, we're ignoring this requirement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-2478) PutMapCommand not correctly removing previous values from indexes
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2478?page=com.atlassian.jira.plugin.... ]
Adrian Nistor commented on ISPN-2478:
-------------------------------------
[~sannegrinovero] The description is not clear and I was a bit puzzled too, I also thought you intended to say values not types. I even started to write a test to prove the value theory, which amazingly did not fail and then it hit me and understood what you meant about types :) If the class of the new vs old value changes this is not also reflected in the index, in case of putAll. The other document fields seem to be updated but not the type of the entity. This leads to very very strange results in my test. [~dan.berindei] I'll put the old description back.
> PutMapCommand not correctly removing previous values from indexes
> -----------------------------------------------------------------
>
> Key: ISPN-2478
> URL: https://issues.jboss.org/browse/ISPN-2478
> Project: Infinispan
> Issue Type: Bug
> Components: Querying
> Affects Versions: 5.1.8.Final
> Reporter: Sanne Grinovero
> Assignee: Adrian Nistor
> Labels: stable_embedded_query
> Fix For: 6.0.0.Final
>
>
> We need to perform certain cleanup operations - potentially - on the return values of commands such as put and remove. In case of a putAll command, we're ignoring this requirement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-3443) WriteCommand may be ignored during state transfer
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-3443?page=com.atlassian.jira.plugin.... ]
Radim Vansa reopened ISPN-3443:
-------------------------------
[~dan.berindei]: Regrettably, the fix is not correct. You have to atomically check and ADD the key to updated keys set. Otherwise, a situation when both transaction and state transfer commit the entry (in this order) is possible, resulting with outdated entry on the node.
> WriteCommand may be ignored during state transfer
> -------------------------------------------------
>
> Key: ISPN-3443
> URL: https://issues.jboss.org/browse/ISPN-3443
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency, State transfer
> Affects Versions: 6.0.0.Alpha3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: jdg62blocker
> Fix For: 6.0.0.CR1
>
>
> Distributed sync non-tx cache.
> Situation:
> 1) A node is joining the cluster, requesting some segment
> 2) RemoveCommand is sent to backup owner with ignorePreviousValue=true
> 3) It looks up the entry and finds null
> 4) State transfer invokes the PutKeyValueCommand and sets the value for removed entry (updateKeys has not the key yet)
> 5) RemoveCommand adds its key to updateKeys set, but it does not remove the value as it is already null (in its context)
> Result: the value is removed on primary but on backup this is still present
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-3599) CommitCommand with replayed PrepareCommand executes rollback and then commit
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-3599?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-3599:
-----------------------------------
Good catch [~rvansa]
> CommitCommand with replayed PrepareCommand executes rollback and then commit
> ----------------------------------------------------------------------------
>
> Key: ISPN-3599
> URL: https://issues.jboss.org/browse/ISPN-3599
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: jdg62GAblocker
> Fix For: 6.0.0.CR2, 6.0.0.Final
>
>
> During state-transfer in tx cache, the node can receive {{CommitCommand}} from other node. After the node gets transaction data for affected segments, it creates the transaction with {{missingLookedUpEntries=true}} and the {{CommitCommand}} can be executed.
> In this command's {{perform(...)}} the transaction is *first* marked as completed, then it enters the interceptor chain. There, the {{PrepareCommand}} is created in {{StateTransferInterceptor.visitCommitCommand}} but after this is processed the {{TxInterceptor}} finds out that the transaction is already completed and executes {{RollbackCommand}}, clearing locks etc.
> Nevertheless, {{StateTransferInterceptor}} executes the initial {{CommitCommand}} afterwards. I suspect that this may be executed without the locks held.
> Anyway, it is not correct to execute both commit and rollback on the same transaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months
[JBoss JIRA] (ISPN-3599) CommitCommand with replayed PrepareCommand executes rollback and then commit
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-3599?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo updated ISPN-3599:
------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/2155
> CommitCommand with replayed PrepareCommand executes rollback and then commit
> ----------------------------------------------------------------------------
>
> Key: ISPN-3599
> URL: https://issues.jboss.org/browse/ISPN-3599
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: jdg62GAblocker
> Fix For: 6.0.0.CR2, 6.0.0.Final
>
>
> During state-transfer in tx cache, the node can receive {{CommitCommand}} from other node. After the node gets transaction data for affected segments, it creates the transaction with {{missingLookedUpEntries=true}} and the {{CommitCommand}} can be executed.
> In this command's {{perform(...)}} the transaction is *first* marked as completed, then it enters the interceptor chain. There, the {{PrepareCommand}} is created in {{StateTransferInterceptor.visitCommitCommand}} but after this is processed the {{TxInterceptor}} finds out that the transaction is already completed and executes {{RollbackCommand}}, clearing locks etc.
> Nevertheless, {{StateTransferInterceptor}} executes the initial {{CommitCommand}} afterwards. I suspect that this may be executed without the locks held.
> Anyway, it is not correct to execute both commit and rollback on the same transaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 3 months