[JBoss JIRA] (ISPN-3613) Stored entries are deleted from table in rebalance
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3613?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-3613:
-------------------------------------
I am unable to reproduce this with the latest code base. Digging further it appears this was fixed by ISPN-2903.
Side note: the xml file provided is not accurate as the log shows L1 was enabled for the cache, but the xml didn't have it enabled.
But it looks like this is caused because eviction was enabled and a rehash causes old values to be evicted. When it was evicted it would notify the cache loader that it was removed with ISPN-2903 and would then remove it from the store as well.
I will add a test that confirms this.
Also I need to verify if when sharing is disabled if it properly evicts the entries from the cache store.
> Stored entries are deleted from table in rebalance
> --------------------------------------------------
>
> Key: ISPN-3613
> URL: https://issues.jboss.org/browse/ISPN-3613
> Project: Infinispan
> Issue Type: Bug
> Reporter: Mircea Markus
> Assignee: William Burns
> Labels: jdg62GAblocker
> Fix For: 6.0.0.CR2, 6.0.0.Final
>
>
> Description of problem:
> When passivation value is false, stored entries are deleted from table in rebalance.
> clustered.xml
> ------------
> <distributed-cache name="myCache" mode="SYNC" start="EAGER">
> <locking isolation="READ_COMMITTED" acquire-timeout="30000" concurrency-level="1000" striping="false"/>
> <transaction mode="NONE"/>
> <eviction strategy="LIRS" max-entries="10000"/>
> <string-keyed-jdbc-store datasource="java:jboss/datasources/InfinispanDS" passivation="false" preload="true" purge="false" shared="true" fetch-state="false">
> ...
> Version-Release number of selected component (if applicable):
> JDG 6.1
> How reproducible:
> I will attache the clustered.xml and trace logs.
> Steps to Reproduce:
> 1.start node1
> 2.put 300 entries
> 3.start node2
> check entries:
> select count(*) from table;
> 300
> 4.start node3
> check entries:
> select count(*) from table;
> 0
> Actual results:
> In step 4, number of entries are 0 in DB table.
> Expected results:
> In step 4, number of entries are 300 in DB table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3600) ignorePreviousValue should not be set on non-origin nodes
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3600?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3600:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1016104|https://bugzilla.redhat.com/show_bug.cgi?id=1016104] from MODIFIED to ON_QA
> ignorePreviousValue should not be set on non-origin nodes
> ---------------------------------------------------------
>
> Key: ISPN-3600
> URL: https://issues.jboss.org/browse/ISPN-3600
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: jdg62blocker
> Fix For: 6.0.0.Final
>
>
> In {{TxDistributionInterceptor.visit(Replace|Remove)Command}} the ignorePreviousValue flag is set to false on nodes which are not executing the transaction. Usually this does not matter, but when some other node requests transactions from this node (during ST), the Remove and Replace commands are sent with this flag set to false, which results in skipping their execution.
> Note: the overall behaviour with ignoring the command when this flag is false and origin is not local seems to me as abusing it - the semantics of the command is far from obvious. When the non-origin node receives the exact command that is executed on the original node (without ignorePreviousValue set to true), it simply ignores it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3443) WriteCommand may be ignored during state transfer
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3443?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3443:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1001634|https://bugzilla.redhat.com/show_bug.cgi?id=1001634] from MODIFIED to ON_QA
> WriteCommand may be ignored during state transfer
> -------------------------------------------------
>
> Key: ISPN-3443
> URL: https://issues.jboss.org/browse/ISPN-3443
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency, State transfer
> Affects Versions: 6.0.0.Alpha3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: jdg62blocker
> Fix For: 6.0.0.CR1
>
>
> Distributed sync non-tx cache.
> Situation:
> 1) A node is joining the cluster, requesting some segment
> 2) RemoveCommand is sent to backup owner with ignorePreviousValue=true
> 3) It looks up the entry and finds null
> 4) State transfer invokes the PutKeyValueCommand and sets the value for removed entry (updateKeys has not the key yet)
> 5) RemoveCommand adds its key to updateKeys set, but it does not remove the value as it is already null (in its context)
> Result: the value is removed on primary but on backup this is still present
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3443) WriteCommand may be ignored during state transfer
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3443?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3443:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1001634|https://bugzilla.redhat.com/show_bug.cgi?id=1001634] from POST to MODIFIED
> WriteCommand may be ignored during state transfer
> -------------------------------------------------
>
> Key: ISPN-3443
> URL: https://issues.jboss.org/browse/ISPN-3443
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency, State transfer
> Affects Versions: 6.0.0.Alpha3
> Reporter: Radim Vansa
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: jdg62blocker
> Fix For: 6.0.0.CR1
>
>
> Distributed sync non-tx cache.
> Situation:
> 1) A node is joining the cluster, requesting some segment
> 2) RemoveCommand is sent to backup owner with ignorePreviousValue=true
> 3) It looks up the entry and finds null
> 4) State transfer invokes the PutKeyValueCommand and sets the value for removed entry (updateKeys has not the key yet)
> 5) RemoveCommand adds its key to updateKeys set, but it does not remove the value as it is already null (in its context)
> Result: the value is removed on primary but on backup this is still present
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3600) ignorePreviousValue should not be set on non-origin nodes
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3600?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3600:
-----------------------------------------------
Tristan Tarrant <ttarrant(a)redhat.com> changed the Status of [bug 1016104|https://bugzilla.redhat.com/show_bug.cgi?id=1016104] from POST to MODIFIED
> ignorePreviousValue should not be set on non-origin nodes
> ---------------------------------------------------------
>
> Key: ISPN-3600
> URL: https://issues.jboss.org/browse/ISPN-3600
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer, Transactions
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: jdg62blocker
> Fix For: 6.0.0.Final
>
>
> In {{TxDistributionInterceptor.visit(Replace|Remove)Command}} the ignorePreviousValue flag is set to false on nodes which are not executing the transaction. Usually this does not matter, but when some other node requests transactions from this node (during ST), the Remove and Replace commands are sent with this flag set to false, which results in skipping their execution.
> Note: the overall behaviour with ignoring the command when this flag is false and origin is not local seems to me as abusing it - the semantics of the command is far from obvious. When the non-origin node receives the exact command that is executed on the original node (without ignorePreviousValue set to true), it simply ignores it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3375) Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-3375?page=com.atlassian.jira.plugin.... ]
Alan Field commented on ISPN-3375:
----------------------------------
[~dan.berindei] Yes, I missed the addition of the combiner in Infinispan 5.2. I just sent a PR to add support for it in RadarGun, so I'll be able to try to see how it affects the performance of this scenario. I am also running this Jenkins job with TRACE enabled for org.jgroups and org.infinispan.remoting:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/user/afield@REDHAT.COM/m...'s%20jobs/job/jdg-radargun-mapreduce-test/232/
> Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
> --------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3375
> URL: https://issues.jboss.org/browse/ISPN-3375
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.3.0.Final
> Reporter: Alan Field
> Assignee: Dan Berindei
> Priority: Critical
> Labels: jdg62GAblocker
> Fix For: 6.0.0.Final
>
> Attachments: jgroups-udp-mr.xml, stdout.zip
>
>
> The following exception occurs on the primary node running a Map/Reduce job with 1KB values:
> {noformat}
> 12:57:32,755 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:369)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:113)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:207)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:149)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:867)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:461)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:363)
> ... 10 more
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:340)
> ... 5 more
> Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> ... 3 more
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> And this exception is occurring on the other node in the cluster:
> {noformat}
> 12:57:32,484 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error
> org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> 12:57:32,494 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-0) Exception executing command
> org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> ISPN-2836 added a timeout for the entire MapReduceTask, but this timeout is happening when the intermediate keys are put into the cache by the map phase. This error does not occur when the value size is 8KB or larger.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3079) xsite replication custom failurePolicy does now work with ASYNC strategy
by dex chen (JIRA)
[ https://issues.jboss.org/browse/ISPN-3079?page=com.atlassian.jira.plugin.... ]
dex chen commented on ISPN-3079:
--------------------------------
Hi guys: this is really key feature we are waiting on. We already pilot the xsite feature. We loved the xsite feature, but we are blocked by this bug. Please help to fix it.
> xsite replication custom failurePolicy does now work with ASYNC strategy
> ------------------------------------------------------------------------
>
> Key: ISPN-3079
> URL: https://issues.jboss.org/browse/ISPN-3079
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Affects Versions: 5.2.5.Final
> Environment: ISPN 5.2.5.final
> Reporter: dex chen
> Assignee: Mircea Markus
>
> I configure xsite backupFailurePolicy="CUSTOM" and specified failurePolicyClass (see below xml configuration).
> When the strategy is set to "ASYNC", the failurePolicyClass code is never got invoked!!
> But, same code will will be invoked if the strategy is set to "SYNC".
> .....
> <sites>
> <backups>
> <backup site="NYC" strategy="ASYNC" backupFailurePolicy="CUSTOM" failurePolicyClass="com.test.cluster.infinispan.CrossSiteReplicationFailurePolicy" timeout="12003"/>
> </backups>
> </sites>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3079) xsite replication custom failurePolicy does not work with ASYNC strategy
by dex chen (JIRA)
[ https://issues.jboss.org/browse/ISPN-3079?page=com.atlassian.jira.plugin.... ]
dex chen updated ISPN-3079:
---------------------------
Summary: xsite replication custom failurePolicy does not work with ASYNC strategy (was: xsite replication custom failurePolicy does now work with ASYNC strategy)
> xsite replication custom failurePolicy does not work with ASYNC strategy
> ------------------------------------------------------------------------
>
> Key: ISPN-3079
> URL: https://issues.jboss.org/browse/ISPN-3079
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Affects Versions: 5.2.5.Final
> Environment: ISPN 5.2.5.final
> Reporter: dex chen
> Assignee: Mircea Markus
>
> I configure xsite backupFailurePolicy="CUSTOM" and specified failurePolicyClass (see below xml configuration).
> When the strategy is set to "ASYNC", the failurePolicyClass code is never got invoked!!
> But, same code will will be invoked if the strategy is set to "SYNC".
> .....
> <sites>
> <backups>
> <backup site="NYC" strategy="ASYNC" backupFailurePolicy="CUSTOM" failurePolicyClass="com.test.cluster.infinispan.CrossSiteReplicationFailurePolicy" timeout="12003"/>
> </backups>
> </sites>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3336) Extended Statistics check list
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3336?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3336:
--------------------------------
Fix Version/s: 6.0.0.CR2
6.0.0.Final
(was: 6.1.0.Final)
> Extended Statistics check list
> ------------------------------
>
> Key: ISPN-3336
> URL: https://issues.jboss.org/browse/ISPN-3336
> Project: Infinispan
> Issue Type: Task
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 6.0.0.CR2, 6.0.0.Final
>
>
> * add documentation
> * check for duplicate stats;
> * check for not exposed statistics
> * try to improve the code (reduce the number of access to the CHM)
> * add tests to check if they are exported via JMX correctly
> * failing test: OptimisticLockingTxClusterExtendedStatisticLogicTest.testWriteSkewOnNonOwner
> * failing test: PessimisticLockingTxClusterExtendedStatisticLogicTest.testDeadlockOnNonOwnerWithRemoteTx
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months
[JBoss JIRA] (ISPN-3336) Extended Statistics check list
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3336?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-3336:
-------------------------------------
[~pruivo] documentation first, the rest if the time allows...
> Extended Statistics check list
> ------------------------------
>
> Key: ISPN-3336
> URL: https://issues.jboss.org/browse/ISPN-3336
> Project: Infinispan
> Issue Type: Task
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 6.0.0.CR2, 6.0.0.Final
>
>
> * add documentation
> * check for duplicate stats;
> * check for not exposed statistics
> * try to improve the code (reduce the number of access to the CHM)
> * add tests to check if they are exported via JMX correctly
> * failing test: OptimisticLockingTxClusterExtendedStatisticLogicTest.testWriteSkewOnNonOwner
> * failing test: PessimisticLockingTxClusterExtendedStatisticLogicTest.testDeadlockOnNonOwnerWithRemoteTx
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 3 months