[JBoss JIRA] (ISPN-3603) Conditional command is committed on originator when it fails on primary owner
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3603?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3603:
-------------------------------
Summary: Conditional command is committed on originator when it fails on primary owner (was: Conditional command is committed when it fails on primary owner)
> Conditional command is committed on originator when it fails on primary owner
> -----------------------------------------------------------------------------
>
> Key: ISPN-3603
> URL: https://issues.jboss.org/browse/ISPN-3603
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 6.0.0.CR1
> Reporter: Radim Vansa
> Assignee: Mircea Markus
> Priority: Critical
> Labels: jdg62blocker
>
> In non-tx cache, when conditional command (e.g. ReplaceCommand) fails on primary owner because the old value is not equal to current value, the NonTxDistributionInterceptor.visitReplaceCommand returns false but the entry is committed in EntryWrappingInterceptor anyway.
> Speaking about EntryWrappingInterceptor.invokeNextAndApplyChanges: NonTxDistributionInterceptor does not mark the command as unsuccessful if it fails on the primary owner, therefore, checking for command.isSuccessful() in order to retry the command upon topology change may seem insufficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3404) ClusterCacheLoader - clusteredGetCommand response value is null
by Martin Gencur (JIRA)
[ https://issues.jboss.org/browse/ISPN-3404?page=com.atlassian.jira.plugin.... ]
Martin Gencur updated ISPN-3404:
--------------------------------
Affects Version/s: 6.0.0.Alpha2
> ClusterCacheLoader - clusteredGetCommand response value is null
> ---------------------------------------------------------------
>
> Key: ISPN-3404
> URL: https://issues.jboss.org/browse/ISPN-3404
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.0.Alpha2
> Reporter: Tomas Sykora
> Assignee: Mircea Markus
> Attachments: clusterCacheLoaderNullPointerResponseValueError.txt, server1DR2.log, server2DR2.log
>
>
> We found this issue in DR2 (Alpha2). In Alpha1 this was ok.
> Our internal test:
> {code:title=Bar.java|borderStyle=solid}
> @Test
> public void testLazyLoadingWhenStateTransferDisabled() throws Exception {
> controller.start(CONTAINER1);
> mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint()
> .getPort());
> mc.set("k1", "v1");
> mc.set("k2", "v2");
> assertEquals("v1", mc.get("k1"));
> assertEquals("v2", mc.get("k2"));
> assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.start(CONTAINER2);
> mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint()
> .getPort());
> assertEquals(2, server2.getDefaultCacheManager().getClusterSize());
> //state-transfer = false -> no entries in the newly joined node
> assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals("v1", mc2.get("k1")); //lazily load the entries
> assertEquals("v2", mc2.get("k2"));
> assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> mc2.set("k3", "v3"); // THIS IS FAILING HERE
> assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries());
> controller.kill(CONTAINER2);
> controller.kill(CONTAINER1);
> }
> {code}
> *mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.*
> You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....
> I also include trace logs for both memcached jdg servers if they can be useful.
> ---------------
> I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error.
> {code:title=Bar.java|borderStyle=solid}
> ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false);
> cfgBuilder.clustering().hash().numOwners(1);
> cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false);
> cfgBuilder.clustering().stateTransfer().timeout(20000);
> cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000);
> RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder();
> addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build());
> startServers();
> ......
> PutMethod put = new PutMethod(PATH1 + "key1");
> put.setRequestEntity(new StringRequestEntity("value1", "application/text", null));
> call(put);
> assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK);
> put.releaseConnection();
> {code}
> This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3375) Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-3375?page=com.atlassian.jira.plugin.... ]
Alan Field updated ISPN-3375:
-----------------------------
Attachment: jgroups-udp-mr.xml
Updated JGroups configuration file for M/R task
> Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
> --------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3375
> URL: https://issues.jboss.org/browse/ISPN-3375
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.3.0.Final
> Reporter: Alan Field
> Assignee: Dan Berindei
> Priority: Critical
> Labels: jdg62GAblocker
> Fix For: 6.0.0.Final
>
> Attachments: jgroups-udp-mr.xml, stdout.zip
>
>
> The following exception occurs on the primary node running a Map/Reduce job with 1KB values:
> {noformat}
> 12:57:32,755 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:369)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:113)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:207)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:149)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:867)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:461)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:363)
> ... 10 more
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:340)
> ... 5 more
> Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> ... 3 more
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> And this exception is occurring on the other node in the cluster:
> {noformat}
> 12:57:32,484 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error
> org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> 12:57:32,494 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-0) Exception executing command
> org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> ISPN-2836 added a timeout for the entire MapReduceTask, but this timeout is happening when the intermediate keys are put into the cache by the map phase. This error does not occur when the value size is 8KB or larger.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3375) Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-3375?page=com.atlassian.jira.plugin.... ]
Alan Field commented on ISPN-3375:
----------------------------------
[~dan.berindei] OK, I have rerun the tests with the suggested changes to the JGroups configuration, and the TimeoutException are still occurring. (With collator: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/user/afield@REDHAT.COM/m...'s%20jobs/job/jdg-radargun-mapreduce-test/230/ and without collator: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/user/afield@REDHAT.COM/m...'s%20jobs/job/jdg-radargun-mapreduce-test/229/) I can get another trace log, if you like.
> Map/Reduce jobs with small value sizes fail because of timeouts putting intermediate keys into the cache
> --------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3375
> URL: https://issues.jboss.org/browse/ISPN-3375
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.3.0.Final
> Reporter: Alan Field
> Assignee: Dan Berindei
> Priority: Critical
> Labels: jdg62GAblocker
> Fix For: 6.0.0.Final
>
> Attachments: stdout.zip
>
>
> The following exception occurs on the primary node running a Map/Reduce job with 1KB values:
> {noformat}
> 12:57:32,755 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.CacheException: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:369)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:113)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:207)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:149)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:867)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:461)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:363)
> ... 10 more
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf03-15313, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:70)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:384)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.remoting.rpc.RpcManagerImpl$2.call(RpcManagerImpl.java:340)
> ... 5 more
> Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> ... 3 more
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> And this exception is occurring on the other node in the cluster:
> {noformat}
> 12:57:32,484 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error
> org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> 12:57:32,494 ERROR [org.infinispan.remoting.InboundInvocationHandlerImpl] (remote-thread-0) Exception executing command
> org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:114)
> at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:93)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
> at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)
> at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task edc7a51c-d25a-4650-b39a-8d8a87d7cd61
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:331)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:112)
> ... 7 more
> Caused by: org.infinispan.util.concurrent.TimeoutException: Node edg-perf02-56077 timed out
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:531)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
> at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:151)
> at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:91)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:290)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:157)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:70)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1337)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:898)
> at org.infinispan.CacheImpl.put(CacheImpl.java:890)
> at org.infinispan.CacheImpl.put(CacheImpl.java:1390)
> at org.infinispan.CacheImpl.put(CacheImpl.java:229)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:326)
> ... 8 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-56077
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:375)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:189)
> ... 48 more
> {noformat}
> ISPN-2836 added a timeout for the entire MapReduceTask, but this timeout is happening when the intermediate keys are put into the cache by the map phase. This error does not occur when the value size is 8KB or larger.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3405) Entries are passivated with wrong ID in DB
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3405?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3405:
-----------------------------------------------
Vitalii Chepeliuk <vchepeli(a)redhat.com> changed the Status of [bug 996877|https://bugzilla.redhat.com/show_bug.cgi?id=996877] from ON_QA to VERIFIED
> Entries are passivated with wrong ID in DB
> ------------------------------------------
>
> Key: ISPN-3405
> URL: https://issues.jboss.org/browse/ISPN-3405
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.3.0.Final, 6.0.0.Alpha1
> Reporter: Vitalii Chepeliuk
> Assignee: Galder Zamarreño
> Fix For: 6.0.0.Alpha4, 6.0.0.CR1
>
>
> Entry passivation into DB, concretly this class LockSupportCacheStore and method store. Look at lines with "<<<" string for more info
> {code:title=LockSupportCacheStore.java|borderStyle=solid}
> @Override
> public final void store(InternalCacheEntry ed) throws CacheLoaderException {
> if (trace) {
> log.tracef("store(%s)", ed);
> }
> if (ed == null) {
> return;
> }
> if (ed.canExpire() && ed.isExpired(timeService.wallClockTime())) {
> if (containsKey(ed.getKey())) {
> if (trace) {
> log.tracef("Entry %s is expired! Removing!", ed);
> }
> remove(ed.getKey());
> } else {
> if (trace) {
> log.tracef("Entry %s is expired! Not doing anything.", ed);
> }
> }
> return;
> }
> L keyHashCode = getLockFromKey(ed.getKey()); <<< here key is generated like ed.getKey().hashCode() & 0xfffffc00;
> lockForWriting(keyHashCode);
> try {
> storeLockSafe(ed, keyHashCode); <<< here it should be stored into Bucket and then stored in DB
> } finally {
> unlock(keyHashCode);
> }
> if (trace) {
> log.tracef("exit store(%s)", ed);
> }
> }
> {code}
> When I use RemoteCacheManager and RemoteCache I am putting entries into cache
> {code:title=Test.java|borderStyle=solid}
> cache.put("key1", "v1");
> cache.put("key2", "v2");
> cache.put("key3", "v3");
> {code}
> Then 2 entries are passivated and stored in DB
> ||ID||DATA||TIMESTAMP||
> |183713792|0301fe032a01034c422b21033e286d7942657374506572736f6e616c4b657957686963684861734e657665724265656e426574746572420521033e02763203620003630000000000000002|-1|
> |23486464|0301fe032a01034c420721033e046b657931420521033e02763103620003630000000000000001|-1|
> IDs are generated from method above and
> {code:title=Test.java|borderStyle=solid}
> byte[] keyBytes = marshaller.objectToByteBuffer("key1"); <<< key is marshalled
> long keyID = ByteArrayEquivalence.INSTANCE.hashCode(keyBytes) & 0xfffffc00 //computation taken from BucketBasedCacheStore <<< this does not work for me
> {code}
> And next step I'd like to retrieve data from DB
> SELECT ID, DATA FROM JDBC_BINARY_DEFAULT WHERE ID=keyID
> But in method
> {code:title=BucketBasedCacheStore.java|borderStyle=solid}
> @Override
> public Integer getLockFromKey(Object key) {
> return key.hashCode() & 0xfffffc00; <<< here should be used Arrays.hashCode((byte[])key) & 0xfffffc00), if key is represented as byte array, or used ByteArrayEquivalence instead of simple byte array(byte[]) as argument
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3364) Tests from org.infinispan.atomic package fail with AssertionError
by Vitalii Chepeliuk (JIRA)
[ https://issues.jboss.org/browse/ISPN-3364?page=com.atlassian.jira.plugin.... ]
Vitalii Chepeliuk updated ISPN-3364:
------------------------------------
Environment: RHEL5_x86, RHEL5_x86_64, RHEL6_x86, RHEL5_x86_64 with IBM JDK7 (was: RHEL5_x86, IBMJDK1.7
RHEL5_x86_64, IBMJDK1.7
RHEL6_x86_64, IBMJDK1.7)
> Tests from org.infinispan.atomic package fail with AssertionError
> -----------------------------------------------------------------
>
> Key: ISPN-3364
> URL: https://issues.jboss.org/browse/ISPN-3364
> Project: Infinispan
> Issue Type: Bug
> Components: Core API
> Affects Versions: 6.0.0.Alpha1
> Environment: RHEL5_x86, RHEL5_x86_64, RHEL6_x86, RHEL5_x86_64 with IBM JDK7
> Reporter: Vitalii Chepeliuk
> Assignee: Mircea Markus
> Labels: testsuite_stability
> Attachments: LocalDeltaAwarePassivationTest.log.zip, ReplDeltaAwarePassivationTest.log.zip
>
>
> Following tests fail from org.infinispan.atomic package
> org.infinispan.atomic.LocalDeltaAwarePassivationTest.testAtomicMap
> org.infinispan.atomic.LocalDeltaAwarePassivationTest.testDeltaAware
> org.infinispan.atomic.LocalDeltaAwarePassivationTest.testFineGrainedAtomicMap
> org.infinispan.atomic.ReplDeltaAwarePassivationTest.testAtomicMap
> org.infinispan.atomic.ReplDeltaAwarePassivationTest.testAtomicMap2
> org.infinispan.atomic.ReplDeltaAwarePassivationTest.testDeltaAware
> org.infinispan.atomic.ReplDeltaAwarePassivationTest.testDeltaAware2
> org.infinispan.atomic.ReplDeltaAwarePassivationTest.testFineGrainedAtomicMap
> org.infinispan.atomic.ReplDeltaAwarePassivationTest.testFineGrainedAtomicMap2
> Add link on jenkins job https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/e...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3504) LevelDB Cache store in server doesn't keep data between restarts
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3504?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3504:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> made a comment on [bug 1007756|https://bugzilla.redhat.com/show_bug.cgi?id=1007756]
Verified for 6.2.0.ER2
> LevelDB Cache store in server doesn't keep data between restarts
> ----------------------------------------------------------------
>
> Key: ISPN-3504
> URL: https://issues.jboss.org/browse/ISPN-3504
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 6.0.0.Alpha4
> Reporter: Michal Linhard
> Assignee: Tristan Tarrant
> Labels: jdg62blocker
> Fix For: 6.0.0.Beta2
>
>
> I have config with passivation="false" purge="false"
> and expect data to survive restarts which doesn't happen.
> {code}
> <cache-container name="default" default-cache="default"
> listener-executor="infinispan-listener">
> <local-cache name="default" start="EAGER" batching="false">
> <locking isolation="REPEATABLE_READ" acquire-timeout="20000"
> concurrency-level="500" striping="false" />
> <transaction mode="NONE" />
> <leveldb-store path="temp-leveldb" block-size="1024"
> cache-size="50000" clear-threshold="100000" passivation="false"
> purge="false" >
> <expiration path="temp-leveldb-expired" queue-size="2000" />
> <compression type="SNAPPY" />
> <implementation type="JAVA" />
> </leveldb-store>
> </local-cache>
> <local-cache name="memcachedCache" start="EAGER" batching="false">
> <locking isolation="REPEATABLE_READ" acquire-timeout="20000"
> concurrency-level="500" striping="false" />
> <transaction mode="NONE" />
> </local-cache>
> </cache-container>
> {code}
> failing test:
> https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....;
> hb=8e1b8a10fed5de5bf3bb9ec6fbc46857e1666798
> for JDG 6.2.0.DR4
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months
[JBoss JIRA] (ISPN-3504) LevelDB Cache store in server doesn't keep data between restarts
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3504?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3504:
-----------------------------------------------
Michal Linhard <mlinhard(a)redhat.com> changed the Status of [bug 1007756|https://bugzilla.redhat.com/show_bug.cgi?id=1007756] from ON_QA to VERIFIED
> LevelDB Cache store in server doesn't keep data between restarts
> ----------------------------------------------------------------
>
> Key: ISPN-3504
> URL: https://issues.jboss.org/browse/ISPN-3504
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 6.0.0.Alpha4
> Reporter: Michal Linhard
> Assignee: Tristan Tarrant
> Labels: jdg62blocker
> Fix For: 6.0.0.Beta2
>
>
> I have config with passivation="false" purge="false"
> and expect data to survive restarts which doesn't happen.
> {code}
> <cache-container name="default" default-cache="default"
> listener-executor="infinispan-listener">
> <local-cache name="default" start="EAGER" batching="false">
> <locking isolation="REPEATABLE_READ" acquire-timeout="20000"
> concurrency-level="500" striping="false" />
> <transaction mode="NONE" />
> <leveldb-store path="temp-leveldb" block-size="1024"
> cache-size="50000" clear-threshold="100000" passivation="false"
> purge="false" >
> <expiration path="temp-leveldb-expired" queue-size="2000" />
> <compression type="SNAPPY" />
> <implementation type="JAVA" />
> </leveldb-store>
> </local-cache>
> <local-cache name="memcachedCache" start="EAGER" batching="false">
> <locking isolation="REPEATABLE_READ" acquire-timeout="20000"
> concurrency-level="500" striping="false" />
> <transaction mode="NONE" />
> </local-cache>
> </cache-container>
> {code}
> failing test:
> https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests....;
> hb=8e1b8a10fed5de5bf3bb9ec6fbc46857e1666798
> for JDG 6.2.0.DR4
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 6 months