[JBoss JIRA] (ISPN-4173) SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-4173?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-4173:
-------------------------------------
There is no fixed version for this as it wasn't fixed. Check out distributed streams at http://infinispan.org/docs/8.2.x/user_guide/user_guide.html#_streams
> SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
> -----------------------------------------------------------------------------------
>
> Key: ISPN-4173
> URL: https://issues.jboss.org/browse/ISPN-4173
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha2
> Reporter: Alan Field
>
> While running the Map/Reduce benchmark with multiple value sizes, I have been seeing this error in the logs from Infinispan 6 and 7:
> {noformat}
> 16:13:51,325 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.commons.CacheException: Error removing cache
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:471)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:353)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:634)
> at org.radargun.cachewrappers.InfinispanMapReduce.executeMapReduceTask(InfinispanMapReduce.java:91)
> at org.radargun.cachewrappers.Infinispan51Wrapper.executeMapReduceTask(Infinispan51Wrapper.java:198)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:212)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:164)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf06-46939, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:463)
> ... 12 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1370)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1556)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)
> ... 25 more
> {noformat}
> These exceptions are happening during the execution of the MapReduceTask. I have also seen SuspectExceptions in these logs. This could be related to shutting down the intermediate cache (https://issues.jboss.org/browse/ISPN-4144), so I will check again once this is addressed. If this is the case the fix for ISPN-4144 will need to be fixed in both versions.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
[JBoss JIRA] (ISPN-4173) SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
by Du Zheng (JIRA)
[ https://issues.jboss.org/browse/ISPN-4173?page=com.atlassian.jira.plugin.... ]
Du Zheng commented on ISPN-4173:
--------------------------------
Can you please update the fixed version?
> SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
> -----------------------------------------------------------------------------------
>
> Key: ISPN-4173
> URL: https://issues.jboss.org/browse/ISPN-4173
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha2
> Reporter: Alan Field
>
> While running the Map/Reduce benchmark with multiple value sizes, I have been seeing this error in the logs from Infinispan 6 and 7:
> {noformat}
> 16:13:51,325 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.commons.CacheException: Error removing cache
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:471)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:353)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:634)
> at org.radargun.cachewrappers.InfinispanMapReduce.executeMapReduceTask(InfinispanMapReduce.java:91)
> at org.radargun.cachewrappers.Infinispan51Wrapper.executeMapReduceTask(Infinispan51Wrapper.java:198)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:212)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:164)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf06-46939, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:463)
> ... 12 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1370)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1556)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)
> ... 25 more
> {noformat}
> These exceptions are happening during the execution of the MapReduceTask. I have also seen SuspectExceptions in these logs. This could be related to shutting down the intermediate cache (https://issues.jboss.org/browse/ISPN-4144), so I will check again once this is addressed. If this is the case the fix for ISPN-4144 will need to be fixed in both versions.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months
[JBoss JIRA] (ISPN-4173) SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
by Du Zheng (JIRA)
[ https://issues.jboss.org/browse/ISPN-4173?page=com.atlassian.jira.plugin.... ]
Du Zheng edited comment on ISPN-4173 at 4/5/16 3:33 PM:
--------------------------------------------------------
Can you please update the Fix version?
was (Author: zsuzhengdu):
Can you please update the fixed version?
> SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
> -----------------------------------------------------------------------------------
>
> Key: ISPN-4173
> URL: https://issues.jboss.org/browse/ISPN-4173
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha2
> Reporter: Alan Field
>
> While running the Map/Reduce benchmark with multiple value sizes, I have been seeing this error in the logs from Infinispan 6 and 7:
> {noformat}
> 16:13:51,325 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.commons.CacheException: Error removing cache
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:471)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:353)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:634)
> at org.radargun.cachewrappers.InfinispanMapReduce.executeMapReduceTask(InfinispanMapReduce.java:91)
> at org.radargun.cachewrappers.Infinispan51Wrapper.executeMapReduceTask(Infinispan51Wrapper.java:198)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:212)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:164)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf06-46939, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:463)
> ... 12 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1370)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1556)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)
> ... 25 more
> {noformat}
> These exceptions are happening during the execution of the MapReduceTask. I have also seen SuspectExceptions in these logs. This could be related to shutting down the intermediate cache (https://issues.jboss.org/browse/ISPN-4144), so I will check again once this is addressed. If this is the case the fix for ISPN-4144 will need to be fixed in both versions.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 8 months