[JBoss JIRA] (ISPN-3744) TopologyAwareTwoNodesMapReduceTest && SimpleTwoNodesMapReduceTest random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3744?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3744:
-------------------------------
Priority: Blocker (was: Major)
Assignee: (was: Vladimir Blagojevic)
> TopologyAwareTwoNodesMapReduceTest && SimpleTwoNodesMapReduceTest random failures
> ---------------------------------------------------------------------------------
>
> Key: ISPN-3744
> URL: https://issues.jboss.org/browse/ISPN-3744
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce, Test Suite - Core
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
>
> The static counter used in FailAfterSecondCallReducer in TopologyAwareTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution and SimpleTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution has several problems:
> 1. Both tests use the same counter, if they run in parallel one could reset the counter for the other.
> 2. The counter update is not atomic, so the async thread that executes the reducer on the originator and the remote executor thread that executes the reducer on the remote node can both see value 0.
> 3. There is no guarantee that the reducer is executed on the originator first, so the exception isn't always thrown on the remote node.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
9 years, 4 months
[JBoss JIRA] (ISPN-4334) MapReduceTaskLifecycleService shouldn't keep the list of found lifecycle implementations
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4334?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-4334:
----------------------------------
Assignee: (was: Vladimir Blagojevic)
> MapReduceTaskLifecycleService shouldn't keep the list of found lifecycle implementations
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-4334
> URL: https://issues.jboss.org/browse/ISPN-4334
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce, Remote Querying
> Reporter: Jakub Markos
>
> The issue is that this class https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o... searches for the lifecycle classes using current threads classloader, and then caches the result, so the returned list of implementations depends only on the 1st thread creating the singleton instance of the service.
> You can replicate the bug from this branch:
> https://github.com/jmarkos/infinispan/tree/queries
> running
> {code}
> mvn clean verify -Dmaven.test.failure.ignore=true -DfailIfNoTests=false -U -Psuite.others -Dtest=RemoteQueryKeySetTest,ManualIndexingTest
> {code}
> from the server/integration/testsuite directory, results in an exception:
> {code}javax.management.MBeanException
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:271)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:791)
> at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527)
> at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263)
> at org.jboss.remotingjmx.protocol.v2.ServerProxy$InvokeHandler.handle(ServerProxy.java:915)
> at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:152)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.infinispan.jmx.ResourceDMBean.invoke(ResourceDMBean.java:269)
> ... 9 more
> Caused by: org.infinispan.commons.CacheException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeHelper(MapReduceTask.java:517)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:427)
> at org.infinispan.query.impl.massindex.MapReduceMassIndexer.start(MapReduceMassIndexer.java:25)
> ... 14 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:1059)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:677)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeHelper(MapReduceTask.java:510)
> ... 16 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.query.impl.massindex.IndexingMapper.map(IndexingMapper.java:38)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl$2.apply(MapReduceManagerImpl.java:207)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl$2.apply(MapReduceManagerImpl.java:202)
> at org.infinispan.container.DefaultDataContainer$1.apply(DefaultDataContainer.java:393)
> at org.infinispan.container.DefaultDataContainer$1.apply(DefaultDataContainer.java:389)
> at org.infinispan.commons.util.concurrent.jdk8backported.ConcurrentParallelHashMapV8$1.apply(ConcurrentParallelHashMapV8.java:48)
> at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ForEachMappingTask.compute(EquivalentConcurrentHashMapV8.java:4894)
> at org.infinispan.commons.util.concurrent.jdk8backported.CountedCompleter.exec(CountedCompleter.java:681)
> at org.infinispan.commons.util.concurrent.jdk8backported.ForkJoinTask.doExec(ForkJoinTask.java:264)
> at org.infinispan.commons.util.concurrent.jdk8backported.ForkJoinTask.doInvoke(ForkJoinTask.java:360)
> at org.infinispan.commons.util.concurrent.jdk8backported.ForkJoinTask.invoke(ForkJoinTask.java:692)
> at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8.forEach(EquivalentConcurrentHashMapV8.java:3592)
> at org.infinispan.commons.util.concurrent.jdk8backported.ConcurrentParallelHashMapV8.forEach(ConcurrentParallelHashMapV8.java:44)
> at org.infinispan.container.DefaultDataContainer.executeTask(DefaultDataContainer.java:389)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.map(MapReduceManagerImpl.java:202)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForLocalReduction(MapReduceManagerImpl.java:87)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocallyForLocalReduction(MapReduceTask.java:1173)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$300(MapReduceTask.java:1112)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$2.call(MapReduceTask.java:1144)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$2.call(MapReduceTask.java:1140)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> at org.jboss.threads.JBossThread.run(JBossThread.java:122)
> {code}
> If you instead use -Dtest=RemoteQueryKeySetTest,ManualIndexingggggggggggggTest (to change the order of the execution, junit probably orders it by length), it passes, because MapReduceTaskLifecycleService is created from a thread which classloader sees the query module and therefore can load this class
> https://github.com/infinispan/infinispan/blob/master/query/src/main/java/... which properly initializes IndexingMapper and avoids the NPE.
> Thanks to Adrian Nistor for his help.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
9 years, 4 months
[JBoss JIRA] (ISPN-4173) SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4173?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-4173:
----------------------------------
Assignee: (was: Vladimir Blagojevic)
> SuspectExceptions thrown during MapReduceTask while removing the intermediate cache
> -----------------------------------------------------------------------------------
>
> Key: ISPN-4173
> URL: https://issues.jboss.org/browse/ISPN-4173
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha2
> Reporter: Alan Field
>
> While running the Map/Reduce benchmark with multiple value sizes, I have been seeing this error in the logs from Infinispan 6 and 7:
> {noformat}
> 16:13:51,325 ERROR [org.radargun.stages.MapReduceStage] (pool-1-thread-1) executeMapReduceTask() returned an exception
> org.infinispan.commons.CacheException: Error removing cache
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:471)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:353)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:634)
> at org.radargun.cachewrappers.InfinispanMapReduce.executeMapReduceTask(InfinispanMapReduce.java:91)
> at org.radargun.cachewrappers.Infinispan51Wrapper.executeMapReduceTask(Infinispan51Wrapper.java:198)
> at org.radargun.stages.MapReduceStage.executeMapReduceTask(MapReduceStage.java:212)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:164)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from edg-perf06-46939, see cause for remote stack trace
> at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)
> at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)
> at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:463)
> ... 12 more
> Caused by: org.infinispan.commons.CacheException: Problems invoking command.
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221)
> at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
> at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)
> at org.jgroups.JChannel.up(JChannel.java:708)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)
> at org.jgroups.protocols.RSVP.up(RSVP.java:187)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)
> at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)
> at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)
> at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)
> at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)
> at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
> at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
> at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
> at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
> at org.jgroups.protocols.Discovery.up(Discovery.java:379)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1370)
> at org.jgroups.protocols.TP$MyHandler.run(TP.java:1556)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)
> at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)
> at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)
> at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)
> at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)
> at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
> at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)
> ... 25 more
> {noformat}
> These exceptions are happening during the execution of the MapReduceTask. I have also seen SuspectExceptions in these logs. This could be related to shutting down the intermediate cache (https://issues.jboss.org/browse/ISPN-4144), so I will check again once this is addressed. If this is the case the fix for ISPN-4144 will need to be fixed in both versions.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
9 years, 4 months
[JBoss JIRA] (ISPN-5033) Lack of EmbeddedCacheManager.undefineConfiguration(String) can cause memory/classloader leaks
by Paul Ferraro (JIRA)
Paul Ferraro created ISPN-5033:
----------------------------------
Summary: Lack of EmbeddedCacheManager.undefineConfiguration(String) can cause memory/classloader leaks
Key: ISPN-5033
URL: https://issues.jboss.org/browse/ISPN-5033
Project: Infinispan
Issue Type: Bug
Components: Configuration
Affects Versions: 7.1.0.Alpha1
Reporter: Paul Ferraro
EmbeddedCacheManager has defineConfiguration(...) methods for defining a new cache configuration, but has no equivalent "undefine" method. Consequently, once defined, a cache manager will retains a reference to the cache configuration object for the lifetime of the cache manager. While this is normally not a big deal, the configuration objects generated by WildFly contain references to the classloaders of foreign modules.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
9 years, 4 months