March 2016 - infinispan-issues - Jboss List Archives

[JBoss JIRA] (ISPN-2991) Management of DistExec / Map Reduce jobs

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-2991?page=com.atlassian.jira.plugin.... ] William Burns commented on ISPN-2991: ------------------------------------- We will want this for distributed streams. > Management of DistExec / Map Reduce jobs > ---------------------------------------- > > Key: ISPN-2991 > URL: https://issues.jboss.org/browse/ISPN-2991 > Project: Infinispan > Issue Type: Feature Request > Components: Distributed Execution and Map/Reduce, JMX, reporting and management > Reporter: NadirX > > We should expose a management interface for both DistExec and Map/reduce jobs so that at the very least we can: > - list running jobs with starting time and duration > - list recently terminated jobs > - allow cancelling running jobs -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-3162) implement a spliterator like interface for parallel traversal of local and replicated cache

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-3162?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-3162. --------------------------------- Resolution: Duplicate This is handled by distributed streams now. > implement a spliterator like interface for parallel traversal of local and replicated cache > ------------------------------------------------------------------------------------------- > > Key: ISPN-3162 > URL: https://issues.jboss.org/browse/ISPN-3162 > Project: Infinispan > Issue Type: Feature Request > Components: Core, Distributed Execution and Map/Reduce, Loaders and Stores > Affects Versions: 5.3.0.CR1 > Reporter: Mathieu Lachance > > The backport of ConcurrentHashMapV8 comes with an interesting interface, Spliterator and it's 3 implementations, Key/Value/EntryIterator. > As java 7 is now being more widely adopted, it could be interesting to offer to take advantage of the java ForkJoinPool for the parallel traversal of local and replicated caches where the Infinispan map reduce framework is maybe less suited for. > At term, it could be used to speed up the search in a cache without the need to index it. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-3386) Cancel Map/Reduce tasks before they cause OOM errors

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-3386?page=com.atlassian.jira.plugin.... ] William Burns commented on ISPN-3386: ------------------------------------- This could also be an issue for distributed streams. > Cancel Map/Reduce tasks before they cause OOM errors > ---------------------------------------------------- > > Key: ISPN-3386 > URL: https://issues.jboss.org/browse/ISPN-3386 > Project: Infinispan > Issue Type: Enhancement > Components: Distributed Execution and Map/Reduce > Affects Versions: 5.3.0.Final > Reporter: Alan Field > > Currently, a MapReduceTask that generates an OutOfMemory error will break a clustered cache. The node where the OOM occurs will generate SuspectExceptions on the other nodes in the cluster. > In order to avoid this situation, Infinispan should monitor free memory and cancel the MapReduceTask if free memory is below a user definable threshold. [~pruivo] has also suggested allowing the API user to set the priority of tasks, so that the Map Reduce Manager can cancel lower priority tasks first. This functional could also be included with ISPN-2991. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-3999) Invoke reducer immediatelly as intermediate VOut values are being collected

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-3999?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-3999. --------------------------------- Resolution: Out of Date Map/Reduce has been removed > Invoke reducer immediatelly as intermediate VOut values are being collected > --------------------------------------------------------------------------- > > Key: ISPN-3999 > URL: https://issues.jboss.org/browse/ISPN-3999 > Project: Infinispan > Issue Type: Enhancement > Components: Core, Distributed Execution and Map/Reduce > Reporter: Vladimir Blagojevic > Priority: Minor > > Currently in intermediate phase [1] as values for every KOut are being collected/grouped in the cluster we simply add every VOut to the list of values keyed by KOut. In reducer phase we then send a reducer function that gets invoked on List<VOut> and reduces those values into single VOut. However, this could be suboptimal as we can invoke reducer function as every VOut gets grouped rather than wait for all VOut values to be collected and then invoke reducers. However, reduce function could potentially take a lot of time to execute even for one value thus causing prolonged locking and congestions. We should implement this enhancement and do thorough performance measurements. This feature should be have an API setting in MapReduceTask. > Sanne Grinovero gets the credit for this idea. > [1] http://blog.infinispan.org/2012/07/mapreduce-improvements-in-infinispan.html -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-3992) ReplSyncDistributedExecutorTest.testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout fails randomly

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-3992?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-3992. --------------------------------- Resolution: Out of Date Map/Reduce has been removed > ReplSyncDistributedExecutorTest.testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout fails randomly > ---------------------------------------------------------------------------------------------------------------------- > > Key: ISPN-3992 > URL: https://issues.jboss.org/browse/ISPN-3992 > Project: Infinispan > Issue Type: Bug > Components: Distributed Execution and Map/Reduce, Test Suite - Core > Affects Versions: 6.0.1.Final > Reporter: Pedro Ruivo > Labels: testsuite_stability > > org.infinispan.distexec.ReplSyncDistributedExecutorTest.testBasicTargetRemoteDistributedCallableWithHighFutureAndLowTaskTimeout > http://ci.infinispan.org/viewLog.html?buildId=6518&tab=buildResultsDiv&bu... -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-3984) CompatModeDistributedTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-3984?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-3984. --------------------------------- Resolution: Out of Date Map/Reduce has been removed > CompatModeDistributedTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures > --------------------------------------------------------------------------------------- > > Key: ISPN-3984 > URL: https://issues.jboss.org/browse/ISPN-3984 > Project: Infinispan > Issue Type: Bug > Components: Core, Distributed Execution and Map/Reduce, Test Suite - Core > Affects Versions: 6.0.1.Final > Reporter: Dan Berindei > Labels: testsuite_stability > Attachments: cmdtnmrt.log.gz > > > The random failures in CompatModeDistributedTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys seem to be caused by state transfer being disabled in the test's caches. > {noformat} > 12:11:51,674 ERROR (testng-CompatModeDistributedTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.CompatModeDistributedTwoNodesMapReduceTest) failed. > java.lang.AssertionError: key 'well' does not have count 1 but 2 > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.verifyResults(BaseWordCountMapReduceTest.java:224) > at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.verifyResults(BaseWordCountMapReduceTest.java:334) > at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:162) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-3744) TopologyAwareTwoNodesMapReduceTest && SimpleTwoNodesMapReduceTest random failures

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-3744?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-3744. --------------------------------- Resolution: Out of Date Map/Reduce has been removed > TopologyAwareTwoNodesMapReduceTest && SimpleTwoNodesMapReduceTest random failures > --------------------------------------------------------------------------------- > > Key: ISPN-3744 > URL: https://issues.jboss.org/browse/ISPN-3744 > Project: Infinispan > Issue Type: Bug > Components: Distributed Execution and Map/Reduce, Test Suite - Core > Affects Versions: 6.0.0.Final > Reporter: Dan Berindei > Priority: Blocker > Labels: testsuite_stability > > The static counter used in FailAfterSecondCallReducer in TopologyAwareTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution and SimpleTwoNodesMapReduceTest.testInvokeMapWithReduceExceptionPhaseInRemoteExecution has several problems: > 1. Both tests use the same counter, if they run in parallel one could reset the counter for the other. > 2. The counter update is not atomic, so the async thread that executes the reducer on the originator and the remote executor thread that executes the reducer on the remote node can both see value 0. > 3. There is no guarantee that the reducer is executed on the originator first, so the exception isn't always thrown on the remote node. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-4019) Write a test with a node joining/leaving during a map/reduce task

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-4019?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-4019. --------------------------------- Resolution: Out of Date Map/Reduce has been removed > Write a test with a node joining/leaving during a map/reduce task > ----------------------------------------------------------------- > > Key: ISPN-4019 > URL: https://issues.jboss.org/browse/ISPN-4019 > Project: Infinispan > Issue Type: Task > Components: Distributed Execution and Map/Reduce, Test Suite - Core > Reporter: Dan Berindei > Assignee: Vladimir Blagojevic > > We should have a test to confirm that the M/R results do not change if a node joins/leaves during each of the map/combine/distributed reduce phases. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-4003) Test Map/Reduce in compatibility mode

by William Burns (JIRA)

[ https://issues.jboss.org/browse/ISPN-4003?page=com.atlassian.jira.plugin.... ] William Burns resolved ISPN-4003. --------------------------------- Resolution: Out of Date Map/Reduce has been removed > Test Map/Reduce in compatibility mode > ------------------------------------- > > Key: ISPN-4003 > URL: https://issues.jboss.org/browse/ISPN-4003 > Project: Infinispan > Issue Type: Feature Request > Components: Distributed Execution and Map/Reduce > Reporter: Mircea Markus > > Once ISPN-4002 is in place, we need a test that: > - starts a cluster in compatibility mode > - populates cache A with Person objects (protobuf) through the java hotrod client > - on the server a Map/Reduce task is run on the Person cache, which filters out all the persons with a given name and, for each matching person, it outputs something like (personId, age) into a result cache (as per ISPN-4002) > - the client reads the result cache and validates that it is correct -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

[JBoss JIRA] (ISPN-6406) NullPointerException while executed javascript returns null to js-client

by Anna Manukyan (JIRA)

Anna Manukyan created ISPN-6406: ----------------------------------- Summary: NullPointerException while executed javascript returns null to js-client Key: ISPN-6406 URL: https://issues.jboss.org/browse/ISPN-6406 Project: Infinispan Issue Type: Bug Components: Remote Protocols Reporter: Anna Manukyan Assignee: Galder Zamarreño In case when executing a javascript on local node, which returns as a result null, the following exception is thrown on the server: {code} 17:25:26,127 ERROR [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-7-119) ISPN005022: Exception writing response with messageId=188: java.lang.NullPointerException at org.infinispan.server.core.transport.ExtendedByteBuf$.writeRangedBytes(ExtendedByteBuf.scala:65) at org.infinispan.server.hotrod.Encoder2x$.writeResponse(Encoder2x.scala:340) at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:45) at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:691) at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:681) at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:716) at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:954) at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:243) at org.infinispan.server.hotrod.HotRodDecoder.writeResponse(HotRodDecoder.scala:250) at org.infinispan.server.hotrod.HotRodDecoder.customDecodeHeader(HotRodDecoder.scala:209) at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeHeader(HotRodDecoder.scala:97) at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:52) at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:224) at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:50) at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:370) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168) at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:32) at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32) at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:32) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:745) {code} The code is: {code} //Executable javascript: // mode=local,language=javascript,datatype='text/plain; charset=utf-8' cacheManager.getAddress() And the Test: it('can execute a script remotely to get node address from cacheManager', function(done) { Promise.all([client, readFile('spec/utils/test-cacheManager.js')]) .then(function(vals) { var c = vals[0]; return c.addScript('test-cacheManager.js', vals[1].toString()) .then(function() { return c; } ); }) .then(t.assert(t.exec('test-cacheManager.js'), t.toBeUndefined)) .catch(failed(done)).finally(done); }); {code} The result of the test execution is: {code} Message: java.lang.NullPointerException Stacktrace: undefined {code} -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

infinispan-issues March 2016