March 2016 - infinispan-issues - Jboss List Archives

[JBoss JIRA] (ISPN-5179) Add distributed execution and map/reduce job statistics

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5179?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5179: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > Add distributed execution and map/reduce job statistics > -------------------------------------------------------- > > Key: ISPN-5179 > URL: https://issues.jboss.org/browse/ISPN-5179 > Project: Infinispan > Issue Type: Feature Request > Components: JMX, reporting and management > Reporter: Vladimir Blagojevic > Assignee: Vladimir Blagojevic > Fix For: 9.0.0.Alpha1 > > > We should add DMR/JMX statistics for the running distributed execution jobs as well as map/reduce jobs. The statistics will also include overview/total system statistics of previously executed jobs; we might store statistics of individual executed jobs in some internal cache. However, the primary objective is to calculate and maintain dist.exec and map/reduce job statistics for Infinispan admin console. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5151) DistributedSharedCacheTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5151?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5151: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > DistributedSharedCacheTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures > ---------------------------------------------------------------------------------------- > > Key: ISPN-5151 > URL: https://issues.jboss.org/browse/ISPN-5151 > Project: Infinispan > Issue Type: Bug > Components: Core, Test Suite - Core > Affects Versions: 7.0.3.Final > Reporter: Dan Berindei > Assignee: Dan Berindei > Priority: Blocker > Labels: testsuite_stability > Fix For: 9.0.0.Alpha1 > > > The method {{invokeMapReduce()}} doesn't really invoke the M/R task, it only creates it, and the execution only starts when the test method calls {{task.execute()}} explicitly. It shouldn't try to check the contents of the shared intermediary cache, because the intermediary cache may not exist yet - and it may accidentally create it with the wrong configuration. I get this error when I run only the {{testInvokeMapReduceOnAllKeys}} method: > {noformat} > 09:55:37,632 TRACE (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [DefaultCacheManager] About to wire and start cache __tmpMapReduce > 09:55:37,646 DEBUG (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [MapReduceTask] Invoking CreateCacheCommand{cacheManager=null, cacheNameToCreate='__tmpMapReduce', cacheConfigurationName='__tmpMapReduce', start=true', size=2} across members [DistributedSharedCacheTwoNodesMapReduceTest-NodeA-19271, DistributedSharedCacheTwoNodesMapReduceTest-NodeB-10341] > 10:32:56,324 ERROR (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest) failed. > org.infinispan.distexec.mapreduce.MapReduceException: Map phase failed > at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:607) > at org.infinispan.distexec.mapreduce.MapReduceTask.executeHelper(MapReduceTask.java:473) > at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:414) > at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:162) > Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException > at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:105) > at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocally(MapReduceTask.java:1174) > at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$300(MapReduceTask.java:1101) > at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:1123) > at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:1119) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapKeysToNodes(MapReduceManagerImpl.java:363) > at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.migrateIntermediateKeysAndValues(MapReduceManagerImpl.java:327) > at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombine(MapReduceManagerImpl.java:260) > at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:103) > ... 10 more > {noformat} > Even if the check is moved after the M/R task is finished, it still wouldn't be correct, because the task only cleans up the shared intermediary cache asynchronously. So it needs to use {{eventually()}} to avoid errors like this: > {noformat} > 04:06:32,260 ERROR (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest) failed. > java.lang.AssertionError: Shared cache __tmpMapReduce is not empty. It has 5 keys/values: [ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=is], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@21ae10d3}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=JUDCon], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@108d6b51}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=cool], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@77949e8f}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=Infinispan], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@712a6071}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=community], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@291bdf76}] expected:<0> but was:<5> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest.invokeMapReduce(DistributedSharedCacheTwoNodesMapReduceTest.java:44) > at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:161) > 04:06:32,579 TRACE (transport-thread-NodeA-p29577-t6:) [InvocationContextInterceptor] Invoked with command RemoveCommand{key=IntermediateCompositeKey [taskId=eb7da48a-5922-4671-9037-4077e209744c, key=RedHat], value=null, flags=null, valueMatcher=MATCH_ALWAYS} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@c0bbc61] > {noformat} -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5093) Granularity of remote event listener implementations doing the same job

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5093?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5093: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > Granularity of remote event listener implementations doing the same job > ----------------------------------------------------------------------- > > Key: ISPN-5093 > URL: https://issues.jboss.org/browse/ISPN-5093 > Project: Infinispan > Issue Type: Enhancement > Components: Remote Protocols > Reporter: Galder Zamarreño > Assignee: Galder Zamarreño > Fix For: 9.0.0.Alpha1 > > > Currently, if N clients add the same listener to a cache that does the same job, e.g. keeping a near cache consistent, this results in N server-side cluster listeners created, each potentially installed in different nodes. If one of those nodes fails, all clients that had a listener registered to that node will have to find a different node for this listener. > The downsides of this approach is that there are as many cluster listeners installed as clients have added listeners (or have near cache enabled), which might not very efficient. If a node goes down, all clients that have cluster listeners there need to failover to some other node. > The advantage of this approach is simplicity of the approach to decide where to add the listener and where to failover to. > For this type of scenarios, an alternative set up might be worth exploring: > If all these client side listeners are interested in exactly the same events, and the client ID would be exposed via the RemoteCache API, a server side cluster listener multi-plexing between all these clients could be potentially built. In other words, instead of having N clients register N cluster listeners, the first client would register the cluster listener with a client listener ID, and if more registrations were added with the same client listener ID, the connections would be added to the existing cluster listener implementation. > The maximise the efficiency of this solution, all clients (even running in different JMVs), given the same client listener ID, should agree upon the node to add the listener in. For a distributed cache, hashing on the cache name would work. For replicated caches, since there's no hashing available, the first node of the view could be used. > Since the logic to be executed server-side varies between being the first node adding the client listener vs the others, synchronization would be added to make sure that the first invocation only creates the cluster listener, and the others simply add the channel to the listener. > Failover is a bit more tricky too, because if the node with the cluster listener goes down, all the clients have to failover, which again exposes a 1st vs the others type of logic. > Advantages of this approach is the reduction in number of cluster listeners and potentially efficiency coming from a single cluster listener implementation server side. > The disadvantages come from the server side logic to add/failover a cluster listener, which need to take into account if the listener is present or not. Other disadvantages come from needing the clients to use some specific routing for adding listeners for same node. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5468) Change the interceptors to move the new sequential interfaces

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5468?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5468: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > Change the interceptors to move the new sequential interfaces > ------------------------------------------------------------- > > Key: ISPN-5468 > URL: https://issues.jboss.org/browse/ISPN-5468 > Project: Infinispan > Issue Type: Task > Components: Core > Affects Versions: 7.2.1.Final > Reporter: Dan Berindei > Assignee: Dan Berindei > Fix For: 9.0.0.Alpha1 > > > The first step is to move all interceptors to the new sequential interfaces, without actually allowing the command execution to be suspended and resumed. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5453) X-site state transfer: retry locally if local state push throws a SuspectException

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5453?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5453: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > X-site state transfer: retry locally if local state push throws a SuspectException > ---------------------------------------------------------------------------------- > > Key: ISPN-5453 > URL: https://issues.jboss.org/browse/ISPN-5453 > Project: Infinispan > Issue Type: Bug > Components: Core, Cross-Site Replication > Affects Versions: 7.2.1.Final > Reporter: Dan Berindei > Priority: Minor > Fix For: 9.0.0.Alpha1 > > > When the local state push command throws a {{SuspectException}}, we currently retry remotely unless the target node was already removed from the cache topology. This will fail with another {{SuspectException}}, so it would be better to retry locally the first time. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5426) Create a remote query tutorial

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5426?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5426: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > Create a remote query tutorial > ------------------------------ > > Key: ISPN-5426 > URL: https://issues.jboss.org/browse/ISPN-5426 > Project: Infinispan > Issue Type: Feature Request > Components: Documentation-Query, Remote Querying > Reporter: Adrian Nistor > Assignee: Adrian Nistor > Fix For: 9.0.0.Alpha1 > > -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5454) XSite: RetryMechanismTest random failures

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5454?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5454: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > XSite: RetryMechanismTest random failures > ----------------------------------------- > > Key: ISPN-5454 > URL: https://issues.jboss.org/browse/ISPN-5454 > Project: Infinispan > Issue Type: Bug > Components: Cross-Site Replication > Affects Versions: 7.2.1.Final > Reporter: Dan Berindei > Priority: Blocker > Labels: testsuite_stability > Fix For: 9.0.0.Alpha1 > > > {{ClusteredCacheBackupReceiver.awaitRemoteTask()}} doesn't respect the state push command's timeout, at least when it's smaller than the sync replication timeout in the target cache. When that happens, the state provider will resend the state, and there will be 2 state push commands executing at the same time. > RetryMechanismTest changes the state push timeout to 2 seconds, but the sync replication timeout stays at 15 seconds. This causes failures in {{testRetryLocally}} and {{testFailRetryLocally}}, if it takes more than 2 seconds to suspect the killed node. > {noformat} > 10:02:13,007 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)] > 10:02:16,008 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)] > 10:02:16,040 TRACE (asyncTransportThread-4,NodeP:) [RpcManagerImpl] replication exception: > org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected > 10:02:16,040 TRACE (asyncTransportThread-0,NodeP:) [RpcManagerImpl] replication exception: > org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected > 10:02:19,147 ERROR (testng-RetryMechanismTest:) [UnitTestTestNGListener] Test testFailRetryLocally(org.infinispan.xsite.statetransfer.failures.RetryMechanismTest) failed. > java.lang.AssertionError: expected:<2> but was:<3> > at org.testng.AssertJUnit.fail(AssertJUnit.java:59) > at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) > at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) > at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245) > at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252) > at org.infinispan.xsite.statetransfer.failures.RetryMechanismTest.testFailRetryLocally(RetryMechanismTest.java:227) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5467) Design new interceptor interfaces for sequential invocation

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5467?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5467: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > Design new interceptor interfaces for sequential invocation > ----------------------------------------------------------- > > Key: ISPN-5467 > URL: https://issues.jboss.org/browse/ISPN-5467 > Project: Infinispan > Issue Type: Task > Components: Core > Affects Versions: 7.2.1.Final > Reporter: Dan Berindei > Assignee: Dan Berindei > Fix For: 9.0.0.Alpha1 > > > We need the interceptors to execute in sequence instead of using a stack in order to allow interrupting the execution of a command on one thread and continuing the execution on another thread. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5424) SemaphoreCompletionService improvements

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5424?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5424: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > SemaphoreCompletionService improvements > --------------------------------------- > > Key: ISPN-5424 > URL: https://issues.jboss.org/browse/ISPN-5424 > Project: Infinispan > Issue Type: Task > Components: Core > Affects Versions: 7.2.0.CR1 > Reporter: Dan Berindei > Assignee: Dan Berindei > Fix For: 9.0.0.Alpha1 > > > Turns out the {{CompletionService}} features aren't that necessary, and it makes the use of {{Runnable}} more cumbersome. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

[JBoss JIRA] (ISPN-5415) Expose protobuf entries to scripting

by Pedro Ruivo (JIRA)

[ https://issues.jboss.org/browse/ISPN-5415?page=com.atlassian.jira.plugin.... ] Pedro Ruivo updated ISPN-5415: ------------------------------ Fix Version/s: 9.0.0.Alpha1 (was: 8.2.0.Final) > Expose protobuf entries to scripting > ------------------------------------ > > Key: ISPN-5415 > URL: https://issues.jboss.org/browse/ISPN-5415 > Project: Infinispan > Issue Type: Feature Request > Components: Remote Querying > Affects Versions: 8.0.0.Final > Reporter: Adrian Nistor > Assignee: Adrian Nistor > Fix For: 9.0.0.Alpha1 > > > We need an alternative API for Protostream marshalling that is easy to consume from scripting languages. The messages need to be unmarshalled into a map-like object that can be accessed easily from scripting languages. No marshaller implementation code should be provided by users, also no annotations. -- This message was sent by Atlassian JIRA (v6.4.11#64026)

10 years, 1 month

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

infinispan-issues March 2016