[JBoss JIRA] (ISPN-5151) DistributedSharedCacheTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5151?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5151:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> DistributedSharedCacheTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-5151
> URL: https://issues.jboss.org/browse/ISPN-5151
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.3.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 9.0.0.CR2
>
>
> The method {{invokeMapReduce()}} doesn't really invoke the M/R task, it only creates it, and the execution only starts when the test method calls {{task.execute()}} explicitly. It shouldn't try to check the contents of the shared intermediary cache, because the intermediary cache may not exist yet - and it may accidentally create it with the wrong configuration. I get this error when I run only the {{testInvokeMapReduceOnAllKeys}} method:
> {noformat}
> 09:55:37,632 TRACE (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [DefaultCacheManager] About to wire and start cache __tmpMapReduce
> 09:55:37,646 DEBUG (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [MapReduceTask] Invoking CreateCacheCommand{cacheManager=null, cacheNameToCreate='__tmpMapReduce', cacheConfigurationName='__tmpMapReduce', start=true', size=2} across members [DistributedSharedCacheTwoNodesMapReduceTest-NodeA-19271, DistributedSharedCacheTwoNodesMapReduceTest-NodeB-10341]
> 10:32:56,324 ERROR (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest) failed.
> org.infinispan.distexec.mapreduce.MapReduceException: Map phase failed
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:607)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeHelper(MapReduceTask.java:473)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:414)
> at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:162)
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:105)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocally(MapReduceTask.java:1174)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$300(MapReduceTask.java:1101)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:1123)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:1119)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapKeysToNodes(MapReduceManagerImpl.java:363)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.migrateIntermediateKeysAndValues(MapReduceManagerImpl.java:327)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombine(MapReduceManagerImpl.java:260)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:103)
> ... 10 more
> {noformat}
> Even if the check is moved after the M/R task is finished, it still wouldn't be correct, because the task only cleans up the shared intermediary cache asynchronously. So it needs to use {{eventually()}} to avoid errors like this:
> {noformat}
> 04:06:32,260 ERROR (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest) failed.
> java.lang.AssertionError: Shared cache __tmpMapReduce is not empty. It has 5 keys/values: [ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=is], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@21ae10d3}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=JUDCon], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@108d6b51}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=cool], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@77949e8f}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=Infinispan], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@712a6071}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=community], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@291bdf76}] expected:<0> but was:<5>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest.invokeMapReduce(DistributedSharedCacheTwoNodesMapReduceTest.java:44)
> at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:161)
> 04:06:32,579 TRACE (transport-thread-NodeA-p29577-t6:) [InvocationContextInterceptor] Invoked with command RemoveCommand{key=IntermediateCompositeKey [taskId=eb7da48a-5922-4671-9037-4077e209744c, key=RedHat], value=null, flags=null, valueMatcher=MATCH_ALWAYS} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@c0bbc61]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-5093) Granularity of remote event listener implementations doing the same job
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5093?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5093:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> Granularity of remote event listener implementations doing the same job
> -----------------------------------------------------------------------
>
> Key: ISPN-5093
> URL: https://issues.jboss.org/browse/ISPN-5093
> Project: Infinispan
> Issue Type: Enhancement
> Components: Remote Protocols
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 9.0.0.CR2
>
>
> Currently, if N clients add the same listener to a cache that does the same job, e.g. keeping a near cache consistent, this results in N server-side cluster listeners created, each potentially installed in different nodes. If one of those nodes fails, all clients that had a listener registered to that node will have to find a different node for this listener.
> The downsides of this approach is that there are as many cluster listeners installed as clients have added listeners (or have near cache enabled), which might not very efficient. If a node goes down, all clients that have cluster listeners there need to failover to some other node.
> The advantage of this approach is simplicity of the approach to decide where to add the listener and where to failover to.
> For this type of scenarios, an alternative set up might be worth exploring:
> If all these client side listeners are interested in exactly the same events, and the client ID would be exposed via the RemoteCache API, a server side cluster listener multi-plexing between all these clients could be potentially built. In other words, instead of having N clients register N cluster listeners, the first client would register the cluster listener with a client listener ID, and if more registrations were added with the same client listener ID, the connections would be added to the existing cluster listener implementation.
> The maximise the efficiency of this solution, all clients (even running in different JMVs), given the same client listener ID, should agree upon the node to add the listener in. For a distributed cache, hashing on the cache name would work. For replicated caches, since there's no hashing available, the first node of the view could be used.
> Since the logic to be executed server-side varies between being the first node adding the client listener vs the others, synchronization would be added to make sure that the first invocation only creates the cluster listener, and the others simply add the channel to the listener.
> Failover is a bit more tricky too, because if the node with the cluster listener goes down, all the clients have to failover, which again exposes a 1st vs the others type of logic.
> Advantages of this approach is the reduction in number of cluster listeners and potentially efficiency coming from a single cluster listener implementation server side.
> The disadvantages come from the server side logic to add/failover a cluster listener, which need to take into account if the listener is present or not. Other disadvantages come from needing the clients to use some specific routing for adding listeners for same node.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-4903) ServerFailureRetrySingleOwnerTest doesn't actually test client retry
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-4903?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-4903:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> ServerFailureRetrySingleOwnerTest doesn't actually test client retry
> --------------------------------------------------------------------
>
> Key: ISPN-4903
> URL: https://issues.jboss.org/browse/ISPN-4903
> Project: Infinispan
> Issue Type: Bug
> Components: Server, Test Suite - Server
> Affects Versions: 7.0.0.CR2
> Reporter: Dan Berindei
> Fix For: 9.0.0.CR2
>
> Attachments: ServerFailureRetrySingleOwnerTest.java
>
>
> With {{useSynchronization = true}} (the default, before ISPN-4166 is integrated), the {{SuspectException}} thrown by the listener is swallowed by the transaction manager and the client doesn't retry. The test doesn't pick that up because the exception is thrown _after_ the entry was updated in the data container (a regular SuspectException would be thrown before).
> I changed the configuration to {{useSynchronization = false}}, but it didn't work because the {{SuspectException}} is wrapped in a {{CacheListenerException}}, so the client throws an exception instead of retrying. I also changed the test to use an interceptor instead of a listener, but then I got a {{ClassCastException}}:
> {noformat}
> Caused by: java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry
> at org.infinispan.cache.impl.CacheImpl.getCacheEntry(CacheImpl.java:424)
> at org.infinispan.cache.impl.CacheImpl.getCacheEntry(CacheImpl.java:429)
> at org.infinispan.server.hotrod.Decoder2x$.customReadKey(Decoder2x.scala:285)
> at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:156)
> at org.infinispan.server.core.AbstractProtocolDecoder.org$infinispan$server$core$AbstractProtocolDecoder$$decodeKey(AbstractProtocolDecoder.scala:176)
> at org.infinispan.server.core.AbstractProtocolDecoder.decodeDispatch(AbstractProtocolDecoder.scala:71) ... 14 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-5499) SizeTest.testPersistentDistributedCacheSize random failures
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5499?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5499:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> SizeTest.testPersistentDistributedCacheSize random failures
> -----------------------------------------------------------
>
> Key: ISPN-5499
> URL: https://issues.jboss.org/browse/ISPN-5499
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 9.0.0.CR2
>
>
> {noformat}
> 16:04:28,678 ERROR (testng-SizeTest:) [UnitTestTestNGListener] Test testPersistentDistributedCacheSize(org.infinispan.client.hotrod.SizeTest) failed.
> java.lang.AssertionError: expected:<20> but was:<38>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
> at org.infinispan.client.hotrod.SizeTest.testPersistentDistributedCacheSize(SizeTest.java:59)
> {noformat}
> I have been able to make the test fail reliably by replacing the assertion on line 57 with this:
> {code}
> for (int i = 0; i < SIZE; i++) {
> assertEquals(SIZE, clients.get(0).getCache(cacheName).size());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-5476) Cross-site tests should run in parallel
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5476?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5476:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> Cross-site tests should run in parallel
> ---------------------------------------
>
> Key: ISPN-5476
> URL: https://issues.jboss.org/browse/ISPN-5476
> Project: Infinispan
> Issue Type: Task
> Components: Core, Cross-Site Replication, Test Suite - Core
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Fix For: 9.0.0.CR2
>
>
> Currently the cross-site tests have to run in a single thread, and that means they're much slower than the regular core tests.
> It also means they need to run with a separate Maven profile, and that (combined with their duration) makes it very unlikely for devs to run the xsite tests on their machines.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-5510) Provide better Hot Rod client socket timeout and retry defaults
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5510?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5510:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> Provide better Hot Rod client socket timeout and retry defaults
> ---------------------------------------------------------------
>
> Key: ISPN-5510
> URL: https://issues.jboss.org/browse/ISPN-5510
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 9.0.0.CR2
>
>
> The current defaults are:
> * Socket timeout = 60 seconds
> * Max retries = 10
> As a result of these defaults, if the server hangs an operation, it'd take 10 minutes (60 second timeout x 10 retries) for the operation to finally return an exception to the client, which is way too much.
> So, these default value should change to be more aggressive: 30 second socket timeout and 3 max retries.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-5515) Purge store if there is another node already running
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5515?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5515:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> Purge store if there is another node already running
> ----------------------------------------------------
>
> Key: ISPN-5515
> URL: https://issues.jboss.org/browse/ISPN-5515
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core, Loaders and Stores
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 9.0.0.CR2
>
>
> Preloading happens before communicating with other nodes that might already have the cache running. When joining the existing members, the cache then waits to receive the first CH in which it is a member, and then deletes only the entries in the segments that it doesn't own in that CH.
> The intention of this was to remove as little as possible from the existing data, e.g. if the first node to start up is not the one that was stopped last. But the preloaded entries are not replicated to the other nodes, so this can lead to inconsistencies.
> It would be better to delay preloading until we know we are the first node to start up, but failing that we could clear the data container and the store before receiving the initial state.
> Note that this will only allow preloading data from one node. Restoring data from more nodes is harder to do, and we will implement it as part of graceful restart.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month
[JBoss JIRA] (ISPN-5513) State Transfer can miss entries that are concurrently activated
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5513?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5513:
----------------------------------
Fix Version/s: 9.0.0.CR2
(was: 9.0.0.CR1)
> State Transfer can miss entries that are concurrently activated
> ---------------------------------------------------------------
>
> Key: ISPN-5513
> URL: https://issues.jboss.org/browse/ISPN-5513
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 8.0.0.Alpha1
> Reporter: William Burns
> Fix For: 9.0.0.CR2
>
>
> Currently the OutboundTransferTask iterates upon the data container and then runs process for the state loader. However if an entry is activated during or after the data container iteration it is possible this entry is then not seen and subsequently is not present in the store when it is processed.
> EntryRetriever had this same issue and it was required to register a cache listener to listen for activations and then replay the data after finishing with the store.
> This can cause duplicate values as well, however replacing the same exact value is fine and if a non ST write occurs the state is ignored anyways.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
9 years, 1 month