[JBoss JIRA] (ISPN-5151) DistributedSharedCacheTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5151?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5151:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> DistributedSharedCacheTwoNodesMapReduceTest.testInvokeMapReduceOnAllKeys random failures
> ----------------------------------------------------------------------------------------
>
> Key: ISPN-5151
> URL: https://issues.jboss.org/browse/ISPN-5151
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.3.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 9.1.0.Final
>
>
> The method {{invokeMapReduce()}} doesn't really invoke the M/R task, it only creates it, and the execution only starts when the test method calls {{task.execute()}} explicitly. It shouldn't try to check the contents of the shared intermediary cache, because the intermediary cache may not exist yet - and it may accidentally create it with the wrong configuration. I get this error when I run only the {{testInvokeMapReduceOnAllKeys}} method:
> {noformat}
> 09:55:37,632 TRACE (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [DefaultCacheManager] About to wire and start cache __tmpMapReduce
> 09:55:37,646 DEBUG (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [MapReduceTask] Invoking CreateCacheCommand{cacheManager=null, cacheNameToCreate='__tmpMapReduce', cacheConfigurationName='__tmpMapReduce', start=true', size=2} across members [DistributedSharedCacheTwoNodesMapReduceTest-NodeA-19271, DistributedSharedCacheTwoNodesMapReduceTest-NodeB-10341]
> 10:32:56,324 ERROR (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest) failed.
> org.infinispan.distexec.mapreduce.MapReduceException: Map phase failed
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:607)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeHelper(MapReduceTask.java:473)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:414)
> at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:162)
> Caused by: org.infinispan.commons.CacheException: java.lang.NullPointerException
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:105)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.invokeMapCombineLocally(MapReduceTask.java:1174)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart.access$300(MapReduceTask.java:1101)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:1123)
> at org.infinispan.distexec.mapreduce.MapReduceTask$MapTaskPart$1.call(MapReduceTask.java:1119)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapKeysToNodes(MapReduceManagerImpl.java:363)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.migrateIntermediateKeysAndValues(MapReduceManagerImpl.java:327)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombine(MapReduceManagerImpl.java:260)
> at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:103)
> ... 10 more
> {noformat}
> Even if the check is moved after the M/R task is finished, it still wouldn't be correct, because the task only cleans up the shared intermediary cache asynchronously. So it needs to use {{eventually()}} to avoid errors like this:
> {noformat}
> 04:06:32,260 ERROR (testng-DistributedSharedCacheTwoNodesMapReduceTest:) [UnitTestTestNGListener] Test testInvokeMapReduceOnAllKeys(org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest) failed.
> java.lang.AssertionError: Shared cache __tmpMapReduce is not empty. It has 5 keys/values: [ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=is], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@21ae10d3}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=JUDCon], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@108d6b51}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=cool], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@77949e8f}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=Infinispan], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@712a6071}, ImmortalCacheEntry{key=IntermediateCompositeKey [taskId=88948a8b-2a8a-4c13-bc45-4dc3a9f6b0fb, key=community], value=org.infinispan.distexec.mapreduce.MapReduceManagerImpl$DeltaAwareList@291bdf76}] expected:<0> but was:<5>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.infinispan.distexec.mapreduce.DistributedSharedCacheTwoNodesMapReduceTest.invokeMapReduce(DistributedSharedCacheTwoNodesMapReduceTest.java:44)
> at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testInvokeMapReduceOnAllKeys(BaseWordCountMapReduceTest.java:161)
> 04:06:32,579 TRACE (transport-thread-NodeA-p29577-t6:) [InvocationContextInterceptor] Invoked with command RemoveCommand{key=IntermediateCompositeKey [taskId=eb7da48a-5922-4671-9037-4077e209744c, key=RedHat], value=null, flags=null, valueMatcher=MATCH_ALWAYS} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@c0bbc61]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years
[JBoss JIRA] (ISPN-5021) Nodes that finish the rebalance later can see outdated values
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5021?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5021:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> Nodes that finish the rebalance later can see outdated values
> -------------------------------------------------------------
>
> Key: ISPN-5021
> URL: https://issues.jboss.org/browse/ISPN-5021
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.2.Final
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Labels: consistency, testsuite_stability
> Fix For: 9.1.0.Final
>
>
> Copied from [ISPN-4444|https://issues.jboss.org/browse/ISPN-4444?focusedCommentId=1302...]
> If the CH_UPDATE command is delayed on the old owner, the new owners might update the key without the old owner knowing, and a locality check on the old owner won't help.
> I remember one thing that struck me when reading the Raft algorithm was that they install configuration changes symmetrically, in 3 phases. We might need to do the same for our rebalance:
> 1. T0: read_ch=old, write_ch=old
> 2. start a rebalance
> 3. T1: read_ch=old, write_ch=old+new
> 4. new owners have all the data
> 5. T2: read_ch=new, write_ch=old+new
> 6. remove old cache entries and ignore further writes
> 7. T3: read_ch=new, write_ch=new
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years
[JBoss JIRA] (ISPN-5073) Improve "Number of Entries" stats
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5073?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5073:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> Improve "Number of Entries" stats
> ---------------------------------
>
> Key: ISPN-5073
> URL: https://issues.jboss.org/browse/ISPN-5073
> Project: Infinispan
> Issue Type: Task
> Components: Core, JMX, reporting and management
> Affects Versions: 7.1.0.Alpha1
> Reporter: Tristan Tarrant
> Assignee: Vladimir Blagojevic
> Fix For: 9.1.0.Final
>
>
> Currently the getNumberOfEntries in CacheMgmtInterceptor returns the size of the datacontainer which doesn't take into account expired entries and cache stores.
> To avoid compatibility issues, modify the description to reflect its behaviour and add proper statistics, possibly with different flag combinations (SKIP_REMOTE, SKIP_CACHE_LOAD)
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years
[JBoss JIRA] (ISPN-5075) org.infinispan.IllegalLifecycleStateException in query tests
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5075?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5075:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> org.infinispan.IllegalLifecycleStateException in query tests
> ------------------------------------------------------------
>
> Key: ISPN-5075
> URL: https://issues.jboss.org/browse/ISPN-5075
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Query
> Affects Versions: 7.1.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.1.0.Final
>
>
> Many query tests log an error after running:
> {code}
> Exception in thread "Hibernate Search: async deletion of index segments-1" org.infinispan.IllegalLifecycleStateException: ISPN000323: Cache 'LuceneIndexesMetadata' is in 'TERMINATED' state and so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:89)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:77)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:44)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:422)
> at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:427)
> at org.infinispan.lucene.impl.FileListOperations.getFileList(FileListOperations.java:160)
> at org.infinispan.lucene.impl.FileListOperations.deleteFileName(FileListOperations.java:129)
> at org.infinispan.lucene.impl.DirectoryImplementor.deleteFile(DirectoryImplementor.java:64)
> at org.infinispan.lucene.impl.DirectoryLuceneV4$DeleteTask.run(DirectoryLuceneV4.java:195)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> That error does not cause the tests to fail, and are due to the async deletes of segments recently implemented. Need to check if the executors are being flushed correctly when the cache manager stops
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years
[JBoss JIRA] (ISPN-5077) Custom remote events can be slightly inefficient
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5077?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5077:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> Custom remote events can be slightly inefficient
> ------------------------------------------------
>
> Key: ISPN-5077
> URL: https://issues.jboss.org/browse/ISPN-5077
> Project: Infinispan
> Issue Type: Enhancement
> Components: Remote Protocols
> Affects Versions: 7.0.2.Final
> Reporter: Galder Zamarreño
> Fix For: 9.1.0.Final
>
>
> Something we might want to improve for Hot Rod 3.0 protocol:
> [16:40] <galderz> i've been thinking further about converters, and I think i've found a slight mismatch between what converter means for embedded listeners vs remote listeners
> [16:40] <wburns> oh yeah?
> [16:40] <galderz> for embedded listeners, it essentially transforms what you see as `value`
> [16:41] <galderz> with the knowledge that key and metadata information will be shipped
> [16:41] <galderz> the way i mapped converter to remote listeners is that whatever the converter returns, we ship that, as is, to the client
> [16:41] <galderz> so, if a remote listener wants a custom event that includes key + value
> [16:41] <galderz> it needs to develop a converter impl that returns bytes containing key + value
> [16:41] <galderz> which is inefficient because you are passing around the key twice
> [16:42] <galderz> once as part of the event itself, and again inside the converted value
> [16:42] <galderz> inefficient from the POV of shipping stuff around from other nodes to where the cluster listener is located
> [16:44] <wburns> yeah makes sense
> [16:44] <galderz> not a major issue but not easy to fix without changing semantics or public protocol
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years
[JBoss JIRA] (ISPN-4903) ServerFailureRetrySingleOwnerTest doesn't actually test client retry
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-4903?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-4903:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> ServerFailureRetrySingleOwnerTest doesn't actually test client retry
> --------------------------------------------------------------------
>
> Key: ISPN-4903
> URL: https://issues.jboss.org/browse/ISPN-4903
> Project: Infinispan
> Issue Type: Bug
> Components: Server, Test Suite - Server
> Affects Versions: 7.0.0.CR2
> Reporter: Dan Berindei
> Fix For: 9.1.0.Final
>
> Attachments: ServerFailureRetrySingleOwnerTest.java
>
>
> With {{useSynchronization = true}} (the default, before ISPN-4166 is integrated), the {{SuspectException}} thrown by the listener is swallowed by the transaction manager and the client doesn't retry. The test doesn't pick that up because the exception is thrown _after_ the entry was updated in the data container (a regular SuspectException would be thrown before).
> I changed the configuration to {{useSynchronization = false}}, but it didn't work because the {{SuspectException}} is wrapped in a {{CacheListenerException}}, so the client throws an exception instead of retrying. I also changed the test to use an interceptor instead of a listener, but then I got a {{ClassCastException}}:
> {noformat}
> Caused by: java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry
> at org.infinispan.cache.impl.CacheImpl.getCacheEntry(CacheImpl.java:424)
> at org.infinispan.cache.impl.CacheImpl.getCacheEntry(CacheImpl.java:429)
> at org.infinispan.server.hotrod.Decoder2x$.customReadKey(Decoder2x.scala:285)
> at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:156)
> at org.infinispan.server.core.AbstractProtocolDecoder.org$infinispan$server$core$AbstractProtocolDecoder$$decodeKey(AbstractProtocolDecoder.scala:176)
> at org.infinispan.server.core.AbstractProtocolDecoder.decodeDispatch(AbstractProtocolDecoder.scala:71) ... 14 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years
[JBoss JIRA] (ISPN-5557) Core threading redesign
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5557?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5557:
----------------------------------
Fix Version/s: 9.1.0.Final
(was: 9.0.0.Final)
> Core threading redesign
> -----------------------
>
> Key: ISPN-5557
> URL: https://issues.jboss.org/browse/ISPN-5557
> Project: Infinispan
> Issue Type: Task
> Components: Core
> Affects Versions: 7.2.2.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 9.1.0.Final
>
>
> Infinispan needs a lot of threads, because everything is synchronous: locking, remote command invocations, cache writers. This causes various issues, from general context switching overhead to the thread pools getting full and causing deadlocks.
> We should redesign the core so that most blocking happens on the application threads, and the number of internal threads is kept to a minimum.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years