[infinispan-issues] [JBoss JIRA] (ISPN-9501) AbstractCacheStream.performOperationRehashAware() can hang

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-9501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631843#comment-13631843 ] 

Dan Berindei commented on ISPN-9501:
------------------------------------

I have reproduced in a test that starts multiple nodes with counter caches concurrently, but it took a lot of iterations:

{noformat}
java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.startCaches() on object of type EmbeddedCounterManager
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.startCaches() on object of type EmbeddedCounterManager
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271)
	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:680)
	at org.infinispan.test.MultipleCacheManagersTest.addClusterEnabledCacheManager(MultipleCacheManagersTest.java:238)
	at org.infinispan.test.MultipleCacheManagersTest.addClusterEnabledCacheManager(MultipleCacheManagersTest.java:208)
	at org.infinispan.counter.impl.CounterConcurrentStartTest.lambda$testConcurrentStart$0(CounterConcurrentStartTest.java:44)
	at org.infinispan.test.AbstractInfinispanTest.lambda$fork$4(AbstractInfinispanTest.java:274)
	at org.infinispan.test.AbstractInfinispanTest$CallableWrapper.call(AbstractInfinispanTest.java:530)
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
	at java.util.concurrent.FutureTask.run(FutureTask.java)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.startCaches() on object of type EmbeddedCounterManager
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
	at org.infinispan.security.Security.doPrivileged(Security.java:44)
	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620)
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267)
	... 11 more
Caused by: org.infinispan.commons.CacheException: java.util.concurrent.TimeoutException
	at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:332)
	at org.infinispan.stream.impl.AbstractCacheStream.performOperation(AbstractCacheStream.java:229)
	at org.infinispan.stream.impl.DistributedCacheStream.anyMatch(DistributedCacheStream.java:397)
	at org.infinispan.util.AbstractDelegatingCacheStream.anyMatch(AbstractDelegatingCacheStream.java:300)
	at org.infinispan.CacheStream.anyMatch(CacheStream.java:462)
	at org.infinispan.counter.impl.manager.CounterConfigurationManager.start(CounterConfigurationManager.java:90)
	at org.infinispan.counter.impl.manager.EmbeddedCounterManager.startCaches(EmbeddedCounterManager.java:97)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
	... 22 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
	at org.infinispan.stream.impl.AbstractCacheStream.performOperationRehashAware(AbstractCacheStream.java:330)
	... 33 more
{noformat}


> AbstractCacheStream.performOperationRehashAware() can hang
> ----------------------------------------------------------
>
>                 Key: ISPN-9501
>                 URL: https://issues.jboss.org/browse/ISPN-9501
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 9.4.0.CR1
>            Reporter: Dan Berindei
>            Assignee: Dan Berindei
>             Fix For: 9.4.0.Final
>
>
> There are actually 2 different issues:
> # {{segmentsToProcess}} reuses the {{remoteResults.lostSegments}} {{ConcurrentSmallIntSet}} instance, so when {{remoteResults.lostSegments}} is cleared, {{segmentsToProcess}} is also cleared.
> # Because of ISPN-9500, {{segmentsToProcess.isEmpty()}} keeps returning {{false}}, so {{performOperationRehashAware()}} keeps waiting for a newer topology. 



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)