Total order caches can hang during join --------------------------------------- Key: ISPN-7172 URL: https://issues.jboss.org/browse/ISPN-7172 Project: Infinispan Issue Type: Bug Components: Core, Test Suite - Core Affects Versions: 9.0.0.Alpha4 Reporter: Dan Berindei Assignee: Dan Berindei Priority: Critical Labels: testsuite_stability Fix For: 9.0.0.Beta1 When two nodes join a cache in quick sequence, it's possible for a {{CacheTopologyControlCommand}} to arrive between {{LocalTopologyManagerImpl}} adding the {{LocalCacheStatus}} to the {{runningCaches}} map, but before it calling {{cacheStatus.getTopologyUpdatesExecutor().executeAsync(() -> joinFuture)}} to block topology updates. This exposes a bug in the way {{LimitedExecutor}} wraps a {{WithinThreadExecutor}}, which causes {{SyncPrepareUseSynchronizationTotalOrderTest}} (and possibly other total order tests) to hang: {noformat} "testng-SyncPrepareUseSynchronizationTotalOrderTest" #15 prio=5 os_prio=0 tid=0x00007fc674b74000 nid=0x2f3f waiting on condition [0x00007fc626682000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000829d03a0> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1687) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3320) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1767) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1907) at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82) at org.infinispan.executors.LimitedExecutor.execute(LimitedExecutor.java:70) at org.infinispan.executors.LimitedExecutor.executeAsync(LimitedExecutor.java:96) at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:141) at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:121) at sun.reflect.GeneratedMethodAccessor146.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:867) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:633) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:622) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:547) - locked <0x00000000829d0520> (a org.infinispan.factories.ComponentRegistry) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:231) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:808) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:639) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:590) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:422) at org.infinispan.test.MultipleCacheManagersTest.getCaches(MultipleCacheManagersTest.java:248) at org.infinispan.test.MultipleCacheManagersTest.waitForClusterToForm(MultipleCacheManagersTest.java:257) at org.infinispan.test.MultipleCacheManagersTest.waitForClusterToForm(MultipleCacheManagersTest.java:266) at org.infinispan.tx.totalorder.simple.BaseSimpleTotalOrderTest.createCacheManagers(BaseSimpleTotalOrderTest.java:326) at org.infinispan.test.MultipleCacheManagersTest.callCreateCacheManagers(MultipleCacheManagersTest.java:109) at org.infinispan.test.MultipleCacheManagersTest.createBeforeMethod(MultipleCacheManagersTest.java:119) {noformat}