HotRodRemoteStreamingIT hangs ----------------------------- Key: ISPN-8724 URL: https://issues.jboss.org/browse/ISPN-8724 Project: Infinispan Issue Type: Bug Affects Versions: 9.2.0.CR1 Reporter: Gustavo Fernandes Priority: Critical Attachments: s1, s2, test This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer(). The surefire process waits forever for the two servers to stop. One of the servers is waiting for the initial state transfer to complete: {noformat} "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source) at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source) at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source) at org.infinispan.security.Security.doPrivileged(Security.java:76) at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440) at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132) at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) {noformat} The other server is running a series of operations after the server stops, such as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads: MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3 {noformat} "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729) at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934) at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394) at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25) at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13) at org.infinispan.security.Security.doPrivileged(Security.java:76) at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39) at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77) - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl) at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486) at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216) at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206) - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056) at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} For some reason, also in the other server, the CounterModule lifecycle is reacting to a cache manager start event, and it's hanged as well: {noformat} "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source) at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source) at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source) at org.infinispan.security.Security.doPrivileged(Security.java:76) at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440) at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132) at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source) at java.lang.Thread.run(Thread.java:748) {noformat}