[JBoss JIRA] (ISPN-8724) Server hangs during stop
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-8724:
-----------------------------------------
I observed the same hanging in the [Infinispan Spark testsuite|https://github.com/infinispan/infinispan-spark/], when stopping the server. It occurs intermittently
> Server hangs during stop
> ------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache manager start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (ISPN-8724) Server hangs during stop
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8724:
------------------------------------
Summary: Server hangs during stop (was: HotRodRemoteStreamingIT hangs)
> Server hangs during stop
> ------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache manager start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month