[JBoss JIRA] (ISPN-8724) HotRodRemoteStreamingIT hangs
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8724:
------------------------------------
Description:
This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
The surefire process waits forever for the two servers to stop.
One of the servers is waiting for the initial state transfer to complete:
{noformat}
"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
The other server is running a series of operations after the server stops, such
as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
{noformat}
"MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
- locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
- locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
For some reason, also in the other server, the CounterModule lifecycle is reacting to a
cache manager start event, and it's hanged as well:
{noformat}
"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
The surefire process waits forever for the two servers to stop.
One of the servers is waiting for the initial state transfer to complete:
{noformat}
"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
The other server is running a series of operations after the server stops, such
as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
{noformat}
"MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
- locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
- locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
For some reason, also in the other server, the CounterModule lifecycle is reacting to a
cache start event, and it's hanged as well:
{noformat}
"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> HotRodRemoteStreamingIT hangs
> -----------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache manager start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8724) HotRodRemoteStreamingIT hangs
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8724:
------------------------------------
Status: Open (was: New)
> HotRodRemoteStreamingIT hangs
> -----------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8724) HotRodRemoteStreamingIT hangs
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8724:
------------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5699
> HotRodRemoteStreamingIT hangs
> -----------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8706) HotRodMergeTest Timed out waiting for rebalancing to complete
by Radim Vansa (JIRA)
[ https://issues.jboss.org/browse/ISPN-8706?page=com.atlassian.jira.plugin.... ]
Radim Vansa commented on ISPN-8706:
-----------------------------------
[~ryanemerson] I've found {{HotRodMergeTest.testNewTopologySentAfterOverlappingMerge}} to be timing out as well, and this is a stack trace I've captured when waiting for it:
{code}
"transport-thread-HotRodMergeTest[DIST_SYNC]-NodeA-p309-t6" #1172 daemon prio=5 os_prio=0 tid=0x00007efdb8011000 nid=0x23bd waiting on condition [0x00007efd56c59000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c5383230> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at org.infinispan.conflict.impl.DefaultConflictManager$ReplicaSpliterator.tryAdvance(DefaultConflictManager.java:452)
at java.util.Spliterator.forEachRemaining(Spliterator.java:326)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.infinispan.conflict.impl.DefaultConflictManager.doResolveConflicts(DefaultConflictManager.java:257)
at org.infinispan.conflict.impl.DefaultConflictManager.resolveConflicts(DefaultConflictManager.java:244)
at org.infinispan.topology.ClusterCacheStatus.updateTopologiesAfterMerge(ClusterCacheStatus.java:193)
- locked <0x00000000c56202a8> (a org.infinispan.topology.ClusterCacheStatus)
at org.infinispan.partitionhandling.impl.PreferAvailabilityStrategy.onPartitionMerge(PreferAvailabilityStrategy.java:218)
at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:597)
- locked <0x00000000c56202a8> (a org.infinispan.topology.ClusterCacheStatus)
at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$6(ClusterTopologyManagerImpl.java:519)
at org.infinispan.topology.ClusterTopologyManagerImpl$$Lambda$966/1179375334.run(Unknown Source)
at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:144)
at org.infinispan.executors.LimitedExecutor.access$100(LimitedExecutor.java:33)
at org.infinispan.executors.LimitedExecutor$Runner.run(LimitedExecutor.java:174)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> HotRodMergeTest Timed out waiting for rebalancing to complete
> -------------------------------------------------------------
>
> Key: ISPN-8706
> URL: https://issues.jboss.org/browse/ISPN-8706
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.2.0.Alpha2
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
>
> The HotRodMergeTest is now consistently failing with the following error:
> {code}
> [OK: 321, KO: 1, SKIP: 0] Test failed: org.infinispan.server.hotrod.HotRodMergeTest.testNewTopologySentAfterCleanMerge[DIST_SYNC]
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node HotRodMergeTest[DIST_SYNC]-NodeA-31095, expected member list is [HotRodMergeTest[DIST_SYNC]-NodeA-31095, HotRodMergeTest[DIST_SYNC]-NodeB-33866], current member list is [HotRodMergeTest[DIST_SYNC]-NodeA-31095]!
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:388)
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:424)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.waitForPartitionToForm(BasePartitionHandlingTest.java:282)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.merge(BasePartitionHandlingTest.java:258)
> at org.infinispan.partitionhandling.BasePartitionHandlingTest$Partition.merge(BasePartitionHandlingTest.java:246)
> at org.infinispan.server.hotrod.HotRodMergeTest.testNewTopologySentAfterCleanMerge(HotRodMergeTest.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8724) HotRodRemoteStreamingIT hangs
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8724?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8724:
------------------------------------
Attachment: s1
s2
test
> HotRodRemoteStreamingIT hangs
> -----------------------------
>
> Key: ISPN-8724
> URL: https://issues.jboss.org/browse/ISPN-8724
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.2.0.CR1
> Reporter: Gustavo Fernandes
> Priority: Critical
> Attachments: s1, s2, test
>
>
> This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
> The surefire process waits forever for the two servers to stop.
> One of the servers is waiting for the initial state transfer to complete:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The other server is running a series of operations after the server stops, such
> as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
> MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
> {noformat}
> "MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
> at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
> at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
> at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
> - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
> - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> For some reason, also in the other server, the CounterModule lifecycle is reacting to a
> cache start event, and it's hanged as well:
> {noformat}
> "Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
> at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
> at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
> at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8724) HotRodRemoteStreamingIT hangs
by Gustavo Fernandes (JIRA)
Gustavo Fernandes created ISPN-8724:
---------------------------------------
Summary: HotRodRemoteStreamingIT hangs
Key: ISPN-8724
URL: https://issues.jboss.org/browse/ISPN-8724
Project: Infinispan
Issue Type: Bug
Affects Versions: 9.2.0.CR1
Reporter: Gustavo Fernandes
Priority: Critical
This was observed during the execution of test {{HotRodRemoteStreamingIT}}, Line 499 on method stopServer().
The surefire process waits forever for the two servers to stop.
One of the servers is waiting for the initial state transfer to complete:
{noformat}
"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
The other server is running a series of operations after the server stops, such
as {{EmbeddedCacheManager#undefineConfiguration}}, this can be seen on 3 threads:
MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3
{noformat}
"MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
- locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
- locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
For some reason, also in the other server, the CounterModule lifecycle is reacting to a
cache start event, and it's hanged as well:
{noformat}
"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8400) Adjust merge policies for JDG Online Services
by Ryan Emerson (JIRA)
[ https://issues.jboss.org/browse/ISPN-8400?page=com.atlassian.jira.plugin.... ]
Ryan Emerson resolved ISPN-8400.
--------------------------------
Fix Version/s: (was: 9.2.0.CR2)
Resolution: Done
> Adjust merge policies for JDG Online Services
> ---------------------------------------------
>
> Key: ISPN-8400
> URL: https://issues.jboss.org/browse/ISPN-8400
> Project: Infinispan
> Issue Type: Feature Request
> Components: Cloud
> Reporter: Sebastian Łaskawiec
> Assignee: Ryan Emerson
>
> Both Shared Memory and Caching Service require custom merge policies.
> In Caching Service we need to clear out all conflicted entries upon split brain. Shared Memory service might be a little bit more tricky and we might want to use some different strategy (but that needs to be checked out).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8717) JCache.put calls RemoteCache in forceReturnValues configuration
by Alon Greenland (JIRA)
[ https://issues.jboss.org/browse/ISPN-8717?page=com.atlassian.jira.plugin.... ]
Alon Greenland updated ISPN-8717:
---------------------------------
Summary: JCache.put calls RemoteCache in forceReturnValues configuration (was: Cache.put calls RemoteCache in forceReturnValues configuration)
> JCache.put calls RemoteCache in forceReturnValues configuration
> ---------------------------------------------------------------
>
> Key: ISPN-8717
> URL: https://issues.jboss.org/browse/ISPN-8717
> Project: Infinispan
> Issue Type: Bug
> Components: JCache
> Affects Versions: 9.2.0.CR1, 9.1.4.Final
> Reporter: Alon Greenland
>
> In JCache Hot Rod client, Cache.put() operation, which doesn't return previous value, calls underlying Hot Rod RemoteCache in force return values mode.
> This is inefficient.
> In the code, I can see that in JCache class there is [RemoteCache<K, V> cacheWithoutReturnValue field|https://github.com/infinispan/infinispan/blob/master/jcache/remote/...], but it is never being used!
>
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8708) Remote iteration with limited batch sends finished segments too early
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8708?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-8708:
-------------------------------------
Also to note that this sometimes fails on master as well, not just the netty branch.
> Remote iteration with limited batch sends finished segments too early
> ---------------------------------------------------------------------
>
> Key: ISPN-8708
> URL: https://issues.jboss.org/browse/ISPN-8708
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.2.0.Beta2
> Reporter: Radim Vansa
> Assignee: William Burns
> Fix For: 9.2.0.Final
>
> Attachments: infinispan-infinispan-client-hotrod.log.zip
>
>
> When a remote cache is iterated and we limit maximum number of entries in one response, we might get the list of finished segments before the entry is actually sent to the client.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-8708) Remote iteration with limited batch sends finished segments too early
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-8708?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-8708:
--------------------------------
Attachment: infinispan-infinispan-client-hotrod.log.zip
> Remote iteration with limited batch sends finished segments too early
> ---------------------------------------------------------------------
>
> Key: ISPN-8708
> URL: https://issues.jboss.org/browse/ISPN-8708
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.2.0.Beta2
> Reporter: Radim Vansa
> Assignee: William Burns
> Fix For: 9.2.0.Final
>
> Attachments: infinispan-infinispan-client-hotrod.log.zip
>
>
> When a remote cache is iterated and we limit maximum number of entries in one response, we might get the list of finished segments before the entry is actually sent to the client.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 3 months
- 1
- 0