[JBoss JIRA] (ISPN-10093) PersistenceManagerImpl stop deadlock with topology update
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10093?page=com.atlassian.jira.plugin... ]
Will Burns edited comment on ISPN-10093 at 5/13/19 12:50 PM:
-------------------------------------------------------------
Have to admit the stack thread here doesn't make much sense. The producer should be the same as the subscriber thread. The DummyInMemoryStore shouldn't be subscribing on a different thread, which means the subscriber should be publishing the entries on the invoking thread, so the fact that it is blocked is kinda weird.
was (Author: william.burns):
Have to admit the stack thread here doesn't make much sense. The producer should be the same as the subscriber thread. The DummyInMemoryStore shouldn't be subscribing on a different thread, which means the subscriber should be publishing the entries on the same thread, so the fact that it is blocked is kinda weird.
> PersistenceManagerImpl stop deadlock with topology update
> ---------------------------------------------------------
>
> Key: ISPN-10093
> URL: https://issues.jboss.org/browse/ISPN-10093
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
> Attachments: threaddump.txt
>
>
> {{DistSyncStoreNotSharedTest.clearContent}} hanged in CI recently:
> {noformat}
> "testng-DistSyncStoreNotSharedTest" #16 prio=5 os_prio=0 cpu=11511.26ms elapsed=435.14s tid=0x00007fdb710b6000 nid=0x3222 waiting on condition [0x00007fdb352d3000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8a22450> (a java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11/AbstractQueuedSynchronizer.java:885)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11/AbstractQueuedSynchronizer.java:1009)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11/AbstractQueuedSynchronizer.java:1324)
> at java.util.concurrent.Semaphore.acquireUninterruptibly(java.base@11/Semaphore.java:504)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.stop(PersistenceManagerImpl.java:222)
> at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(java.base@11/Method.java:566)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$237/0x0000000100661c40.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:431)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:366)
> at org.infinispan.cache.impl.CacheImpl.performImmediateShutdown(CacheImpl.java:1160)
> at org.infinispan.cache.impl.CacheImpl.stop(CacheImpl.java:1125)
> at org.infinispan.cache.impl.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:521)
> at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:747)
> at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:799)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:846)
> at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:158)
> "persistence-thread-DistSyncStoreNotSharedTest-NodeB-p16432-t1" #53654 daemon prio=5 os_prio=0 cpu=1.26ms elapsed=301.93s tid=0x00007fdb3c3d8000 nid=0x8ef waiting on condition [0x00007fdb00055000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8b1fb88> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11/AbstractQueuedSynchronizer.java:885)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11/AbstractQueuedSynchronizer.java:1009)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11/AbstractQueuedSynchronizer.java:1324)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11/ReentrantReadWriteLock.java:738)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.pollStoreAvailability(PersistenceManagerImpl.java:196)
> at org.infinispan.persistence.manager.PersistenceManagerImpl$$Lambda$492/0x00000001007fb440.run(Unknown Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11/Executors.java:515)
> at java.util.concurrent.FutureTask.runAndReset(java.base@11/FutureTask.java:305)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11/ScheduledThreadPoolExecutor.java:305)
> "transport-thread-DistSyncStoreNotSharedTest-NodeB-p16424-t5" #53646 daemon prio=5 os_prio=0 cpu=3.15ms elapsed=301.94s tid=0x00007fdb2007a000 nid=0x8e8 waiting on condition [0x00007fdb0b406000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8d2abb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11/AbstractQueuedSynchronizer.java:2081)
> at io.reactivex.internal.operators.flowable.BlockingFlowableIterable$BlockingFlowableIterator.hasNext(BlockingFlowableIterable.java:94)
> at io.reactivex.Flowable.blockingForEach(Flowable.java:5682)
> at org.infinispan.statetransfer.StateConsumerImpl.removeStaleData(StateConsumerImpl.java:1011)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:453)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:202)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:58)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:114)
> at org.infinispan.topology.LocalTopologyManagerImpl.resetLocalTopologyBeforeRebalance(LocalTopologyManagerImpl.java:437)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:519)
> - locked <0x00000000c8b30b30> (a org.infinispan.topology.LocalCacheStatus)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:484)
> at org.infinispan.topology.LocalTopologyManagerImpl$$Lambda$574/0x000000010089a040.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175){noformat}
> [Full thread dump|https://ci.infinispan.org/job/Infinispan/job/master/1133/artifact/core/]
> Somehow the producer thread for the transport-thread iteration is blocked, but without waiting for the persistence mutex. Maybe it's waiting for a topology? Not sure if it's relevant, but the last test to run was {{testClearWithFlag}}, so the data container was empty and the store had 5 entries.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (ISPN-10093) PersistenceManagerImpl stop deadlock with topology update
by Will Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10093?page=com.atlassian.jira.plugin... ]
Will Burns commented on ISPN-10093:
-----------------------------------
Have to admit the stack thread here doesn't make much sense. The producer should be the same as the subscriber thread. The DummyInMemoryStore shouldn't be subscribing on a different thread, which means the subscriber should be publishing the entries on the same thread, so the fact that it is blocked is kinda weird.
> PersistenceManagerImpl stop deadlock with topology update
> ---------------------------------------------------------
>
> Key: ISPN-10093
> URL: https://issues.jboss.org/browse/ISPN-10093
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
> Attachments: threaddump.txt
>
>
> {{DistSyncStoreNotSharedTest.clearContent}} hanged in CI recently:
> {noformat}
> "testng-DistSyncStoreNotSharedTest" #16 prio=5 os_prio=0 cpu=11511.26ms elapsed=435.14s tid=0x00007fdb710b6000 nid=0x3222 waiting on condition [0x00007fdb352d3000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8a22450> (a java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11/AbstractQueuedSynchronizer.java:885)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11/AbstractQueuedSynchronizer.java:1009)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11/AbstractQueuedSynchronizer.java:1324)
> at java.util.concurrent.Semaphore.acquireUninterruptibly(java.base@11/Semaphore.java:504)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.stop(PersistenceManagerImpl.java:222)
> at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(java.base@11/Method.java:566)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> at org.infinispan.commons.util.SecurityActions$$Lambda$237/0x0000000100661c40.run(Unknown Source)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:431)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:366)
> at org.infinispan.cache.impl.CacheImpl.performImmediateShutdown(CacheImpl.java:1160)
> at org.infinispan.cache.impl.CacheImpl.stop(CacheImpl.java:1125)
> at org.infinispan.cache.impl.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:521)
> at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:747)
> at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:799)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:775)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:846)
> at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:158)
> "persistence-thread-DistSyncStoreNotSharedTest-NodeB-p16432-t1" #53654 daemon prio=5 os_prio=0 cpu=1.26ms elapsed=301.93s tid=0x00007fdb3c3d8000 nid=0x8ef waiting on condition [0x00007fdb00055000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8b1fb88> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11/AbstractQueuedSynchronizer.java:885)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11/AbstractQueuedSynchronizer.java:1009)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11/AbstractQueuedSynchronizer.java:1324)
> at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11/ReentrantReadWriteLock.java:738)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.pollStoreAvailability(PersistenceManagerImpl.java:196)
> at org.infinispan.persistence.manager.PersistenceManagerImpl$$Lambda$492/0x00000001007fb440.run(Unknown Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11/Executors.java:515)
> at java.util.concurrent.FutureTask.runAndReset(java.base@11/FutureTask.java:305)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11/ScheduledThreadPoolExecutor.java:305)
> "transport-thread-DistSyncStoreNotSharedTest-NodeB-p16424-t5" #53646 daemon prio=5 os_prio=0 cpu=3.15ms elapsed=301.94s tid=0x00007fdb2007a000 nid=0x8e8 waiting on condition [0x00007fdb0b406000]
> java.lang.Thread.State: WAITING (parking)
> at jdk.internal.misc.Unsafe.park(java.base@11/Native Method)
> - parking to wait for <0x00000000c8d2abb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(java.base@11/LockSupport.java:194)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11/AbstractQueuedSynchronizer.java:2081)
> at io.reactivex.internal.operators.flowable.BlockingFlowableIterable$BlockingFlowableIterator.hasNext(BlockingFlowableIterable.java:94)
> at io.reactivex.Flowable.blockingForEach(Flowable.java:5682)
> at org.infinispan.statetransfer.StateConsumerImpl.removeStaleData(StateConsumerImpl.java:1011)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:453)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:202)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:58)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:114)
> at org.infinispan.topology.LocalTopologyManagerImpl.resetLocalTopologyBeforeRebalance(LocalTopologyManagerImpl.java:437)
> at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:519)
> - locked <0x00000000c8b30b30> (a org.infinispan.topology.LocalCacheStatus)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$3(LocalTopologyManagerImpl.java:484)
> at org.infinispan.topology.LocalTopologyManagerImpl$$Lambda$574/0x000000010089a040.run(Unknown Source)
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175){noformat}
> [Full thread dump|https://ci.infinispan.org/job/Infinispan/job/master/1133/artifact/core/]
> Somehow the producer thread for the transport-thread iteration is blocked, but without waiting for the persistence mutex. Maybe it's waiting for a topology? Not sure if it's relevant, but the last test to run was {{testClearWithFlag}}, so the data container was empty and the store had 5 entries.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (ISPN-10180) Automatically take a site offline if the Cross-Site is in ASYNC mode
by Pedro Ruivo (Jira)
[ https://issues.jboss.org/browse/ISPN-10180?page=com.atlassian.jira.plugin... ]
Pedro Ruivo updated ISPN-10180:
-------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6907
> Automatically take a site offline if the Cross-Site is in ASYNC mode
> --------------------------------------------------------------------
>
> Key: ISPN-10180
> URL: https://issues.jboss.org/browse/ISPN-10180
> Project: Infinispan
> Issue Type: Feature Request
> Components: Cross-Site Replication
> Affects Versions: 10.0.0.Beta3, 9.4.13.Final
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
>
> With a SYNC XSite replication, it is possible to automatically mark a site offline with the following configuration
> {code:xml}
> <distributed-cache name="XSiteCache" ....>
> ...
> <backups>
> <backup site="site02" failure-policy="IGNORE" strategy="SYNC">
> <take-offline after-failures="10" min-wait="60000"/>
> </backup>
> </backups>
> </distributed-cache>
> {code}
> If the strategy is ASYNC (i.e. for performance reason) the element take-offline will have no effect as the site will 'fire and forget' the updates and don't care about failure.
> It should be possible to use the take-offline configuration in ASYNC mode as well
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (ISPN-10186) Error stopping cache manager
by Jive JIRA Integration (Jira)
[ https://issues.jboss.org/browse/ISPN-10186?page=com.atlassian.jira.plugin... ]
Jive JIRA Integration updated ISPN-10186:
-----------------------------------------
Forum Reference: https://developer.jboss.org/message/988970#988970
> Error stopping cache manager
> ----------------------------
>
> Key: ISPN-10186
> URL: https://issues.jboss.org/browse/ISPN-10186
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 10.0.0.Beta3
> Reporter: Galder Zamarreño
> Priority: Major
> Fix For: 10.0.0.Final
>
>
> {code}
> [m [1;31m09:44:50,038 ERROR org.infinispan.factories.impl.BasicComponentRegistryImpl [] - Error stopping component org.infinispan.counter.api.CounterManager
> org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.stop() on object of type EmbeddedCounterManager
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:431)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:366)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:777)
> at lucom.ip.web.context.WebApplicationInitializer.destroy(WebApplicationInitializer.java:815)
> at lucom.ip.web.context.WebApplicationInitializer.contextDestroyed(WebApplicationInitializer.java:711)
> at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4624)
> at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5263)
> at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
> at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1443)
> at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1432)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
> at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:999)
> at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
> at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1443)
> at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1432)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
> at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:999)
> at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
> at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:471)
> at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
> at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:791)
> at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
> at org.apache.catalina.startup.Catalina.stop(Catalina.java:755)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:717)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
> Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
> at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:68)
> at org.infinispan.notifications.Listenable.removeListener(Listenable.java:43)
> at org.infinispan.cache.impl.AbstractDelegatingCache.removeListener(AbstractDelegatingCache.java:561)
> at org.infinispan.counter.impl.listener.CounterManagerNotificationManager.stop(CounterManagerNotificationManager.java:130)
> at org.infinispan.counter.impl.manager.EmbeddedCounterManager.stop(EmbeddedCounterManager.java:101)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> ... 40 more
> Caused by: org.infinispan.commons.CacheException: org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.lambda$removeListenerAsync$16(CacheNotifierImpl.java:2039)
> at org.infinispan.manager.impl.LocalClusterExecutor.lambda$submitConsumer$3(LocalClusterExecutor.java:78)
> at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at org.infinispan.manager.impl.LocalClusterExecutor.lambda$localInvocation$6(LocalClusterExecutor.java:97)
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:212)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
> at org.infinispan.manager.DefaultCacheManager.assertIsNotTerminated(DefaultCacheManager.java:1023)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:474)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:470)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:456)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:188)
> at org.infinispan.manager.impl.UnwrappingEmbeddedCacheManager.getCache(UnwrappingEmbeddedCacheManager.java:25)
> at org.infinispan.notifications.cachelistener.cluster.ClusterListenerRemoveCallable.apply(ClusterListenerRemoveCallable.java:39)
> at org.infinispan.notifications.cachelistener.cluster.ClusterListenerRemoveCallable.apply(ClusterListenerRemoveCallable.java:25)
> at org.infinispan.manager.impl.LocalClusterExecutor.lambda$localInvocation$6(LocalClusterExecutor.java:94)
> ... 4 more
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (ISPN-10186) Error stopping cache manager
by Galder Zamarreño (Jira)
Galder Zamarreño created ISPN-10186:
---------------------------------------
Summary: Error stopping cache manager
Key: ISPN-10186
URL: https://issues.jboss.org/browse/ISPN-10186
Project: Infinispan
Issue Type: Bug
Affects Versions: 10.0.0.Beta3
Reporter: Galder Zamarreño
Fix For: 10.0.0.Final
{code}
[m [1;31m09:44:50,038 ERROR org.infinispan.factories.impl.BasicComponentRegistryImpl [] - Error stopping component org.infinispan.counter.api.CounterManager
org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.counter.impl.manager.EmbeddedCounterManager.stop() on object of type EmbeddedCounterManager
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:601)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:590)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:431)
at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:366)
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:777)
at lucom.ip.web.context.WebApplicationInitializer.destroy(WebApplicationInitializer.java:815)
at lucom.ip.web.context.WebApplicationInitializer.contextDestroyed(WebApplicationInitializer.java:711)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4624)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5263)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1443)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1432)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:999)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1443)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1432)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:999)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:471)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:791)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:755)
at org.apache.catalina.startup.Catalina.start(Catalina.java:717)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:68)
at org.infinispan.notifications.Listenable.removeListener(Listenable.java:43)
at org.infinispan.cache.impl.AbstractDelegatingCache.removeListener(AbstractDelegatingCache.java:561)
at org.infinispan.counter.impl.listener.CounterManagerNotificationManager.stop(CounterManagerNotificationManager.java:130)
at org.infinispan.counter.impl.manager.EmbeddedCounterManager.stop(EmbeddedCounterManager.java:101)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
... 40 more
Caused by: org.infinispan.commons.CacheException: org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
at org.infinispan.notifications.cachelistener.CacheNotifierImpl.lambda$removeListenerAsync$16(CacheNotifierImpl.java:2039)
at org.infinispan.manager.impl.LocalClusterExecutor.lambda$submitConsumer$3(LocalClusterExecutor.java:78)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at org.infinispan.manager.impl.LocalClusterExecutor.lambda$localInvocation$6(LocalClusterExecutor.java:97)
at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:212)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.infinispan.IllegalLifecycleStateException: Cache container has been stopped and cannot be reused. Recreate the cache container.
at org.infinispan.manager.DefaultCacheManager.assertIsNotTerminated(DefaultCacheManager.java:1023)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:474)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:470)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:456)
at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:188)
at org.infinispan.manager.impl.UnwrappingEmbeddedCacheManager.getCache(UnwrappingEmbeddedCacheManager.java:25)
at org.infinispan.notifications.cachelistener.cluster.ClusterListenerRemoveCallable.apply(ClusterListenerRemoveCallable.java:39)
at org.infinispan.notifications.cachelistener.cluster.ClusterListenerRemoveCallable.apply(ClusterListenerRemoveCallable.java:25)
at org.infinispan.manager.impl.LocalClusterExecutor.lambda$localInvocation$6(LocalClusterExecutor.java:94)
... 4 more
{code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (ISPN-10185) hotrod-client tests hang on Windows
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10185?page=com.atlassian.jira.plugin... ]
Dan Berindei commented on ISPN-10185:
-------------------------------------
If the problem is indeed not enough {{ForkJoinPool.commonPool}} threads, we should be able to reproduce it with {{-Djava.util.concurrent.ForkJoinPool.common.parallelism=1}}.
We should also reduce the cache size in {{BaseIterationFailOverTest.testFailOver}}: with 10k entries it's more like a stress test than a functional test.
> hotrod-client tests hang on Windows
> -----------------------------------
>
> Key: ISPN-10185
> URL: https://issues.jboss.org/browse/ISPN-10185
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.4.13.Final
> Reporter: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
> Attachments: ReplFailOverRemoteIteratorTest.txt
>
>
> All the {{ForkJoin.commonPool}} in the thread dump are busy doing blocking operations:
> {noformat}
> "ForkJoinPool.commonPool-worker-1" #66 prio=0 tid=0x42 nid=NA timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x75d9efe2> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at org.infinispan.client.hotrod.impl.Util.await(Util.java:21)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:335)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at org.infinispan.client.hotrod.impl.iteration.AbstractRemoteIteratorTest.lambda$populateCache$0(AbstractRemoteIteratorTest.java:34)
> {noformat}
> {{ForkJoinPool.commonPool}} is supposed to add new threads to maintain the default parallelism level (= number of cpus), but that doesn't seem to work because {{RemoteCacheManagerTest}} is trying to start a cache and it didn't get a {{commonPool}} thread:
> {noformat}
> "testng-RemoteCacheManagerTest" #61 prio=0 tid=0x3d nid=NA waiting
> java.lang.Thread.State: WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x23a92f0a> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.infinispan.client.hotrod.RemoteCacheManagerTest.testStartStopAsync(RemoteCacheManagerTest.java:55)
> {noformat}
> Server threads are all free. The first test to time out was {{ProtobufRemoteIteratorIndexingTest}}, but most put operations on {{ForkJoinPool.commonPool}} are issued by {{ReplFailOverRemoteIteratorTest}}:
> {noformat}
> "testng-ReplFailOverRemoteIteratorTest" #62 prio=0 tid=0x3e nid=NA waiting
> java.lang.Thread.State: WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <0x404a8e70> (a java.util.stream.ForEachOps$ForEachTask)
> at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
> at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
> at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
> at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
> at java.util.stream.ForEachOps$ForEachOp$OfInt.evaluateParallel(ForEachOps.java:189)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
> at java.util.stream.IntPipeline.forEach(IntPipeline.java:404)
> at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:560)
> at org.infinispan.client.hotrod.impl.iteration.AbstractRemoteIteratorTest.populateCache(AbstractRemoteIteratorTest.java:34)
> at org.infinispan.client.hotrod.impl.iteration.BaseIterationFailOverTest.testFailOver(BaseIterationFailOverTest.java:38)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (ISPN-10185) hotrod-client tests hang on Windows
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10185?page=com.atlassian.jira.plugin... ]
Dan Berindei reassigned ISPN-10185:
-----------------------------------
Assignee: Dan Berindei
> hotrod-client tests hang on Windows
> -----------------------------------
>
> Key: ISPN-10185
> URL: https://issues.jboss.org/browse/ISPN-10185
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.4.13.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
> Attachments: ReplFailOverRemoteIteratorTest.txt
>
>
> All the {{ForkJoin.commonPool}} in the thread dump are busy doing blocking operations:
> {noformat}
> "ForkJoinPool.commonPool-worker-1" #66 prio=0 tid=0x42 nid=NA timed_waiting
> java.lang.Thread.State: TIMED_WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x75d9efe2> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at org.infinispan.client.hotrod.impl.Util.await(Util.java:21)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:335)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at org.infinispan.client.hotrod.impl.iteration.AbstractRemoteIteratorTest.lambda$populateCache$0(AbstractRemoteIteratorTest.java:34)
> {noformat}
> {{ForkJoinPool.commonPool}} is supposed to add new threads to maintain the default parallelism level (= number of cpus), but that doesn't seem to work because {{RemoteCacheManagerTest}} is trying to start a cache and it didn't get a {{commonPool}} thread:
> {noformat}
> "testng-RemoteCacheManagerTest" #61 prio=0 tid=0x3d nid=NA waiting
> java.lang.Thread.State: WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x23a92f0a> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at org.infinispan.client.hotrod.RemoteCacheManagerTest.testStartStopAsync(RemoteCacheManagerTest.java:55)
> {noformat}
> Server threads are all free. The first test to time out was {{ProtobufRemoteIteratorIndexingTest}}, but most put operations on {{ForkJoinPool.commonPool}} are issued by {{ReplFailOverRemoteIteratorTest}}:
> {noformat}
> "testng-ReplFailOverRemoteIteratorTest" #62 prio=0 tid=0x3e nid=NA waiting
> java.lang.Thread.State: WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <0x404a8e70> (a java.util.stream.ForEachOps$ForEachTask)
> at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
> at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
> at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
> at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
> at java.util.stream.ForEachOps$ForEachOp$OfInt.evaluateParallel(ForEachOps.java:189)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
> at java.util.stream.IntPipeline.forEach(IntPipeline.java:404)
> at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:560)
> at org.infinispan.client.hotrod.impl.iteration.AbstractRemoteIteratorTest.populateCache(AbstractRemoteIteratorTest.java:34)
> at org.infinispan.client.hotrod.impl.iteration.BaseIterationFailOverTest.testFailOver(BaseIterationFailOverTest.java:38)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months