[JBoss JIRA] (ISPN-11524) PersistenceManagerImpl locks should be able to block
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-11524?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-11524:
--------------------------------
Status: Open (was: New)
> PersistenceManagerImpl locks should be able to block
> ----------------------------------------------------
>
> Key: ISPN-11524
> URL: https://issues.redhat.com/browse/ISPN-11524
> Project: Infinispan
> Issue Type: Sub-task
> Reporter: Will Burns
> Priority: Major
> Fix For: 11.0.0.Dev05
>
>
> Here is an example stack trace acquired
> {code}
> java.lang.AssertionError: Blocking call! jdk.internal.misc.Unsafe#park on thread Thread[non-blocking-thread-SharedStoreTest-NodeB-p19754-t5,5,main]
> at org.infinispan.util.CoreTestBlockHoundIntegration.lambda$applyTo$0(CoreTestBlockHoundIntegration.java:43)
> at reactor.blockhound.BlockHound$Builder.lambda$install$6(BlockHound.java:318)
> at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:46)
> at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
> at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:1009)
> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1324)
> at java.base/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:738)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.getFirstSegmentedStore(PersistenceManagerImpl.java:665)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.publishEntries(PersistenceManagerImpl.java:704)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.publishEntries(PersistenceManagerImpl.java:121)
> at org.infinispan.statetransfer.StateProviderImpl.publishStoreEntries(StateProviderImpl.java:307)
> at org.infinispan.scattered.impl.ScatteredStateProviderImpl.replicateAndInvalidate(ScatteredStateProviderImpl.java:94)
> at org.infinispan.scattered.impl.ScatteredStateProviderImpl.onTopologyUpdate(ScatteredStateProviderImpl.java:53)
> at org.infinispan.statetransfer.StateTransferManagerImpl.updateProviderAndConsumer(StateTransferManagerImpl.java:202)
> at org.infinispan.statetransfer.StateTransferManagerImpl.lambda$doTopologyUpdate$0(StateTransferManagerImpl.java:188)
> at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:187)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:66)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:124)
> at org.infinispan.topology.LocalTopologyManagerImpl.lambda$doHandleRebalance$19(LocalTopologyManagerImpl.java:578)
> at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
> at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:840)
> at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (ISPN-11729) REST endpoint to handle rebalancing per cache
by Katia Aresti (Jira)
Katia Aresti created ISPN-11729:
-----------------------------------
Summary: REST endpoint to handle rebalancing per cache
Key: ISPN-11729
URL: https://issues.redhat.com/browse/ISPN-11729
Project: Infinispan
Issue Type: Feature Request
Components: REST
Affects Versions: 10.1.6.Final
Reporter: Katia Aresti
Assignee: Tristan Tarrant
Add a Rest endpoint to the CacheV2Resource to enable/disable rebalancing for a cache
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (ISPN-11728) REST endpoint to handle rebalancing at cluster level
by Katia Aresti (Jira)
Katia Aresti created ISPN-11728:
-----------------------------------
Summary: REST endpoint to handle rebalancing at cluster level
Key: ISPN-11728
URL: https://issues.redhat.com/browse/ISPN-11728
Project: Infinispan
Issue Type: Feature Request
Components: REST
Affects Versions: 10.1.6.Final
Reporter: Katia Aresti
Assignee: Tristan Tarrant
Add a REST endpoint in the CacheManagerResource to enable and disable rebalancing at cache manager level
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (ISPN-11727) Async Cache Writer is blocking
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11727?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-11727:
------------------------------
Description:
The current AsyncCacheWriter is blocking on a few different levels.
It first has a lock to add a modification to its queue, which is able to be configured by the end user. It may even be possible to configure a value so low that if a bulkUpdate is done that is larger than the configured value it may hang (needs to be confirmed).
However this lock can and will block if an async update takes too long and the queue is full of writes. We should at least return a future that is complete when the value is written to the queue instead.
We may want to even use something like https://github.com/IBM/java-async-util with their AsyncSemaphore to handle this queueing in a non blocking way. This however will require https://issues.redhat.com/browse/ISPN-10373 to be implemented first, before it can be made fully non blocking.
was:
The current AsyncCacheWriter is blocking on a few different levels.
It first has a lock to add a modification to its queue, which is able to be configured by the end user. It may even be possible to configure a value so low that if a bulkUpdate is done that is larger than the configured value it may hang (needs to be confirmed).
However this lock can and will block if an async update takes too long and the queue is full of writes. We should at least return a future that is complete when the value is written to the queue instead.
We may want to even use something like https://github.com/IBM/java-async-util with their AsyncSemaphore to handle this queueing in a non blocking way.
> Async Cache Writer is blocking
> ------------------------------
>
> Key: ISPN-11727
> URL: https://issues.redhat.com/browse/ISPN-11727
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: Will Burns
> Priority: Major
>
> The current AsyncCacheWriter is blocking on a few different levels.
> It first has a lock to add a modification to its queue, which is able to be configured by the end user. It may even be possible to configure a value so low that if a bulkUpdate is done that is larger than the configured value it may hang (needs to be confirmed).
> However this lock can and will block if an async update takes too long and the queue is full of writes. We should at least return a future that is complete when the value is written to the queue instead.
> We may want to even use something like https://github.com/IBM/java-async-util with their AsyncSemaphore to handle this queueing in a non blocking way. This however will require https://issues.redhat.com/browse/ISPN-10373 to be implemented first, before it can be made fully non blocking.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (ISPN-11727) Async Cache Writer is blocking
by Will Burns (Jira)
Will Burns created ISPN-11727:
---------------------------------
Summary: Async Cache Writer is blocking
Key: ISPN-11727
URL: https://issues.redhat.com/browse/ISPN-11727
Project: Infinispan
Issue Type: Enhancement
Components: Loaders and Stores
Reporter: Will Burns
The current AsyncCacheWriter is blocking on a few different levels.
It first has a lock to add a modification to its queue, which is able to be configured by the end user. It may even be possible to configure a value so low that if a bulkUpdate is done that is larger than the configured value it may hang (needs to be confirmed).
However this lock can and will block if an async update takes too long and the queue is full of writes. We should at least return a future that is complete when the value is written to the queue instead.
We may want to even use something like https://github.com/IBM/java-async-util with their AsyncSemaphore to handle this queueing in a non blocking way.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years
[JBoss JIRA] (ISPN-11726) BlockingLocalTopologyManager is blocking - ignore in BlockHound
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11726?page=com.atlassian.jira.plugi... ]
Will Burns commented on ISPN-11726:
-----------------------------------
Example error:
{code}
<failure type="java.lang.AssertionError" message="Blocking call! jdk.internal.misc.Unsafe#park on thread Thread[non-blocking-thread-RemoteGetDuringStateTransferTest-NodeA-p30805-t4,5,ISPN-non-blocking-thread-group]">
<![CDATA[java.lang.AssertionError: Blocking call! jdk.internal.misc.Unsafe#park on thread Thread[non-blocking-thread-RemoteGetDuringStateTransferTest-NodeA-p30805-t4,5,ISPN-non-blocking-thread-group]
at org.infinispan.util.CoreTestBlockHoundIntegration.lambda$applyTo$0(CoreTestBlockHoundIntegration.java:44)
at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:383)
at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
at java.base/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
at java.base/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:173)
at java.base/java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:421)
at java.base/java.util.AbstractQueue.add(AbstractQueue.java:95)
at org.infinispan.util.BlockingLocalTopologyManager.beforeConfirmRebalancePhase(BlockingLocalTopologyManager.java:218)
at org.infinispan.util.AbstractControlledLocalTopologyManager.confirmRebalancePhase(AbstractControlledLocalTopologyManager.java:70)
at org.infinispan.statetransfer.StateTransferManagerImpl.lambda$updateProviderAndConsumer$3(StateTransferManagerImpl.java:209)
at java.base/java.util.concurrent.CompletableFuture.biRun(CompletableFuture.java:1375)
at java.base/java.util.concurrent.CompletableFuture.biRunStage(CompletableFuture.java:1393)
at java.base/java.util.concurrent.CompletableFuture.runAfterBoth(CompletableFuture.java:2171)
at java.base/java.util.concurrent.CompletableFuture.runAfterBoth(CompletableFuture.java:143)
at org.infinispan.statetransfer.StateTransferManagerImpl.updateProviderAndConsumer(StateTransferManagerImpl.java:203)
at org.infinispan.statetransfer.StateTransferManagerImpl.lambda$doTopologyUpdate$0(StateTransferManagerImpl.java:188)
at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:187)
at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:66)
at org.infinispan.statetransfer.StateTransferManagerImpl$1.rebalance(StateTransferManagerImpl.java:124)
at org.infinispan.topology.LocalTopologyManagerImpl.lambda$doHandleRebalance$19(LocalTopologyManagerImpl.java:578)
at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
at org.infinispan.topology.LocalTopologyManagerImpl.doHandleRebalance(LocalTopologyManagerImpl.java:574)
at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleRebalance$16(LocalTopologyManagerImpl.java:532)
at org.infinispan.topology.LocalTopologyManagerImpl.lambda$orderOnCache$24(LocalTopologyManagerImpl.java:737)
at org.infinispan.util.concurrent.ActionSequencer.safeNonBlockingCall(ActionSequencer.java:57)
at org.infinispan.util.concurrent.ActionSequencer.access$400(ActionSequencer.java:32)
at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.run(ActionSequencer.java:226)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
> BlockingLocalTopologyManager is blocking - ignore in BlockHound
> ---------------------------------------------------------------
>
> Key: ISPN-11726
> URL: https://issues.redhat.com/browse/ISPN-11726
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 11.0.0.Dev05
>
>
> BlockingLocalTopologyManager blocks all over to confirm things during tests. We should add blockhound exclusions for it.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years