[Red Hat JIRA] (ISPN-12430) AsyncNonBlockingStore can have many more modifications than modification queue size
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12430?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-12430:
------------------------------
Status: Open (was: New)
> AsyncNonBlockingStore can have many more modifications than modification queue size
> -----------------------------------------------------------------------------------
>
> Key: ISPN-12430
> URL: https://issues.redhat.com/browse/ISPN-12430
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Reporter: Will Burns
> Priority: Major
>
> The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map.
> We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12430) AsyncNonBlockingStore can have many more modifications than modification queue size
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12430?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-12430:
------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/8783
> AsyncNonBlockingStore can have many more modifications than modification queue size
> -----------------------------------------------------------------------------------
>
> Key: ISPN-12430
> URL: https://issues.redhat.com/browse/ISPN-12430
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Reporter: Will Burns
> Priority: Major
>
> The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map.
> We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12430) AsyncNonBlockingStore can have many more modifications than modification queue size
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12430?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-12430:
------------------------------
Fix Version/s: 12.0.0.Dev05
> AsyncNonBlockingStore can have many more modifications than modification queue size
> -----------------------------------------------------------------------------------
>
> Key: ISPN-12430
> URL: https://issues.redhat.com/browse/ISPN-12430
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
> Fix For: 12.0.0.Dev05
>
>
> The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map. https://github.com/ReactiveX/RxJava/blob/3.x/src/main/java/io/reactivex/r...
>
> We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12430) AsyncNonBlockingStore can have many more modifications than modification queue size
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12430?page=com.atlassian.jira.plugi... ]
Will Burns reassigned ISPN-12430:
---------------------------------
Assignee: Will Burns
> AsyncNonBlockingStore can have many more modifications than modification queue size
> -----------------------------------------------------------------------------------
>
> Key: ISPN-12430
> URL: https://issues.redhat.com/browse/ISPN-12430
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
>
> The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map.
> We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12430) AsyncNonBlockingStore can have many more modifications than modification queue size
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12430?page=com.atlassian.jira.plugi... ]
Will Burns updated ISPN-12430:
------------------------------
Description:
The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map. https://github.com/ReactiveX/RxJava/blob/3.x/src/main/java/io/reactivex/r...
We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
was:
The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map.
We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
> AsyncNonBlockingStore can have many more modifications than modification queue size
> -----------------------------------------------------------------------------------
>
> Key: ISPN-12430
> URL: https://issues.redhat.com/browse/ISPN-12430
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Reporter: Will Burns
> Assignee: Will Burns
> Priority: Major
>
> The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map. https://github.com/ReactiveX/RxJava/blob/3.x/src/main/java/io/reactivex/r...
>
> We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12430) AsyncNonBlockingStore can have many more modifications than modification queue size
by Will Burns (Jira)
Will Burns created ISPN-12430:
---------------------------------
Summary: AsyncNonBlockingStore can have many more modifications than modification queue size
Key: ISPN-12430
URL: https://issues.redhat.com/browse/ISPN-12430
Project: Infinispan
Issue Type: Bug
Components: Loaders and Stores
Reporter: Will Burns
The AsyncNonBlockingStore is known to allow for concurrent writes more than modification queue size. Unfortunately, due to how synchronized Publisher works it can be much much more if a given thread is delayed as other threads can sneak in. For example in the stress test with 1024 modification queue size I could get 6K entries in a given queue. The problem is that the concurrent publisher would add the value to its own queue which was only processed on the first thread, which means many could sneak in as no changes were actually populated in the map.
We should change this to be a bit more strict and limit the writes to only queue + # of threads with concurrent modifications.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12423) Infispan thread freezing (STUCK) - dead lock occured
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12423?page=com.atlassian.jira.plugi... ]
Will Burns commented on ISPN-12423:
-----------------------------------
A stack trace of it hung and the log together would show the best picture. Just make sure your log format includes the thread name :)
> Infispan thread freezing (STUCK) - dead lock occured
> ----------------------------------------------------
>
> Key: ISPN-12423
> URL: https://issues.redhat.com/browse/ISPN-12423
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.4.Final
> Reporter: Evgenii Balakhonov
> Priority: Major
> Attachments: Thread dump - heap storage.txt, Thread dump sync.txt, Thread dump.txt, cache-configuration-jdbc.xml, cache-configuration-jdbc.xml
>
>
> During huge load some threads hangs on
> java.lang.Thread.State: WAITING (parking)
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000005d2e654a8> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
> at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peek(OffHeapConcurrentMap.java:682)
> I attached Infinispan configuration and three thread dumps:
> * off heap storage enabled (Thread dump.txt)
> * heap storage enabled (Thread dump - heap storage.txt)
> * off heap storage enabled and replicated cache mode="SYNC" (thread dump sync.txt)
> Under high load, Infinspan freezes 100% of the cases.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months
[Red Hat JIRA] (ISPN-12423) Infispan thread freezing (STUCK) - dead lock occured
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12423?page=com.atlassian.jira.plugi... ]
Will Burns commented on ISPN-12423:
-----------------------------------
Just add
{code}
<Logger name="org.infinispan" level="TRACE"/>
{code} to your loggers.
> Infispan thread freezing (STUCK) - dead lock occured
> ----------------------------------------------------
>
> Key: ISPN-12423
> URL: https://issues.redhat.com/browse/ISPN-12423
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.4.Final
> Reporter: Evgenii Balakhonov
> Priority: Major
> Attachments: Thread dump - heap storage.txt, Thread dump sync.txt, Thread dump.txt, cache-configuration-jdbc.xml, cache-configuration-jdbc.xml
>
>
> During huge load some threads hangs on
> java.lang.Thread.State: WAITING (parking)
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000005d2e654a8> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
> at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peek(OffHeapConcurrentMap.java:682)
> I attached Infinispan configuration and three thread dumps:
> * off heap storage enabled (Thread dump.txt)
> * heap storage enabled (Thread dump - heap storage.txt)
> * off heap storage enabled and replicated cache mode="SYNC" (thread dump sync.txt)
> Under high load, Infinspan freezes 100% of the cases.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 2 months