[Red Hat JIRA] (ISPN-12423) Infispan thread freezing (STUCK) - dead lock occured
by Evgenii Balakhonov (Jira)
[ https://issues.redhat.com/browse/ISPN-12423?page=com.atlassian.jira.plugi... ]
Evgenii Balakhonov commented on ISPN-12423:
-------------------------------------------
Please write me how to correctly enable TRACE level for Infinispan. I use Log4J2 in my app without Sf4J
Both caches are same, but in testing I use only first cache - GS46CACHE_GS46NEW
> Infispan thread freezing (STUCK) - dead lock occured
> ----------------------------------------------------
>
> Key: ISPN-12423
> URL: https://issues.redhat.com/browse/ISPN-12423
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.4.Final
> Reporter: Evgenii Balakhonov
> Priority: Major
> Attachments: Thread dump - heap storage.txt, Thread dump sync.txt, Thread dump.txt, cache-configuration-jdbc.xml, cache-configuration-jdbc.xml
>
>
> During huge load some threads hangs on
> java.lang.Thread.State: WAITING (parking)
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000005d2e654a8> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
> at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peek(OffHeapConcurrentMap.java:682)
> I attached Infinispan configuration and three thread dumps:
> * off heap storage enabled (Thread dump.txt)
> * heap storage enabled (Thread dump - heap storage.txt)
> * off heap storage enabled and replicated cache mode="SYNC" (thread dump sync.txt)
> Under high load, Infinspan freezes 100% of the cases.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[Red Hat JIRA] (ISPN-12423) Infispan thread freezing (STUCK) - dead lock occured
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12423?page=com.atlassian.jira.plugi... ]
Will Burns commented on ISPN-12423:
-----------------------------------
Also just to confirm there are 2 caches in the xml file you posted. Which one is the one that was having the issue?
> Infispan thread freezing (STUCK) - dead lock occured
> ----------------------------------------------------
>
> Key: ISPN-12423
> URL: https://issues.redhat.com/browse/ISPN-12423
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.4.Final
> Reporter: Evgenii Balakhonov
> Priority: Major
> Attachments: Thread dump - heap storage.txt, Thread dump sync.txt, Thread dump.txt, cache-configuration-jdbc.xml, cache-configuration-jdbc.xml
>
>
> During huge load some threads hangs on
> java.lang.Thread.State: WAITING (parking)
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000005d2e654a8> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
> at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peek(OffHeapConcurrentMap.java:682)
> I attached Infinispan configuration and three thread dumps:
> * off heap storage enabled (Thread dump.txt)
> * heap storage enabled (Thread dump - heap storage.txt)
> * off heap storage enabled and replicated cache mode="SYNC" (thread dump sync.txt)
> Under high load, Infinspan freezes 100% of the cases.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[Red Hat JIRA] (ISPN-12423) Infispan thread freezing (STUCK) - dead lock occured
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-12423?page=com.atlassian.jira.plugi... ]
Will Burns commented on ISPN-12423:
-----------------------------------
Is it possible you could run with org.infinispan set to TRACE level? I know it will be a really big file, but maybe just keep the last bit of it around right before it started to hang?
> Infispan thread freezing (STUCK) - dead lock occured
> ----------------------------------------------------
>
> Key: ISPN-12423
> URL: https://issues.redhat.com/browse/ISPN-12423
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.4.Final
> Reporter: Evgenii Balakhonov
> Priority: Major
> Attachments: Thread dump - heap storage.txt, Thread dump sync.txt, Thread dump.txt, cache-configuration-jdbc.xml, cache-configuration-jdbc.xml
>
>
> During huge load some threads hangs on
> java.lang.Thread.State: WAITING (parking)
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000005d2e654a8> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
> at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peek(OffHeapConcurrentMap.java:682)
> I attached Infinispan configuration and three thread dumps:
> * off heap storage enabled (Thread dump.txt)
> * heap storage enabled (Thread dump - heap storage.txt)
> * off heap storage enabled and replicated cache mode="SYNC" (thread dump sync.txt)
> Under high load, Infinspan freezes 100% of the cases.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months
[Red Hat JIRA] (ISPN-12423) Infispan thread freezing (STUCK) - dead lock occured
by Evgenii Balakhonov (Jira)
[ https://issues.redhat.com/browse/ISPN-12423?page=com.atlassian.jira.plugi... ]
Evgenii Balakhonov commented on ISPN-12423:
-------------------------------------------
Will Burns,
Now I tries to switch on old binary memory storage mode:
<memory max-size="10GB" when-full="REMOVE" storage="BINARY"/>
And in this mode Infinispan works reliably :-\
But if I return back storage="OFF_HEAP" then I have after 30-40 minutes high load:
"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" #31 daemon prio=5 os_prio=0 tid=0x00007f5d8c9a4800 nid=0x2005 waiting on condition [0x00007f5d7e184000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005d72b1140> (a java.util.concurrent.locks.StampedLock)
at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> Infispan thread freezing (STUCK) - dead lock occured
> ----------------------------------------------------
>
> Key: ISPN-12423
> URL: https://issues.redhat.com/browse/ISPN-12423
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 11.0.4.Final
> Reporter: Evgenii Balakhonov
> Priority: Major
> Attachments: Thread dump - heap storage.txt, Thread dump sync.txt, Thread dump.txt, cache-configuration-jdbc.xml, cache-configuration-jdbc.xml
>
>
> During huge load some threads hangs on
> java.lang.Thread.State: WAITING (parking)
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000005d2e654a8> (a java.util.concurrent.locks.StampedLock)
> at java.util.concurrent.locks.StampedLock.acquireRead(StampedLock.java:1215)
> at java.util.concurrent.locks.StampedLock.readLock(StampedLock.java:428)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peekOrGet(OffHeapConcurrentMap.java:615)
> at org.infinispan.container.offheap.OffHeapConcurrentMap.peek(OffHeapConcurrentMap.java:682)
> I attached Infinispan configuration and three thread dumps:
> * off heap storage enabled (Thread dump.txt)
> * heap storage enabled (Thread dump - heap storage.txt)
> * off heap storage enabled and replicated cache mode="SYNC" (thread dump sync.txt)
> Under high load, Infinspan freezes 100% of the cases.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 4 months