[JBoss JIRA] (ISPN-10346) BlockingTaskAwareExecutorServiceImpl may leak controller thread
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10346?page=com.atlassian.jira.plugin... ]
Dan Berindei commented on ISPN-10346:
-------------------------------------
The thread leak was actually caused by one of my changes for ISPN-10330. I thought it would be a good idea to not start the dependencies of factories too early, but I didn't realize that {{NamedExecutorsFactory}} is responsible for stopping all the executors it creates, so it must start before any of them is injected.
> …
[View More]BlockingTaskAwareExecutorServiceImpl may leak controller thread
> ---------------------------------------------------------------
>
> Key: ISPN-10346
> URL: https://issues.jboss.org/browse/ISPN-10346
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> I got a thread leak when running the test suite with trace logging enabled and {{taskset -c 1-2}}:
> {noformat}
> 17:06:05,307 DEBUG (testng-Test:[]) [TestSuiteProgress] Test configuration finished: org.infinispan.remoting.transport.InitialClusterSizeTest.testClassFinished
> 17:15:17,284 ERROR [TestSuiteProgress] Test failed: InitialClusterSizeTest.ThreadLeakChecker
> java.lang.RuntimeException: Leaked thread Controller-remote-thread-InitialClusterSizeTest-NodeB
> at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) ~[?:?]
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:318) ~[?:?]
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159) ~[classes/:?]
> {noformat}
> I believe it may be caused by {{doExecute()}} adding a task to the blocked tasks queue without calling {{checkForReadyTasks()}} (after the executor rejected the task). {{ControllerThread.run()}} needs a semaphore permit to start polling for tasks, so it needs a {{semaphore.release()}} after each {{blockedTasks.add().
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10161) ThreadLeakChecker is blocking PolarionJUnitXMLReporter to generate the reports
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10161?page=com.atlassian.jira.plugin... ]
Dan Berindei reassigned ISPN-10161:
-----------------------------------
Assignee: Dan Berindei (was: Diego Lovison)
> ThreadLeakChecker is blocking PolarionJUnitXMLReporter to generate the reports
> ------------------------------------------------------------------------------
>
> Key: ISPN-10161
> URL: https://issues.jboss.org/browse/ISPN-10161
> …
[View More] Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Final
> Reporter: Diego Lovison
> Assignee: Dan Berindei
> Priority: Critical
>
> When we have Leaked threads we are throwing a new RuntimeException that is preventing the next listener to be running.
> From `master` execute `rm -rf hibernate/cache-v51/target/surefire-reports/ && rm -rf hibernate/cache-v53/target/surefire-reports/ && mvn verify -pl hibernate/cache-v51,hibernate/cache-v53 -Dtest=MultiTenancyTest -fae`
> After that, check the hibernate/cache-v53/target/surefire-reports/ folder, you will see no reports there.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10161) ThreadLeakChecker is blocking PolarionJUnitXMLReporter to generate the reports
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10161?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10161:
--------------------------------
Sprint: DataGrid Sprint #30
> ThreadLeakChecker is blocking PolarionJUnitXMLReporter to generate the reports
> ------------------------------------------------------------------------------
>
> Key: ISPN-10161
> URL: https://issues.jboss.org/browse/ISPN-10161
> Project: Infinispan
…
[View More]> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Final
> Reporter: Diego Lovison
> Assignee: Dan Berindei
> Priority: Critical
>
> When we have Leaked threads we are throwing a new RuntimeException that is preventing the next listener to be running.
> From `master` execute `rm -rf hibernate/cache-v51/target/surefire-reports/ && rm -rf hibernate/cache-v53/target/surefire-reports/ && mvn verify -pl hibernate/cache-v51,hibernate/cache-v53 -Dtest=MultiTenancyTest -fae`
> After that, check the hibernate/cache-v53/target/surefire-reports/ folder, you will see no reports there.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10161) ThreadLeakChecker is blocking PolarionJUnitXMLReporter to generate the reports
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10161?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10161:
--------------------------------
Status: Open (was: Pull Request Sent)
> ThreadLeakChecker is blocking PolarionJUnitXMLReporter to generate the reports
> ------------------------------------------------------------------------------
>
> Key: ISPN-10161
> URL: https://issues.jboss.org/browse/ISPN-10161
> Project: …
[View More]Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Final
> Reporter: Diego Lovison
> Assignee: Dan Berindei
> Priority: Critical
>
> When we have Leaked threads we are throwing a new RuntimeException that is preventing the next listener to be running.
> From `master` execute `rm -rf hibernate/cache-v51/target/surefire-reports/ && rm -rf hibernate/cache-v53/target/surefire-reports/ && mvn verify -pl hibernate/cache-v51,hibernate/cache-v53 -Dtest=MultiTenancyTest -fae`
> After that, check the hibernate/cache-v53/target/surefire-reports/ folder, you will see no reports there.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10346) BlockingTaskAwareExecutorServiceImpl may leak controller thread
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10346?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10346:
--------------------------------
Sprint: DataGrid Sprint #30
> BlockingTaskAwareExecutorServiceImpl may leak controller thread
> ---------------------------------------------------------------
>
> Key: ISPN-10346
> URL: https://issues.jboss.org/browse/ISPN-10346
> Project: Infinispan
> Issue Type: Bug
…
[View More]> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> I got a thread leak when running the test suite with trace logging enabled and {{taskset -c 1-2}}:
> {noformat}
> 17:06:05,307 DEBUG (testng-Test:[]) [TestSuiteProgress] Test configuration finished: org.infinispan.remoting.transport.InitialClusterSizeTest.testClassFinished
> 17:15:17,284 ERROR [TestSuiteProgress] Test failed: InitialClusterSizeTest.ThreadLeakChecker
> java.lang.RuntimeException: Leaked thread Controller-remote-thread-InitialClusterSizeTest-NodeB
> at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) ~[?:?]
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:318) ~[?:?]
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159) ~[classes/:?]
> {noformat}
> I believe it may be caused by {{doExecute()}} adding a task to the blocked tasks queue without calling {{checkForReadyTasks()}} (after the executor rejected the task). {{ControllerThread.run()}} needs a semaphore permit to start polling for tasks, so it needs a {{semaphore.release()}} after each {{blockedTasks.add().
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10349) PersistenceManager.addSegments may never complete
by Dan Berindei (Jira)
Dan Berindei created ISPN-10349:
-----------------------------------
Summary: PersistenceManager.addSegments may never complete
Key: ISPN-10349
URL: https://issues.jboss.org/browse/ISPN-10349
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 10.0.0.Beta3
Reporter: Dan Berindei
Assignee: Will Burns
Fix For: 10.0.0.Beta4
{{PersistenceManager.addSegments}}…
[View More] returns a {{CompletionStage}} which should be completed on a persistence executor thread. If the persistence executor was already stopped at the time {{addSegments()}} was called, however, the {{CompletionStage}} is never completed.
This doesn't prevent the cache manager from stopping, but it causes a thread leak because a transport thread waits forever on the {{CompletionStage}}. E.g.
{noformat}
16:57:06,128 TRACE (testng-Test:[]) [BasicComponentRegistryImpl] Changed status of org.infinispan.executors.persistence to STOPPED
16:57:06,150 TRACE (transport-thread-Test-NodeB-p2059-t4:[Topology-testCache]) [PersistenceManagerImpl] Adding segments for id 584
# no other mention of id 584
17:15:17,300 ERROR [TestSuiteProgress] Test failed: SharedStoreInvalidationDuringRehashTest.ThreadLeakChecker
java.lang.RuntimeException: Leaked thread transport-thread-SharedStoreInvalidationDuringRehashTest-NodeB-p2059-t4
at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) ~[?:?]
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1798) ~[?:?]
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) ~[?:?]
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1868) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:110) ~[classes/:?]
at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:73) ~[classes/:?]
at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:306) ~[classes/:?]
at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:102) ~[classes/:?]
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10349) PersistenceManager.addSegments may never complete
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10349?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10349:
--------------------------------
Status: Open (was: New)
> PersistenceManager.addSegments may never complete
> -------------------------------------------------
>
> Key: ISPN-10349
> URL: https://issues.jboss.org/browse/ISPN-10349
> Project: Infinispan
> Issue Type: Bug
> Components: Core
…
[View More]> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Will Burns
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta4
>
>
> {{PersistenceManager.addSegments}} returns a {{CompletionStage}} which should be completed on a persistence executor thread. If the persistence executor was already stopped at the time {{addSegments()}} was called, however, the {{CompletionStage}} is never completed.
> This doesn't prevent the cache manager from stopping, but it causes a thread leak because a transport thread waits forever on the {{CompletionStage}}. E.g.
> {noformat}
> 16:57:06,128 TRACE (testng-Test:[]) [BasicComponentRegistryImpl] Changed status of org.infinispan.executors.persistence to STOPPED
> 16:57:06,150 TRACE (transport-thread-Test-NodeB-p2059-t4:[Topology-testCache]) [PersistenceManagerImpl] Adding segments for id 584
> # no other mention of id 584
> 17:15:17,300 ERROR [TestSuiteProgress] Test failed: SharedStoreInvalidationDuringRehashTest.ThreadLeakChecker
> java.lang.RuntimeException: Leaked thread transport-thread-SharedStoreInvalidationDuringRehashTest-NodeB-p2059-t4
> at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) ~[?:?]
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1798) ~[?:?]
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128) ~[?:?]
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1868) ~[?:?]
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?]
> at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:110) ~[classes/:?]
> at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:73) ~[classes/:?]
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:306) ~[classes/:?]
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:102) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10346) BlockingTaskAwareExecutorServiceImpl may leak controller thread
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10346?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10346:
--------------------------------
Status: Open (was: New)
> BlockingTaskAwareExecutorServiceImpl may leak controller thread
> ---------------------------------------------------------------
>
> Key: ISPN-10346
> URL: https://issues.jboss.org/browse/ISPN-10346
> Project: Infinispan
> Issue Type: Bug
…
[View More]> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> I got a thread leak when running the test suite with trace logging enabled and {{taskset -c 1-2}}:
> {noformat}
> 17:06:05,307 DEBUG (testng-Test:[]) [TestSuiteProgress] Test configuration finished: org.infinispan.remoting.transport.InitialClusterSizeTest.testClassFinished
> 17:15:17,284 ERROR [TestSuiteProgress] Test failed: InitialClusterSizeTest.ThreadLeakChecker
> java.lang.RuntimeException: Leaked thread Controller-remote-thread-InitialClusterSizeTest-NodeB
> at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) ~[?:?]
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:318) ~[?:?]
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159) ~[classes/:?]
> {noformat}
> I believe it may be caused by {{doExecute()}} adding a task to the blocked tasks queue without calling {{checkForReadyTasks()}} (after the executor rejected the task). {{ControllerThread.run()}} needs a semaphore permit to start polling for tasks, so it needs a {{semaphore.release()}} after each {{blockedTasks.add().
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months
[JBoss JIRA] (ISPN-10346) BlockingTaskAwareExecutorServiceImpl may leak controller thread
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10346?page=com.atlassian.jira.plugin... ]
Dan Berindei reassigned ISPN-10346:
-----------------------------------
Assignee: Dan Berindei
> BlockingTaskAwareExecutorServiceImpl may leak controller thread
> ---------------------------------------------------------------
>
> Key: ISPN-10346
> URL: https://issues.jboss.org/browse/ISPN-10346
> Project: Infinispan
> Issue Type: …
[View More]Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> I got a thread leak when running the test suite with trace logging enabled and {{taskset -c 1-2}}:
> {noformat}
> 17:06:05,307 DEBUG (testng-Test:[]) [TestSuiteProgress] Test configuration finished: org.infinispan.remoting.transport.InitialClusterSizeTest.testClassFinished
> 17:15:17,284 ERROR [TestSuiteProgress] Test failed: InitialClusterSizeTest.ThreadLeakChecker
> java.lang.RuntimeException: Leaked thread Controller-remote-thread-InitialClusterSizeTest-NodeB
> at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) ~[?:?]
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) ~[?:?]
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:318) ~[?:?]
> at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$ControllerThread.run(BlockingTaskAwareExecutorServiceImpl.java:159) ~[classes/:?]
> {noformat}
> I believe it may be caused by {{doExecute()}} adding a task to the blocked tasks queue without calling {{checkForReadyTasks()}} (after the executor rejected the task). {{ControllerThread.run()}} needs a semaphore permit to start polling for tasks, so it needs a {{semaphore.release()}} after each {{blockedTasks.add().
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
[View Less]
5 years, 7 months