[jboss-jira] [JBoss JIRA] (WFCORE-992) Memory leak in host controller

[Ladislav Thon (JIRA)]

     [ https://issues.jboss.org/browse/WFCORE-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ladislav Thon updated WFCORE-992:
---------------------------------
    Description: 
The {{ManagedServerProxy}} class in the {{host-controller}} module contains a memory leak, which can be easily reproduced using the RBAC soak test, like so:

{code}
cd wildfly-core # checkout WildFly Core 2.0.0.CR1, which is in WildFly 10.0.0.CR1, or current master
mvn clean install -DskipTests
cd testsuite/domain
mvn clean test -Dts.noSmoke -Dts.domain -Prbac-soak -Djboss.test.rbac.soak.clients=100 -Djboss.test.rbac.soak.iterations=50
{code}

After acquiring a heap dump ({{-XX:+HeapDumpOnOutOfMemoryError}}), it's clear that out of the 512 MB of heap space a host controller has, more than 400 MB is taken by a single instance of the {{ManagedServerProxy}} class, specifically by its {{activeRequests}} field. This {{Map}} has a single entry whose value is a {{Set}} of "pending" futures. I added a couple of debug prints to the places where this {{Set}} is modified, only to see this:

{code}
2015-09-21 12:32:00,704 INFO  [stdout] (management-handler-thread - 4) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 142a1c24
2015-09-21 12:32:00,707 INFO  [stdout] (Remoting "master:MANAGEMENT" task-3) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 5bfbf423
2015-09-21 12:32:00,707 INFO  [stdout] (management-handler-thread - 4) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 27e2e96c
2015-09-21 12:32:00,708 INFO  [stdout] (Remoting "master:MANAGEMENT" task-2) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 72817a0d
2015-09-21 12:32:00,713 INFO  [stdout] (management-handler-thread - 2) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 73d373a7
2015-09-21 12:32:00,718 INFO  [stdout] (Remoting "master:MANAGEMENT" task-9) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 20bd6db1
{code}

Skipping a few intermediate steps, the root cause is that {{AbstractDelegatingAsyncFuture}} delegates {{addListener}} directly to an underlying future, so the callbacks don't get the original instance, but the underlying one.

  was:
The {{ManagedServerProxy}} class in the {{host-controller}} module contains a memory leak, which can be easily reproduced using the RBAC soak test, like so:

{code}
cd wildfly-core # checkout WildFly Core 2.0.0.CR1, which is in EAP 7.0.0.DR10, or current master
mvn clean install -DskipTests
cd testsuite/domain
mvn clean test -Dts.noSmoke -Dts.domain -Prbac-soak -Djboss.test.rbac.soak.clients=100 -Djboss.test.rbac.soak.iterations=50
{code}

After acquiring a heap dump ({{-XX:+HeapDumpOnOutOfMemoryError}}), it's clear that out of the 512 MB of heap space a host controller has, more than 400 MB is taken by a single instance of the {{ManagedServerProxy}} class, specifically by its {{activeRequests}} field. This {{Map}} has a single entry whose value is a {{Set}} of "pending" futures. I added a couple of debug prints to the places where this {{Set}} is modified, only to see this:

{code}
2015-09-21 12:32:00,704 INFO  [stdout] (management-handler-thread - 4) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 142a1c24
2015-09-21 12:32:00,707 INFO  [stdout] (Remoting "master:MANAGEMENT" task-3) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 5bfbf423
2015-09-21 12:32:00,707 INFO  [stdout] (management-handler-thread - 4) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 27e2e96c
2015-09-21 12:32:00,708 INFO  [stdout] (Remoting "master:MANAGEMENT" task-2) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 72817a0d
2015-09-21 12:32:00,713 INFO  [stdout] (management-handler-thread - 2) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 73d373a7
2015-09-21 12:32:00,718 INFO  [stdout] (Remoting "master:MANAGEMENT" task-9) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 20bd6db1
{code}

Skipping a few intermediate steps, the root cause is that {{AbstractDelegatingAsyncFuture}} delegates {{addListener}} directly to an underlying future, so the callbacks don't get the original instance, but the underlying one.



> Memory leak in host controller
> ------------------------------
>
>                 Key: WFCORE-992
>                 URL: https://issues.jboss.org/browse/WFCORE-992
>             Project: WildFly Core
>          Issue Type: Bug
>          Components: Domain Management
>            Reporter: Ladislav Thon
>            Assignee: Brian Stansberry
>            Priority: Blocker
>
> The {{ManagedServerProxy}} class in the {{host-controller}} module contains a memory leak, which can be easily reproduced using the RBAC soak test, like so:
> {code}
> cd wildfly-core # checkout WildFly Core 2.0.0.CR1, which is in WildFly 10.0.0.CR1, or current master
> mvn clean install -DskipTests
> cd testsuite/domain
> mvn clean test -Dts.noSmoke -Dts.domain -Prbac-soak -Djboss.test.rbac.soak.clients=100 -Djboss.test.rbac.soak.iterations=50
> {code}
> After acquiring a heap dump ({{-XX:+HeapDumpOnOutOfMemoryError}}), it's clear that out of the 512 MB of heap space a host controller has, more than 400 MB is taken by a single instance of the {{ManagedServerProxy}} class, specifically by its {{activeRequests}} field. This {{Map}} has a single entry whose value is a {{Set}} of "pending" futures. I added a couple of debug prints to the places where this {{Set}} is modified, only to see this:
> {code}
> 2015-09-21 12:32:00,704 INFO  [stdout] (management-handler-thread - 4) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 142a1c24
> 2015-09-21 12:32:00,707 INFO  [stdout] (Remoting "master:MANAGEMENT" task-3) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 5bfbf423
> 2015-09-21 12:32:00,707 INFO  [stdout] (management-handler-thread - 4) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 27e2e96c
> 2015-09-21 12:32:00,708 INFO  [stdout] (Remoting "master:MANAGEMENT" task-2) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 72817a0d
> 2015-09-21 12:32:00,713 INFO  [stdout] (management-handler-thread - 2) !!! added org.jboss.as.controller.remote.TransactionalProtocolClientImpl$1 at 73d373a7
> 2015-09-21 12:32:00,718 INFO  [stdout] (Remoting "master:MANAGEMENT" task-9) !!! removed org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl at 20bd6db1
> {code}
> Skipping a few intermediate steps, the root cause is that {{AbstractDelegatingAsyncFuture}} delegates {{addListener}} directly to an underlying future, so the callbacks don't get the original instance, but the underlying one.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)