[jboss-jira] [JBoss JIRA] (WFCORE-1560) Cli calls leak resources in Host Controller when repeatedly calling jboss-cli.sh

[Michael Noack (JIRA)]

     [ https://issues.jboss.org/browse/WFCORE-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Noack updated WFCORE-1560:
----------------------------------
    Description: 
When executing management commands using jboss-cli.sh against the domain controller of a cluster repeatedly the host controller uses up more and more memory in oldgen. After several thousands of runs of jboss-cli the host controller eventually becomes unresponsive (see attached picture for memory consumption, dc became entirely unresponsive at roughly 6:30am):

[root at dc broken]# /opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect  --user="username" --password="password" --command=":read-children-names(child-type=host)"
Failed to connect to the controller: The controller is not available at xx.xx.xx.xx:9993: java.net.ConnectException: WFLYPRT0023: Could not connect to https-remoting://xx.xx.xx.xx:9993. The connection timed out: WFLYPRT0023: Could not connect to https-remoting://xx.xx.xx.xx:9993. The connection timed out

I discovered the issue when testing whether https://issues.jboss.org/browse/WFCORE-974 was actually resolved in wildfly-10.0.0.Final as advertised. I can confirm that the issue is different, since no OOM-Exceptions are thrown. However the DC still becomes useless, since it won't accept any connections anymore. I will check whether the work-around from WFCORE-974 applies to this issue as well.

Please note that the attached logs are UTC, while the monitoring is UTC+2.

  was:
When executing management commands using jboss-cli.sh against the domain controller of a cluster repeatedly the host controller uses up more and more memory in oldgen. After several thousands of runs of jboss-cli the host controller eventually becomes unresponsive (see attached picture for memory consumption, dc became entirely unresponsive at roughly 6:30am):

[root at dc broken]# /opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect  --user="username" --password="password" --command=":read-children-names(child-type=host)"
Failed to connect to the controller: The controller is not available at 64.30.129.5:9993: java.net.ConnectException: WFLYPRT0023: Could not connect to https-remoting://64.30.129.5:9993. The connection timed out: WFLYPRT0023: Could not connect to https-remoting://64.30.129.5:9993. The connection timed out

I discovered the issue when testing whether https://issues.jboss.org/browse/WFCORE-974 was actually resolved in wildfly-10.0.0.Final as advertised. I can confirm that the issue is different, since no OOM-Exceptions are thrown. However the DC still becomes useless, since it won't accept any connections anymore. I will check whether the work-around from WFCORE-974 applies to this issue as well.

Please note that the attached logs are UTC, while the monitoring is UTC+2.



> Cli calls leak resources in Host Controller when repeatedly calling jboss-cli.sh
> --------------------------------------------------------------------------------
>
>                 Key: WFCORE-1560
>                 URL: https://issues.jboss.org/browse/WFCORE-1560
>             Project: WildFly Core
>          Issue Type: Bug
>          Components: CLI
>    Affects Versions: 2.0.8.Final
>         Environment: OS: CentOS 7.2
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> Wildfly-10.0.0-Final
>            Reporter: Michael Noack
>            Assignee: Alexey Loubyansky
>         Attachments: console-dc.log, host-controller.log, JVM-DC.png, process-controller.log
>
>
> When executing management commands using jboss-cli.sh against the domain controller of a cluster repeatedly the host controller uses up more and more memory in oldgen. After several thousands of runs of jboss-cli the host controller eventually becomes unresponsive (see attached picture for memory consumption, dc became entirely unresponsive at roughly 6:30am):
> [root at dc broken]# /opt/wildfly-10.0.0.Final-DC/bin/./jboss-cli.sh --connect  --user="username" --password="password" --command=":read-children-names(child-type=host)"
> Failed to connect to the controller: The controller is not available at xx.xx.xx.xx:9993: java.net.ConnectException: WFLYPRT0023: Could not connect to https-remoting://xx.xx.xx.xx:9993. The connection timed out: WFLYPRT0023: Could not connect to https-remoting://xx.xx.xx.xx:9993. The connection timed out
> I discovered the issue when testing whether https://issues.jboss.org/browse/WFCORE-974 was actually resolved in wildfly-10.0.0.Final as advertised. I can confirm that the issue is different, since no OOM-Exceptions are thrown. However the DC still becomes useless, since it won't accept any connections anymore. I will check whether the work-around from WFCORE-974 applies to this issue as well.
> Please note that the attached logs are UTC, while the monitoring is UTC+2.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)