[jboss-jira] [JBoss JIRA] (WFCORE-4801) RemoteConnectionHandler and ConnectionImpl objects leak when org.jboss.as.cli is used

[Vinoth Kumar Natarajan (Jira)]

    [ https://issues.redhat.com/browse/WFCORE-4801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13953431#comment-13953431 ] 

Vinoth Kumar Natarajan commented on WFCORE-4801:
------------------------------------------------

Further investigation shows that issue is found in *Release: 9.0.1.Final* but not seen with *Release: 10.0.3.Final*. But we are not in a position to upgrade Wildfly. Please take a look into this issue.

*Test results from Release: 9.0.1.Final :*
{code}
$ winpty docker exec -it f370379d0044 bash
[jboss at f370379d0044 ~]$ find ./ -name 'wildfly-cli*jar'
./wildfly/modules/system/layers/base/org/wildfly/client/config/main/wildfly-client-config-1.0.1.Final.jar
./wildfly/modules/system/layers/base/org/jboss/as/cli/main/wildfly-cli-9.0.1.Final.jar
[jboss at f370379d0044 ~]$ jmap -histo:live 90 | grep ConnectionImpl
1859:             1             80  org.jboss.remoting3.ConnectionImpl
5691:             1             16  org.jboss.remoting3.ConnectionImpl$$Lambda$663/0x000000084098fc40
[jboss at f370379d0044 ~]$ /opt/jboss/wildfly/bin/jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect
[standalone at localhost:9990 /] version
JBoss Admin Command-line Interface
JBOSS_HOME: /opt/jboss/wildfly
Release: 9.0.1.Final
Product: WildFly Full 17.0.0.Final
JAVA_HOME: /usr/lib/jvm/java
java.version: 11.0.1
java.vm.vendor: Oracle Corporation
java.vm.version: 11.0.1+13-LTS
os.name: Linux
os.version: 4.9.184-linuxkit

[standalone at localhost:9990 /] exit
[jboss at f370379d0044 ~]$ jmap -histo:live 90 | grep ConnectionImpl
1152:             2            160  org.jboss.remoting3.ConnectionImpl
5681:             1             16  org.jboss.remoting3.ConnectionImpl$$Lambda$663/0x000000084098fc40
[jboss at f370379d0044 ~]$
[jboss at f370379d0044 ~]$ jmap -histo:live 90 | grep RemoteConnectionHandler
1033:             2            192  org.jboss.remoting3.remote.RemoteConnectionHandler
[jboss at f370379d0044 ~]$ jmap -histo:live 90 | grep RemoteConnectionHandler
1033:             2            192  org.jboss.remoting3.remote.RemoteConnectionHandler
[jboss at f370379d0044 ~]$
{code}

*Test results from Release: 10.0.3.Final :*
{code}
$ winpty docker exec -it b3c484904468 bash
[jboss at b3c484904468 ~]$
[jboss at b3c484904468 ~]$
[jboss at b3c484904468 ~]$ find ./ -name 'wildfly-cli*jar'
./wildfly/modules/system/layers/base/org/wildfly/client/config/main/wildfly-client-config-1.0.1.Final.jar
./wildfly/modules/system/layers/base/org/jboss/as/cli/main/wildfly-cli-10.0.3.Final.jar
./wildfly/modules/system/layers/base/org/jboss/as/cli/main/wildfly-cli-11.0.0.Beta8-SNAPSHOT.jar
[jboss at b3c484904468 ~]$
[jboss at b3c484904468 ~]$ jmap -histo:live 91 | grep RemoteConnectionHandler
[jboss at b3c484904468 ~]$
[jboss at b3c484904468 ~]$ /opt/jboss/wildfly/bin/jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] connect
[standalone at localhost:9990 /] version
JBoss Admin Command-line Interface
JBOSS_HOME: /opt/jboss/wildfly
Release: 10.0.3.Final
Product: WildFly Full 18.0.1.Final
JAVA_HOME: /usr/lib/jvm/java
java.version: 11.0.5
java.vm.vendor: Oracle Corporation
java.vm.version: 11.0.5+10-LTS
os.name: Linux
os.version: 4.9.184-linuxkit

[standalone at localhost:9990 /] exit
[jboss at b3c484904468 ~]$
[jboss at b3c484904468 ~]$ jmap -histo:live 91 | grep RemoteConnectionHandler
1515:             1             96  org.jboss.remoting3.remote.RemoteConnectionHandler
[jboss at b3c484904468 ~]$ jmap -histo:live 91 | grep RemoteConnectionHandler
[jboss at b3c484904468 ~]$
[jboss at b3c484904468 ~]$ jmap -histo 91 | grep RemoteConnectionHandler
[jboss at b3c484904468 ~]$ exit
{code}

Thanks,
Vinoth.

> RemoteConnectionHandler and ConnectionImpl objects leak when org.jboss.as.cli is used
> -------------------------------------------------------------------------------------
>
>                 Key: WFCORE-4801
>                 URL: https://issues.redhat.com/browse/WFCORE-4801
>             Project: WildFly Core
>          Issue Type: Bug
>          Components: CLI, Management
>    Affects Versions: 9.0.1.Final
>            Reporter: Vinoth Kumar Natarajan
>            Priority: Major
>
> Heap dump analysis of services deployed in Wildfly standalone deployment show a large number of stale objects of class org.jboss.remoting3.ConnectionImpl and org.jboss.remoting3.remote.RemoteConnectionHandler. These objects are not getting garbage collected and eventually cause JVM to go out of memory.
> Analysis of the heap dump indicate that,
> * All org.jboss.remoting3.ConnectionImpl objects lingering around are CLOSED as the state and associated with JBOSS management endpoint.
> * All org.jboss.remoting3.remote.RemoteConnectionHandler objects lingering around are associated with Remote Endpoint Name "cli-client". which is the endpoint name used by org.jboss.as.cli.
> Above observations seem to indicate that JBOSS CLI commands that are executed periodically in this deployment, to monitor status of deployments, is causing these stale objects. 
> To confirm this observation, we have executed jboss-cli.sh a couple of times on our deployment and could see that each execution of this script is leaving nehind one instance of each of these classes.
> *Wildfly version details:*
> bash-4.4# ./jboss-cli.sh
> You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
> [disconnected /] connect
> [standalone at localhost:9990 /] version
> JBoss Admin Command-line Interface
> JBOSS_HOME: /ericsson/3pp/wildfly
> Release: 9.0.1.Final
> Product: WildFly Full 17.0.0.Final
> JAVA_HOME: /usr/
> java.version: 1.8.0_222
> java.vm.vendor: IcedTea
> java.vm.version: 25.222-b10
> os.name: Linux
> os.version: 4.12.14-197.15-default



--
This message was sent by Atlassian Jira
(v7.13.8#713008)