[JBoss JIRA] (WFLY-12167) Memory leak in metrics in standalone-ha configuration
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFLY-12167?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-12167:
-------------------------------
Priority: Blocker (was: Major)
> Memory leak in metrics in standalone-ha configuration
> -----------------------------------------------------
>
> Key: WFLY-12167
> URL: https://issues.jboss.org/browse/WFLY-12167
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, MP Metrics
> Affects Versions: 16.0.0.Final
> Reporter: Bernd Stolle
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: memoryleak
> Attachments: Screenshot 2019-06-06 at 11.07.00.png
>
>
> When started in standalone HA configuration every request to the recently added metrics endpoint ({{<management-if>:9990/metrics}}) lead to an increase in memory consumption until the JVM is slowed down significantly by GC to a point where even the requests to {{/health}} fail within a reasonable timeout (2s) and untlimately lead to OOM.
> The same issue does not occur when WildFly is started in the default standalone configuration (non HA).
> I can provide a (compressed) heap dump if required.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (WFLY-12167) Memory leak in metrics in standalone-ha configuration
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFLY-12167?page=com.atlassian.jira.plugin... ]
Jeff Mesnil commented on WFLY-12167:
------------------------------------
I investigated it a bit and the issue seems to be related to the clustering component.
When a /metrics HTTP request is queried, the metrics subsystem will invoke the :read-attribute for the clustering resources (e.g. thread-pool-min-threads on the channel=ee/protocol=UDP resource).
Since the returned value is undefined, the metrics subsystem will ignore this attribute.
However comparing heap dumps between /metrics queries reveals a significant increase of ServiceName and ServiceRegistrationImpl instances (that are not GCed).
It seems the issue is related to https://github.com/wildfly/wildfly/blob/master/clustering/service/src/mai... (as the service names are identified with UUID).
It seems that the ServiceRegistrationImpl created by the one-off service is not been properly removed from the Map in org.jboss.msc.service.ServiceContainerImpl#getOrCreateRegistration
[~pferraro] I'm assigning this issue to you so that you can double check if the code in ServiceSupplier is correct or if the leak comes below from MSC.
[~brian.stansberry] [~jamezp] I'm raising the priority to blocker as this issue will eventually lead to memory exhaustion when the app server metrics are queried.
Depending on the time scale, we also have a workaround:
* explicitly state the subsystems that expose metrics (now it is set to the * wildcard to expose metrics from all subsystems) so that groups metrics are not queried at all
> Memory leak in metrics in standalone-ha configuration
> -----------------------------------------------------
>
> Key: WFLY-12167
> URL: https://issues.jboss.org/browse/WFLY-12167
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 16.0.0.Final
> Reporter: Bernd Stolle
> Assignee: Jeff Mesnil
> Priority: Major
> Labels: memoryleak
> Attachments: Screenshot 2019-06-06 at 11.07.00.png
>
>
> When started in standalone HA configuration every request to the recently added metrics endpoint ({{<management-if>:9990/metrics}}) lead to an increase in memory consumption until the JVM is slowed down significantly by GC to a point where even the requests to {{/health}} fail within a reasonable timeout (2s) and untlimately lead to OOM.
> The same issue does not occur when WildFly is started in the default standalone configuration (non HA).
> I can provide a (compressed) heap dump if required.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (WFLY-12167) Memory leak in metrics in standalone-ha configuration
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFLY-12167?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-12167:
-------------------------------
Attachment: Screenshot 2019-06-06 at 11.07.00.png
> Memory leak in metrics in standalone-ha configuration
> -----------------------------------------------------
>
> Key: WFLY-12167
> URL: https://issues.jboss.org/browse/WFLY-12167
> Project: WildFly
> Issue Type: Bug
> Components: MP Metrics
> Affects Versions: 16.0.0.Final
> Reporter: Bernd Stolle
> Assignee: Jeff Mesnil
> Priority: Major
> Labels: memoryleak
> Attachments: Screenshot 2019-06-06 at 11.07.00.png
>
>
> When started in standalone HA configuration every request to the recently added metrics endpoint ({{<management-if>:9990/metrics}}) lead to an increase in memory consumption until the JVM is slowed down significantly by GC to a point where even the requests to {{/health}} fail within a reasonable timeout (2s) and untlimately lead to OOM.
> The same issue does not occur when WildFly is started in the default standalone configuration (non HA).
> I can provide a (compressed) heap dump if required.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (WFLY-10554) OpenSAML 3.3.0 complains for missing class from "metrics-core"
by Jim Ma (Jira)
[ https://issues.jboss.org/browse/WFLY-10554?page=com.atlassian.jira.plugin... ]
Jim Ma commented on WFLY-10554:
-------------------------------
[~rady66] I looked at CXF and WSS4j code. OpenSaml is initialized and configured by passing the xml configuration files and it should go to initialize metrics. Can you please provide more info about your opensaml usage ? Do you have some webservice endpoint with WS-Trust , SecurityTokenService ?
> OpenSAML 3.3.0 complains for missing class from "metrics-core"
> --------------------------------------------------------------
>
> Key: WFLY-10554
> URL: https://issues.jboss.org/browse/WFLY-10554
> Project: WildFly
> Issue Type: Bug
> Components: Web Services
> Affects Versions: 13.0.0.Final
> Reporter: Radoslav Ivanov
> Assignee: Jim Ma
> Priority: Critical
> Attachments: screenshot-1.png
>
>
> Module OpenSAML 3.3.0 requires depedency to module "io.dropwizard.metrics:metrics-core" but it does not present. As a result ClassNotFoundException is thrown.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months
[JBoss JIRA] (DROOLS-4139) Improve Test Scenario editor unit tests reliability
by Daniele Zonca (Jira)
Daniele Zonca created DROOLS-4139:
-------------------------------------
Summary: Improve Test Scenario editor unit tests reliability
Key: DROOLS-4139
URL: https://issues.jboss.org/browse/DROOLS-4139
Project: Drools
Issue Type: Task
Components: Scenario Simulation and Testing
Reporter: Daniele Zonca
Assignee: Yeser Amer
Verify all {{any()}} and {{any(Class)}} usages in Test Scenario's unit tests and try to replace them with {{eq()}} with actual values to have tests more reliable
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 11 months