[JBoss JIRA] (DROOLS-4645) High memory consumption with JIT
by Mario Fusco (Jira)
[ https://issues.jboss.org/browse/DROOLS-4645?page=com.atlassian.jira.plugi... ]
Mario Fusco updated DROOLS-4645:
--------------------------------
Sprint: 2019 Week 41-43 (from Okt 7)
> High memory consumption with JIT
> --------------------------------
>
> Key: DROOLS-4645
> URL: https://issues.jboss.org/browse/DROOLS-4645
> Project: Drools
> Issue Type: Bug
> Components: executable model
> Affects Versions: 7.29.0.Final
> Reporter: Radovan Synek
> Assignee: Mario Fusco
> Priority: Critical
> Attachments: Screenshot from 2019-10-15 10-30-27.png, Screenshot from 2019-10-15 10-31-53.png
>
>
> Employee Rostering example for OptaPlanner shows excessive memory consumption, which is connected with Drools score calculation.
> Please see the reproducer - running it without drools JIT finishes on time with 3GB of memory while with drools JIT being active it fails due to "GC overhead limit exceeded" despite the fact it god twice as much memory.
> Logging (add -Dlogback.level.org.optaplanner=trace to the execute_jit.sh script) showed there is a huge pillar move changing ~hundreds of entities which takes seconds to calculate score. During this move's evaluation, the memory consumption increases to a point when GC takes over and later fails.
> Memory sampling and heap dump investigation showed there are ~10^7 objects of org.drools.core.reteoo.FromNodeLeftTuple class, taking more memory than any other data type in the VM.
> See the attached screenshots.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (DROOLS-4645) High memory consumption with JIT
by Radovan Synek (Jira)
[ https://issues.jboss.org/browse/DROOLS-4645?page=com.atlassian.jira.plugi... ]
Radovan Synek updated DROOLS-4645:
----------------------------------
Attachment: Screenshot from 2019-10-15 10-30-27.png
Screenshot from 2019-10-15 10-31-53.png
> High memory consumption with JIT
> --------------------------------
>
> Key: DROOLS-4645
> URL: https://issues.jboss.org/browse/DROOLS-4645
> Project: Drools
> Issue Type: Bug
> Components: executable model
> Affects Versions: 7.29.0.Final
> Reporter: Radovan Synek
> Assignee: Mario Fusco
> Priority: Major
> Attachments: Screenshot from 2019-10-15 10-30-27.png, Screenshot from 2019-10-15 10-31-53.png
>
>
> Employee Rostering example for OptaPlanner shows excessive memory consumption, which is connected with Drools score calculation.
> Please see the reproducer - running it without drools JIT finishes on time with 3GB of memory while with drools JIT being active it fails due to "GC overhead limit exceeded" despite the fact it god twice as much memory.
> Logging (add -Dlogback.level.org.optaplanner=trace to the execute_jit.sh script) showed there is a huge pillar move changing ~hundreds of entities which takes seconds to calculate score. During this move's evaluation, the memory consumption increases to a point when GC takes over and later fails.
> Memory sampling and heap dump investigation showed there are ~10^7 objects of org.drools.core.reteoo.FromNodeLeftTuple class, taking more memory than any other data type in the VM.
> See the attached screenshots.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (DROOLS-4645) High memory consumption with JIT
by Radovan Synek (Jira)
[ https://issues.jboss.org/browse/DROOLS-4645?page=com.atlassian.jira.plugi... ]
Radovan Synek updated DROOLS-4645:
----------------------------------
Priority: Critical (was: Major)
> High memory consumption with JIT
> --------------------------------
>
> Key: DROOLS-4645
> URL: https://issues.jboss.org/browse/DROOLS-4645
> Project: Drools
> Issue Type: Bug
> Components: executable model
> Affects Versions: 7.29.0.Final
> Reporter: Radovan Synek
> Assignee: Mario Fusco
> Priority: Critical
> Attachments: Screenshot from 2019-10-15 10-30-27.png, Screenshot from 2019-10-15 10-31-53.png
>
>
> Employee Rostering example for OptaPlanner shows excessive memory consumption, which is connected with Drools score calculation.
> Please see the reproducer - running it without drools JIT finishes on time with 3GB of memory while with drools JIT being active it fails due to "GC overhead limit exceeded" despite the fact it god twice as much memory.
> Logging (add -Dlogback.level.org.optaplanner=trace to the execute_jit.sh script) showed there is a huge pillar move changing ~hundreds of entities which takes seconds to calculate score. During this move's evaluation, the memory consumption increases to a point when GC takes over and later fails.
> Memory sampling and heap dump investigation showed there are ~10^7 objects of org.drools.core.reteoo.FromNodeLeftTuple class, taking more memory than any other data type in the VM.
> See the attached screenshots.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (DROOLS-4645) High memory consumption with JIT
by Radovan Synek (Jira)
[ https://issues.jboss.org/browse/DROOLS-4645?page=com.atlassian.jira.plugi... ]
Radovan Synek reassigned DROOLS-4645:
-------------------------------------
Assignee: Mario Fusco (was: Luca Molteni)
> High memory consumption with JIT
> --------------------------------
>
> Key: DROOLS-4645
> URL: https://issues.jboss.org/browse/DROOLS-4645
> Project: Drools
> Issue Type: Bug
> Components: executable model
> Affects Versions: 7.29.0.Final
> Reporter: Radovan Synek
> Assignee: Mario Fusco
> Priority: Major
>
> Employee Rostering example for OptaPlanner shows excessive memory consumption, which is connected with Drools score calculation.
> Please see the reproducer - running it without drools JIT finishes on time with 3GB of memory while with drools JIT being active it fails due to "GC overhead limit exceeded" despite the fact it god twice as much memory.
> Logging (add -Dlogback.level.org.optaplanner=trace to the execute_jit.sh script) showed there is a huge pillar move changing ~hundreds of entities which takes seconds to calculate score. During this move's evaluation, the memory consumption increases to a point when GC takes over and later fails.
> Memory sampling and heap dump investigation showed there are ~10^7 objects of org.drools.core.reteoo.FromNodeLeftTuple class, taking more memory than any other data type in the VM.
> See the attached screenshots.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (DROOLS-4645) High memory consumption with JIT
by Radovan Synek (Jira)
Radovan Synek created DROOLS-4645:
-------------------------------------
Summary: High memory consumption with JIT
Key: DROOLS-4645
URL: https://issues.jboss.org/browse/DROOLS-4645
Project: Drools
Issue Type: Bug
Components: executable model
Affects Versions: 7.29.0.Final
Reporter: Radovan Synek
Assignee: Luca Molteni
Employee Rostering example for OptaPlanner shows excessive memory consumption, which is connected with Drools score calculation.
Please see the reproducer - running it without drools JIT finishes on time with 3GB of memory while with drools JIT being active it fails due to "GC overhead limit exceeded" despite the fact it god twice as much memory.
Logging (add -Dlogback.level.org.optaplanner=trace to the execute_jit.sh script) showed there is a huge pillar move changing ~hundreds of entities which takes seconds to calculate score. During this move's evaluation, the memory consumption increases to a point when GC takes over and later fails.
Memory sampling and heap dump investigation showed there are ~10^7 objects of org.drools.core.reteoo.FromNodeLeftTuple class, taking more memory than any other data type in the VM.
See the attached screenshots.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (WFLY-12671) Memory Leak of PersistenceUnitMetadataImpl
by Robin Schimpf (Jira)
Robin Schimpf created WFLY-12671:
------------------------------------
Summary: Memory Leak of PersistenceUnitMetadataImpl
Key: WFLY-12671
URL: https://issues.jboss.org/browse/WFLY-12671
Project: WildFly
Issue Type: Bug
Affects Versions: 17.0.1.Final
Reporter: Robin Schimpf
Assignee: Brian Stansberry
Attachments: Trying_to_catch_memory_leak_with_more_trace_logging.patch, memory leak after 4 deployments.PNG, memory leak after 5 deployments.PNG, persistence-unit-not-stopped.txt, persistence-unit-stopped.txt
After updating from Wildfly 13 to Wildfly 17 we noticed a slowdown of our application after multiple deployments to the application server without restarting it between the deployments. Heapdumps are showing that after redeployment the number of {{PersistenceUnitMetadataImpl}} are growing (see screenshots).
We are deploying 2 ear files and 2 war files. Only one ear has a database connection configured. We are using the bundeled hibernate version of Wildfly 17.
The ear structure is like the following:
{noformat}
- app.ear
|
|- META-INF
|
|- lib
| |
| |- all dependencies
|
|- own jars and wars (e.g. jpa)
{noformat}
Since I failed to create a reproducing project for this behaviour I added some trace logging (Patch attached). The only difference I noticed was that sometimes the {{Stopping Persistence Unit}} logstatement is missing before the {{Stopped subdeployment}} logstatement.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (WFCORE-4710) Wrong BufferPoolMXBean name used in lookup
by Andrew Dinn (Jira)
[ https://issues.jboss.org/browse/WFCORE-4710?page=com.atlassian.jira.plugi... ]
Andrew Dinn commented on WFCORE-4710:
-------------------------------------
Thanks for looking at this Brian. I'm sure you know far better than I do what the solution is.
It is actually quite lucky the error method was passed the escaped name as otherwise the message would have been even more confusing.
I'm not at all surprised this has not turned up before. Changes to the buffer pool are rare and changes which install names with e.g. spaces that need escaping even more so. Let me know if you have problems recreating this.
> Wrong BufferPoolMXBean name used in lookup
> ------------------------------------------
>
> Key: WFCORE-4710
> URL: https://issues.jboss.org/browse/WFCORE-4710
> Project: WildFly Core
> Issue Type: Enhancement
> Components: JMX
> Affects Versions: 10.0.0.Final
> Reporter: Andrew Dinn
> Assignee: Brian Stansberry
> Priority: Major
>
> Class BufferPoolMXBeanResource transforms the name of retrieved MXBeans to handle embedded spaces and other special characters by calling method escapeMBeanName. Later on class BufferPoolMXBeanResourceAttributeHandler uses the name to identify the corresponding bean. Unfortunately, when BufferPoolMXBeanResourceAttributeHandler checks that the bean is registered it does not reverse the transformation by calling unescapeMBeanName.
> This causes a lookup failure when the new buffer pool non-volatiel memory MXBean is encountered on JDK14. Although Wildlfy appears to continue running the error message appears to cause failures in some of the Wildfly tests, notably those that check error stream output.
> See the referenced forum thread for full details.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months
[JBoss JIRA] (WFLY-12669) Clustering: scattered cache errors
by Tommasso Borgato (Jira)
[ https://issues.jboss.org/browse/WFLY-12669?page=com.atlassian.jira.plugin... ]
Tommasso Borgato updated WFLY-12669:
------------------------------------
Description:
The issue is about scattered cache replication across cluster nodes;
WildFly is started in clustered mode using a scattered cache for replicating HTTP session data across cluster nodes; all 4 nodes in the cluster are initialized with the following {{cli}} script:
{noformat}
embed-server --server-config=standalone-ha.xml
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
/subsystem=infinispan/cache-container=web/scattered-cache=testScattered:add()
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testScattered)
{noformat}
The test is run with [wildfly-18.0.0.Final.zip|https://download.jboss.org/wildfly/18.0.0.Final...]; the same tests run with version [wildfly-17.0.1.Final.zip|https://download.jboss.org/wildfly/17.0.1.Final...] do not have any problem; hence this looks like a regression;
As usual, we test that the serial value stored in the scattered cache is incremented at every call: when this is not true, we say we have a *sampling error*;
Using MOD_JK as load balancer, we have a *22.46%* of *sampling errors* with wildfly-18.0.0.Final.zip (0% with wildfly-17.0.1.Final.zip):
- OK [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
- ERROR [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
Using HAPROXY as load balancer, we have a *1.71%* of *sampling errors* with wildfly-18.0.0.Final.zip (0% with wildfly-17.0.1.Final.zip):
- OK [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
- ERROR [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
On the server we deploy, as usual, {{clusterbench}} (see attachment clusterbench-ee8.ear).
was:
The issue is about scattered cache replication across cluster nodes;
WildFly is started in clustered mode using a scattered cache for replicating HTTP session data across cluster nodes; all 4 nodes in the cluster are initialized with the following {{cli}} script:
{noformat}
embed-server --server-config=standalone-ha.xml
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
/subsystem=infinispan/cache-container=web/scattered-cache=testScattered:add()
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testScattered)
{noformat}
The test is run with [wildfly-18.0.0.Final.zip|https://download.jboss.org/wildfly/18.0.0.Final...]; the same tests run with version [wildfly-17.0.1.Final.zip|https://download.jboss.org/wildfly/17.0.1.Final...] do not have any problem; hence this looks like a regression;
As usual, we test that the serial value stored in the scattered cache is incremented at every call: when this is not true, we say we have a *sampling error*;
Using MOD_JK as load balancer, we have a *22.46%* of *sampling errors* with wildfly-18.0.0.Final.zip (0% with wildfly-17.0.1.Final.zip):
- OK [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
- ERROR [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
Using HAPROXY as load balancer, we have a *1.71%* of *sampling errors* with wildfly-18.0.0.Final.zip (0% with wildfly-17.0.1.Final.zip):
- OK [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
- ERROR [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
On the server we deploy, as usual, `clusterbench` (see attachment clusterbench-ee8.ear).
> Clustering: scattered cache errors
> ----------------------------------
>
> Key: WFLY-12669
> URL: https://issues.jboss.org/browse/WFLY-12669
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 18.0.0.Final
> Reporter: Tommasso Borgato
> Assignee: Paul Ferraro
> Priority: Critical
> Attachments: clusterbench-ee8.ear
>
>
> The issue is about scattered cache replication across cluster nodes;
> WildFly is started in clustered mode using a scattered cache for replicating HTTP session data across cluster nodes; all 4 nodes in the cluster are initialized with the following {{cli}} script:
> {noformat}
> embed-server --server-config=standalone-ha.xml
> /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
> /subsystem=infinispan/cache-container=web/scattered-cache=testScattered:add()
> /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testScattered)
> {noformat}
> The test is run with [wildfly-18.0.0.Final.zip|https://download.jboss.org/wildfly/18.0.0.Final...]; the same tests run with version [wildfly-17.0.1.Final.zip|https://download.jboss.org/wildfly/17.0.1.Final...] do not have any problem; hence this looks like a regression;
> As usual, we test that the serial value stored in the scattered cache is incremented at every call: when this is not true, we say we have a *sampling error*;
> Using MOD_JK as load balancer, we have a *22.46%* of *sampling errors* with wildfly-18.0.0.Final.zip (0% with wildfly-17.0.1.Final.zip):
> - OK [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
> - ERROR [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
> Using HAPROXY as load balancer, we have a *1.71%* of *sampling errors* with wildfly-18.0.0.Final.zip (0% with wildfly-17.0.1.Final.zip):
> - OK [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
> - ERROR [eap-7.x-clustering-http-session-shutdown-scattered with wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.red...]
> On the server we deploy, as usual, {{clusterbench}} (see attachment clusterbench-ee8.ear).
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
6 years, 9 months