[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Jan Stourac updated WFLY-10991:
-------------------------------
Priority: Blocker (was: Critical)
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Blocker
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> Size of the extra heap in use is about 25MB plus when compared to the initial size.
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Also, when microprofile-opentracing-smallrye subsystem is removed via:
> {code}
> /subsystem=microprofile-opentracing-smallrye:remove()
> {code}
> then no more memory leak is detected, after adding the subsystem back, the leak is present again. Thus opentracing is involved somehow into this indeed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10992) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
Jan Stourac created WFLY-10992:
----------------------------------
Summary: Memory leak when deployment is redeployed multiple times
Key: WFLY-10992
URL: https://issues.jboss.org/browse/WFLY-10992
Project: WildFly
Issue Type: Bug
Components: MP OpenTracing
Affects Versions: 14.0.0.Beta2
Reporter: Jan Stourac
Assignee: Juraci Paixão Kröhling
Priority: Critical
Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
Size of the extra heap in use is about 25MB plus when compared to the initial size.
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
Also, when microprofile-opentracing-smallrye subsystem is removed via:
{code}
/subsystem=microprofile-opentracing-smallrye:remove()
{code}
then no more memory leak is detected, after adding the subsystem back, the leak is present again. Thus opentracing is involved somehow into this indeed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Juraci Paixão Kröhling (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Juraci Paixão Kröhling commented on WFLY-10991:
-----------------------------------------------
So, we probably have a leak in Jaeger itself. I'll investigate this as part of TRACING-231
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> Size of the extra heap in use is about 25MB plus when compared to the initial size.
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Also, when microprofile-opentracing-smallrye subsystem is removed via:
> {code}
> /subsystem=microprofile-opentracing-smallrye:remove()
> {code}
> then no more memory leak is detected, after adding the subsystem back, the leak is present again. Thus opentracing is involved somehow into this indeed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Jan Stourac commented on WFLY-10991:
------------------------------------
Thanks for the tip, [~juraci.costa].
I've tried to add {{MockTracer}}, {{MockTracer}} and {{MockTracerFactory}} into deployment as is done in the example you posted. Now the leak is suddenly fixed. The very same applies also in case when I replace {{MockTracer}} with {{NoopTracer}} - no leak is present anymore.
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> Size of the extra heap in use is about 25MB plus when compared to the initial size.
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Also, when microprofile-opentracing-smallrye subsystem is removed via:
> {code}
> /subsystem=microprofile-opentracing-smallrye:remove()
> {code}
> then no more memory leak is detected, after adding the subsystem back, the leak is present again. Thus opentracing is involved somehow into this indeed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Jan Stourac updated WFLY-10991:
-------------------------------
Description:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
Size of the extra heap in use is about 25MB plus when compared to the initial size.
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
Also, when microprofile-opentracing-smallrye subsystem is removed via:
{code}
/subsystem=microprofile-opentracing-smallrye:remove()
{code}
then no more memory leak is detected, after adding the subsystem back, the leak is present again. Thus opentracing is involved somehow into this indeed.
was:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
Size of the extra heap in use is about 25MB plus when compared to the initial size.
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
Also, when microprofile-opentracing-smallrye subsystem is removed via:
{code}
/subsystem=microprofile-opentracing-smallrye:remove()
{code}
then no more memory leak is detected. Thus opentracing is involved somehow into this indeed.
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> Size of the extra heap in use is about 25MB plus when compared to the initial size.
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Also, when microprofile-opentracing-smallrye subsystem is removed via:
> {code}
> /subsystem=microprofile-opentracing-smallrye:remove()
> {code}
> then no more memory leak is detected, after adding the subsystem back, the leak is present again. Thus opentracing is involved somehow into this indeed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Jan Stourac updated WFLY-10991:
-------------------------------
Description:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
Size of the extra heap in use is about 25MB plus when compared to the initial size.
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
Also, when microprofile-opentracing-smallrye subsystem is removed via:
{code}
/subsystem=microprofile-opentracing-smallrye:remove()
{code}
then no more memory leak is detected. Thus opentracing is involved somehow into this indeed.
was:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
Size of the extra heap in use is about 25MB plus when compared to the initial size.
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> Size of the extra heap in use is about 25MB plus when compared to the initial size.
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Also, when microprofile-opentracing-smallrye subsystem is removed via:
> {code}
> /subsystem=microprofile-opentracing-smallrye:remove()
> {code}
> then no more memory leak is detected. Thus opentracing is involved somehow into this indeed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Jan Stourac updated WFLY-10991:
-------------------------------
Description:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
Size of the extra heap in use is about 25MB plus when compared to the initial size.
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
was:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> Size of the extra heap in use is about 25MB plus when compared to the initial size.
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Juraci Paixão Kröhling (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Juraci Paixão Kröhling commented on WFLY-10991:
-----------------------------------------------
Could you try adding a TracerFactory service to your deployment, to get Jaeger out of the picture? We have examples of how to do that in the test suite and the NoopTracer could be used instead of the MockTracer.
https://github.com/wildfly/wildfly/blob/master/testsuite/integration/basi...
IIRC, no new Jaeger instances are created when a custom TraceFactory is provided, so, if the memory leak is related to Jaeger, it should disappear once you use NoopTracer.
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months
[JBoss JIRA] (WFLY-10991) Memory leak when deployment is redeployed multiple times
by Jan Stourac (JIRA)
[ https://issues.jboss.org/browse/WFLY-10991?page=com.atlassian.jira.plugin... ]
Jan Stourac updated WFLY-10991:
-------------------------------
Description:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
was:
It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue.
Brief description of the test:
# attached deployment is deployed
# System.gc() is called via EJB call
# it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
# deployment is redeployed 100 times
# calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
> Memory leak when deployment is redeployed multiple times
> --------------------------------------------------------
>
> Key: WFLY-10991
> URL: https://issues.jboss.org/browse/WFLY-10991
> Project: WildFly
> Issue Type: Bug
> Components: MP OpenTracing
> Affects Versions: 14.0.0.Beta2
> Reporter: Jan Stourac
> Assignee: Juraci Paixão Kröhling
> Priority: Critical
> Attachments: memoryLeaks.war, problematic-wf-build-heap-diff-start-end-org-filter.png, problematic-wf-build-heap-diff-start-end.png, sane-wf-build-heap-diff-start-end-org-filter.png, sane-wf-build-heap-diff-start-end.png
>
>
> It looks like redeploying attached deployment [^memoryLeaks.war] multiple times (our automated test does it 100 times), there seems to be some memory leak in the WildFly. Using git bisect, the commit that started to fail is [651dc8a2f0d4ccad6a35f7b1d28626b43ebb2a5d in WildFly repository|https://github.com/wildfly/wildfly/commits/651dc8a2f0d4ccad6a3...]. That is why I selected MP OpenTracing component for this issue. Actual testing deployment is very simple and contains just some jsp file with some EJB classes that are used to call GC and check current heap usage.
> Brief description of the test:
> # attached deployment is deployed
> # System.gc() is called via EJB call
> # it checks initial heap usage at the beginning of the test via EJB call that checks all instances of ManagementFactory.getMemoryPoolMXBeans()
> # deployment is redeployed 100 times
> # calls System.gc() via EJP call again and checks heap usage again via EJB call and compares the results
> I tried to check manually via [visualVM|https://visualvm.github.io/] tool. You can create a heapdump at the beginning of the test and at the end of the test. Then compare them easily and check the differences. Using a filter I can see quite a huge increase of instances of the {{org.jboss.modules.*}} classes. However, biggest memory footprint is huge increase of the instances of the {{java.util.HashMap$Node}}, {{char[]}}, {{java.util.HashMap$Node[]}} and few others. I don't see such huge increase of the instances of these classes when I use older build of WildFly prior to the problematic commit. To be honest I don't see direct link with the problematic commit I linked before. As the heapdumps are too big to be attached here, I am attaching some screenshots.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years, 10 months