10:03 a.m.
Hi,
during weekend we have been investigating some rather big memory leaks we
have in current WildFly code.
What our research has found out is that we had problems in
- JPA integration
- Validation integration
- undertow / undertow integration
- msc
- resteasy
- hibernate
For now we have fixed most of the issues in integration code which can be
found at
https://github.com/ctomc/wildfly/commits/mem-leak
undertow
https://github.com/undertow-io/undertow/commits/master
msc
https://github.com/jbossas/jboss-msc/commit/1bcdbd0d874a5d5e22ef2a00fa1eb...
RestEasy
https://github.com/resteasy/Resteasy/pull/344
I am not sure for RestEasy fixes and should code be properly fixed by
people that know code bit better.
All this components need new release to be cut before they can be included
into WildFly.
I am not sure if this will hold WildFly Alpha 3 release or not, IMHO it
should.
Also beyond this leaks we also found that we leak somewhere deep in
hibernate integration and properties(or better copy of them) that are
passed as part of JPA initialization remain referenced by hibernate service
loader after application has already been undeployed.
This is problematic as part of this properties are instances of BeanManager
and ValidatorFactory and similar if they would be just primitive values it
would be lesser of issue.
I was unable to find proper place to fix this. So I would like to ask
people that are more familiar with that area to take a look.
This is GC root from Yourkit profile session:
Paths from "GC Roots" to "Object SerializableValidatorFactory
#323736"
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
|
Name | Retained Size |
Shallow Size |
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
|
+---org.jboss.as.jpa.validator.SerializableValidatorFactory
| 16 | 16 |
|
|
| | |
| +---value of
java.util.HashMap$Entry
| 64 | 24 |
|
|
| | |
| +---[83] of
java.util.HashMap$Entry[256]
| 4.088 | 1.040 |
|
|
| | |
| +---table of
java.util.HashMap
| 4.152 | 48 |
|
|
| | |
| +---configurationValues of
org.hibernate.boot.registry.internal.StandardServiceRegistryImpl |
15.688 | 24 |
|
|
| | |
| +---serviceRegistry of
org.jboss.as.jpa.hibernate4.JBossAppServerJtaPlatform
| 15.776 | 40 |
|
|
| | |
| +---delegate of
org.jboss.as.jpa.hibernate4.DefaultJtaPlatform
| 16.128 | 352 |
|
|
| | |
| +---[7] of
java.lang.Object[2560]
| 258.520 | 10.256 |
|
|
| | |
| +---elementData of
java.util.Vector |
258.544 | 24 |
|
|
| | |
| +---classes of
org.jboss.modules.ModuleClassLoader
| 518.696 | 80 |
|
|
| | |
| +---referent of
java.util.WeakHashMap$Entry
| 248 | 40 |
|
|
| | |
| +---next of
java.util.WeakHashMap$Entry
| 976 | 40 |
|
|
| | |
| +---[29] of
java.util.WeakHashMap$Entry[32] |
9.720 | 144 |
|
|
| | |
| +---table of
java.util.WeakHashMap |
9.808 | 56 |
|
|
| | |
| +---loaderToCache of java.lang.reflect.Proxy
[Class] | 14.864 | 344 |
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
I was testing with
http://www.jboss.org/jdf/quickstarts/jboss-as-quickstart/kitchensink-html...
to reproduce it all you need to do is deploy app, undeploy it and
profile app server. There should be no BeanManager & ValidatorFactory
instances laying around.
I would also like to encgurage everyone else to profile their part of code
just to make sure we haven't missed anything.
--
tomaz
10:24 a.m.
WFLY-1697 is for the JPA memory leak. Thanks for finding the leaks!
Scott
On 07/15/2013 11:03 AM, Tomaž Cerar wrote:
Hi,
during weekend we have been investigating some rather big memory leaks
we have in current WildFly code.
What our research has found out is that we had problems in
- JPA integration
- Validation integration
- undertow / undertow integration
- msc
- resteasy
- hibernate
For now we have fixed most of the issues in integration code which can
be found at
https://github.com/ctomc/wildfly/commits/mem-leak
undertow
https://github.com/undertow-io/undertow/commits/master
msc
https://github.com/jbossas/jboss-msc/commit/1bcdbd0d874a5d5e22ef2a00fa1eb...
RestEasy
https://github.com/resteasy/Resteasy/pull/344
I am not sure for RestEasy fixes and should code be properly fixed by
people that know code bit better.
All this components need new release to be cut before they can be
included into WildFly.
I am not sure if this will hold WildFly Alpha 3 release or not, IMHO it
should.
Also beyond this leaks we also found that we leak somewhere deep in
hibernate integration and properties(or better copy of them) that are
passed as part of JPA initialization remain referenced by hibernate
service loader after application has already been undeployed.
This is problematic as part of this properties are instances of
BeanManager and ValidatorFactory and similar if they would be just
primitive values it would be lesser of issue.
I was unable to find proper place to fix this. So I would like to ask
people that are more familiar with that area to take a look.
This is GC root from Yourkit profile session:
Paths from "GC Roots" to "Object SerializableValidatorFactory
#323736"
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
|
Name | Retained Size
| Shallow Size |
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
|
+---org.jboss.as.jpa.validator.SerializableValidatorFactory
| 16 | 16 |
|
|
| | |
| +---value of
java.util.HashMap$Entry
| 64 | 24 |
|
|
| | |
| +---[83] of
java.util.HashMap$Entry[256]
| 4.088 | 1.040 |
|
|
| | |
| +---table of
java.util.HashMap
| 4.152 | 48 |
|
|
| | |
| +---configurationValues of
org.hibernate.boot.registry.internal.StandardServiceRegistryImpl
| 15.688 | 24 |
|
|
| | |
| +---serviceRegistry of
org.jboss.as.jpa.hibernate4.JBossAppServerJtaPlatform
| 15.776 | 40 |
|
|
| | |
| +---delegate of
org.jboss.as.jpa.hibernate4.DefaultJtaPlatform
| 16.128 | 352 |
|
|
| | |
| +---[7] of
java.lang.Object[2560]
| 258.520 | 10.256 |
|
|
| | |
| +---elementData of
java.util.Vector
| 258.544 | 24 |
|
|
| | |
| +---classes of
org.jboss.modules.ModuleClassLoader
| 518.696 | 80 |
|
|
| | |
| +---referent of
java.util.WeakHashMap$Entry
| 248 | 40 |
|
|
| | |
| +---next of
java.util.WeakHashMap$Entry
| 976 | 40 |
|
|
| | |
| +---[29] of
java.util.WeakHashMap$Entry[32]
| 9.720 | 144 |
|
|
| | |
| +---table of
java.util.WeakHashMap |
9.808 | 56 |
|
|
| | |
| +---loaderToCache of
java.lang.reflect.Proxy [Class] | 14.864
| 344 |
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
I was testing with
http://www.jboss.org/jdf/quickstarts/jboss-as-quickstart/kitchensink-html...
and to reproduce it all you need to do is deploy app, undeploy it and
profile app server. There should be no BeanManager & ValidatorFactory
instances laying around.
I would also like to encgurage everyone else to profile their part of
code just to make sure we haven't missed anything.
--
tomaz
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
5:48 p.m.
Do we want a temporary workaround for the JPA leak? I have a temp
workaround that I could create a pr for.
On 07/15/2013 11:24 AM, Scott Marlow wrote:
WFLY-1697 is for the JPA memory leak. Thanks for finding the leaks!
Scott
On 07/15/2013 11:03 AM, Tomaž Cerar wrote:
> Hi,
>
> during weekend we have been investigating some rather big memory leaks
> we have in current WildFly code.
>
> What our research has found out is that we had problems in
>
> - JPA integration
> - Validation integration
> - undertow / undertow integration
> - msc
> - resteasy
> - hibernate
>
> For now we have fixed most of the issues in integration code which can
> be found at
> https://github.com/ctomc/wildfly/commits/mem-leak
>
> undertow
> https://github.com/undertow-io/undertow/commits/master
>
> msc
> https://github.com/jbossas/jboss-msc/commit/1bcdbd0d874a5d5e22ef2a00fa1eb...
>
> RestEasy
> https://github.com/resteasy/Resteasy/pull/344
>
> I am not sure for RestEasy fixes and should code be properly fixed by
> people that know code bit better.
>
> All this components need new release to be cut before they can be
> included into WildFly.
> I am not sure if this will hold WildFly Alpha 3 release or not, IMHO it
> should.
>
> Also beyond this leaks we also found that we leak somewhere deep in
> hibernate integration and properties(or better copy of them) that are
> passed as part of JPA initialization remain referenced by hibernate
> service loader after application has already been undeployed.
> This is problematic as part of this properties are instances of
> BeanManager and ValidatorFactory and similar if they would be just
> primitive values it would be lesser of issue.
> I was unable to find proper place to fix this. So I would like to ask
> people that are more familiar with that area to take a look.
>
> This is GC root from Yourkit profile session:
>
> Paths from "GC Roots" to "Object SerializableValidatorFactory
#323736"
>
>
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
> |
> Name | Retained Size
> | Shallow Size |
>
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
> |
> +---org.jboss.as.jpa.validator.SerializableValidatorFactory
> | 16 | 16 |
> |
> |
> | | |
> | +---value of
> java.util.HashMap$Entry
> | 64 | 24 |
> |
> |
> | | |
> | +---[83] of
> java.util.HashMap$Entry[256]
> | 4.088 | 1.040 |
> |
> |
> | | |
> | +---table of
> java.util.HashMap
> | 4.152 | 48 |
> |
> |
> | | |
> | +---configurationValues of
> org.hibernate.boot.registry.internal.StandardServiceRegistryImpl
> | 15.688 | 24 |
> |
> |
> | | |
> | +---serviceRegistry of
> org.jboss.as.jpa.hibernate4.JBossAppServerJtaPlatform
> | 15.776 | 40 |
> |
> |
> | | |
> | +---delegate of
> org.jboss.as.jpa.hibernate4.DefaultJtaPlatform
> | 16.128 | 352 |
> |
> |
> | | |
> | +---[7] of
> java.lang.Object[2560]
> | 258.520 | 10.256 |
> |
> |
> | | |
> | +---elementData of
> java.util.Vector
> | 258.544 | 24 |
> |
> |
> | | |
> | +---classes of
> org.jboss.modules.ModuleClassLoader
> | 518.696 | 80 |
> |
> |
> | | |
> | +---referent of
> java.util.WeakHashMap$Entry
> | 248 | 40 |
> |
> |
> | | |
> | +---next of
> java.util.WeakHashMap$Entry
> | 976 | 40 |
> |
> |
> | | |
> | +---[29] of
> java.util.WeakHashMap$Entry[32]
> | 9.720 | 144 |
> |
> |
> | | |
> | +---table of
> java.util.WeakHashMap |
> 9.808 | 56 |
> |
> |
> | | |
> | +---loaderToCache of
> java.lang.reflect.Proxy [Class] | 14.864
> | 344 |
>
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
>
> I was testing with
> http://www.jboss.org/jdf/quickstarts/jboss-as-quickstart/kitchensink-html...
> and to reproduce it all you need to do is deploy app, undeploy it and
> profile app server. There should be no BeanManager & ValidatorFactory
> instances laying around.
>
> I would also like to encgurage everyone else to profile their part of
> code just to make sure we haven't missed anything.
>
> --
> tomaz
>
>
>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
11:06 a.m.
On 07/15/2013 05:03 PM, Tomaž Cerar wrote:
Hi,
during weekend we have been investigating some rather big memory leaks
we have in current WildFly code.
What our research has found out is that we had problems in
- JPA integration
- Validation integration
- undertow / undertow integration
- msc
- resteasy
- hibernate
For now we have fixed most of the issues in integration code which can
be found at
https://github.com/ctomc/wildfly/commits/mem-leak
undertow
https://github.com/undertow-io/undertow/commits/master
msc
https://github.com/jbossas/jboss-msc/commit/1bcdbd0d874a5d5e22ef2a00fa1eb...
Thanks for identification of this MSC leak.
RestEasy
https://github.com/resteasy/Resteasy/pull/344
I am not sure for RestEasy fixes and should code be properly fixed by
people that know code bit better.
All this components need new release to be cut before they can be
included into WildFly.
I am not sure if this will hold WildFly Alpha 3 release or not, IMHO it
should.
Also beyond this leaks we also found that we leak somewhere deep in
hibernate integration and properties(or better copy of them) that are
passed as part of JPA initialization remain referenced by hibernate
service loader after application has already been undeployed.
This is problematic as part of this properties are instances of
BeanManager and ValidatorFactory and similar if they would be just
primitive values it would be lesser of issue.
I was unable to find proper place to fix this. So I would like to ask
people that are more familiar with that area to take a look.
This is GC root from Yourkit profile session:
Paths from "GC Roots" to "Object SerializableValidatorFactory
#323736"
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
|
Name | Retained Size
| Shallow Size |
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
|
+---org.jboss.as.jpa.validator.SerializableValidatorFactory
| 16 | 16 |
|
|
| | |
| +---value of
java.util.HashMap$Entry
| 64 | 24 |
|
|
| | |
| +---[83] of
java.util.HashMap$Entry[256]
| 4.088 | 1.040 |
|
|
| | |
| +---table of
java.util.HashMap
| 4.152 | 48 |
|
|
| | |
| +---configurationValues of
org.hibernate.boot.registry.internal.StandardServiceRegistryImpl
| 15.688 | 24 |
|
|
| | |
| +---serviceRegistry of
org.jboss.as.jpa.hibernate4.JBossAppServerJtaPlatform
| 15.776 | 40 |
|
|
| | |
| +---delegate of
org.jboss.as.jpa.hibernate4.DefaultJtaPlatform
| 16.128 | 352 |
|
|
| | |
| +---[7] of
java.lang.Object[2560]
| 258.520 | 10.256 |
|
|
| | |
| +---elementData of
java.util.Vector
| 258.544 | 24 |
|
|
| | |
| +---classes of
org.jboss.modules.ModuleClassLoader
| 518.696 | 80 |
|
|
| | |
| +---referent of
java.util.WeakHashMap$Entry
| 248 | 40 |
|
|
| | |
| +---next of
java.util.WeakHashMap$Entry
| 976 | 40 |
|
|
| | |
| +---[29] of
java.util.WeakHashMap$Entry[32]
| 9.720 | 144 |
|
|
| | |
| +---table of
java.util.WeakHashMap |
9.808 | 56 |
|
|
| | |
| +---loaderToCache of
java.lang.reflect.Proxy [Class] | 14.864
| 344 |
+-------------------------------------------------------------------------------------------------------+-----------------+----------------+
I was testing with
http://www.jboss.org/jdf/quickstarts/jboss-as-quickstart/kitchensink-html...
and to reproduce it all you need to do is deploy app, undeploy it and
profile app server. There should be no BeanManager & ValidatorFactory
instances laying around.
I would also like to encgurage everyone else to profile their part of
code just to make sure we haven't missed anything.
--
tomaz
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Richard Opalka
Principal Software Engineer
JBoss, by Red Hat
Cell: +420 731 186 942
4171
days inactive
4178
days old
3 comments
3 participants
participants (3)
-
Richard Opalka -
Scott Marlow -
Tomaž Cerar