[jboss-user] [Performance Tuning] - Help getting rid of a memory leak

[Angelo Marletta]

Angelo Marletta [http://community.jboss.org/people/JohnnyStrozzino] created the discussion

"Help getting rid of a memory leak"

To view the discussion, visit: http://community.jboss.org/message/555304#555304

--------------------------------------------------------------
Hello everybody,

 
I've recently encountered some problems with memory leaks on JBoss, and even after having googled a lot, I still cannot manage to solve them.
 
My application has several EJB3 web services and handles about 100k soap requests per day. Each request involves reading/writing Oracle db, using JPA and EJB3 over Hibernate. But after 10 days of production it runs out of memory and I must restart the application server. The application is normally deployed on a 4 nodes cluster.
 
 
So I followed this strategy:
1) Run only 1 cluster node, for 4 days
2) GC and get an head dump every morning
3) Compare the head dumps with Eclipse Memory Analyzer in order to understand what's growing
 
 
I found out that there is an instance of class org.jboss.classloader.spi.base.BaseClassLoader which after 1 day is retaining 81 MB of memory, after 2 days 168 MB, after 3 days 245 MB. This object has a java.util.concurrent.ConcurrentHashMap called "blacklist" which actually is responsible for almost all the used memory (respectively 70 MB, 157 MB, 234 MB) of the container object.
 
 
This blacklist is a ConcurrentHashMap<String,String> with each key equals to its value. The latest snapshot contains over 1 million of Strings.
All Strings objects inside the map seem to appear in the following format: $obj$XXXXXXXX/E1/E2/.../En.class
 
where [XXXXXXXX] is an hexadecimal number, and [E1]...[En] are names of application JPA entities, sorted in a way thet reflects the real relationship order. For instance, entity E1 has a reference to entity E2, which has a reference to entity E3, and so on.
 
 
Some info about my setup:
 
Max heap size: 1.2 GB
JBoss 5.1.0-GA updated with JBoss EJB3 Plugin 1.0.19
Quartz 1.8.3 in cluster configuration
Hot deployment disabled
Oracle Driver: 11g
JDK and JRE 1.5 (latest) 32 bit
CentOS Linux 5.3
 
 
Any hint to help me figure out what's wrong in my code, or in my configuration is very welcome.
In attach there are three screenshots taken in different days.
 
 
Thank you

--------------------------------------------------------------

Reply to this message by going to Community
[http://community.jboss.org/message/555304#555304]

Start a new discussion in Performance Tuning at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20100801/96abb984/attachment.html