[jboss-jira] [JBoss JIRA] Created: (JBAS-8066) Big memory usuage increase after upgraded from 4.2.1.GA to 5.1.0.GA

[Samuel Cai (JIRA)]

Big memory usuage increase after upgraded from 4.2.1.GA to 5.1.0.GA
-------------------------------------------------------------------

                 Key: JBAS-8066
                 URL: https://jira.jboss.org/browse/JBAS-8066
             Project: JBoss Application Server
          Issue Type: Bug
      Security Level: Public (Everyone can see)
          Components: VFS
    Affects Versions: JBossAS-5.1.0.GA
         Environment: JDK 1.6.0_20
Redhat (not sure what version) 2.6.9-78.ELsmp
Use the "web" instance, not "Default", I only have one additional war file deployed, there're around 80 jars under WEB-INF/lib in it.

JVM parameter:
-XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -verbose:gc -Dfile.encoding=iso-8859-1 -server -Djava.net.preferIPv4Stack=true -Doracle.jdbc.V8Compatible=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dnetworkaddress.cache.ttl=300 -Xss128k -Xmn500m -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -XX:+PrintGCDetails -XX:PermSize=256m -XX:MaxPermSize=256m -Xms1500m -Xmx1500m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
            Reporter: Samuel Cai


I had similar problem as issue https://jira.jboss.org/browse/JBAS-6549
But using JDK 1.6.0_16 and 1.6.0_17 doesn't help (I was using 1.6.0_20), not sure if anyone met same problem, below is my finding:
4.2.1.GA:
VSZ: 1972516
RSS: 683992

5.1.0.GA:
VSZ: 2546604
RSS: 1359864
(These values changed every time server started, range around 2.5G - 2.9G for VSZ)

The value was taken after deployment was finished and server was started, use "ps aux".
The wars deployed in 4 and 5 are not the same, because I precompiled jsp, and 5 uses new jar to compile. 

[Found a way to cut down]:
I heard that VFS in JBoss 5 may introduce some overhead, so I tried the options it provided, and found if I set jboss.vfs.forceCopy to false:
VSZ: 2057408
RSS: 1100008

I'm confused, according to the document, when setting forceCopy to false, JBoss will use much more memory, but now seems not.
Now our QA is doing performance and pressure testings based on this configuration, will know the result soon.

[More finding]:

The heap usuage is normal, I've checked via JMX console. And it should be normal since I've limited the Xms and Xmx.
The more memory usuage should be introduced by JBoss, maybe via JNI.
By using pmap and comparing the result, I found the main differences between JBoss 4, 5 and 5 (forceCopy=false) is: in 5, there are many 1M block reserved, in 5 (forceCopy=false), less block, in 4, hardly fine, here is example of the 1M block:

3d000000    996K rwx--    [ anon ]
3d0f9000     28K -----    [ anon ]
3d100000    996K rwx--    [ anon ]
3d1f9000     28K -----    [ anon ]
3d200000   1016K rwx--    [ anon ]
3d2fe000      8K -----    [ anon ]
3d300000   1012K rwx--    [ anon ]
3d3fd000     12K -----    [ anon ]

I've attached the pmap log.

[My Guess]:
Memory leak in JBoss due to VFS?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira