[jboss-user] [Performance Tuning] - Full Garbage Collection every few seconds – GC frees very little memory (JBoss 4.2.3 with JDK 1.5.0 64-bit)

[Constantine Pinakoulakis]

Constantine Pinakoulakis [http://community.jboss.org/people/kopin] created the discussion

"Full Garbage Collection every few seconds – GC frees very little memory (JBoss 4.2.3 with JDK 1.5.0 64-bit)"

To view the discussion, visit: http://community.jboss.org/message/571975#571975

--------------------------------------------------------------
Setup
8-core Xeon / 10 GB RAM
RedHat EL 5.3 (64-bit)
JDK 1.5.0._22 (64-bit)
JBoss AS 4.2.3.GA with JBoss Messaging installed
JAVA_OPTS=”-server -Xrs -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -XX:NewRatio=1 -XX:ThreadStackSize=160 -XX:+UseParallelGC -Djava.awt.headless=true -Djava.library.path=$JBOSS_NATIVE_DIR -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000”

Description
We are running an EE web application on JBoss with an Apache httpd as a reverse proxy (using mod_jk) on another machine.  Our stress tests have shown that the server can handle up to 400 concurrent requests, and we have therefore set the Tomcat AJP pool to that size.

When usage is normal, we can see about 40 active threads out of 400 being used, and the server works fine.  Sometimes (we have not been able to discover a pattern – it may happen anytime from every couple of days to every couple of weeks) the number of active AJP threads reaches the maximum (400) and from that time onwards, server performance degrades significantly and does not get back to normal, even after the active AJP thread count has fallen to 10-20.  Eventually, performance reaches unacceptable levels and we have to restart JBoss.

We have observed that when the problem occurs, the garbage collector behaves like this (and never recovers) :
/usr/java/jdk_5_64/jdk1.5.0_22/bin/jstat -gcutil 21801 5000 10
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00  46.39  99.81  99.97  43111  839.352 14271 38382.902 39222.254
  0.00   0.00  99.81  99.78  99.97  43111  839.352 14273 38385.821 39225.173
  0.00   0.00  99.98  99.78  99.99  43111  839.352 14274 38388.653 39228.005
  0.00   0.00  99.96  99.78  99.98  43111  839.352 14276 38394.327 39233.679
  0.00   0.00  99.80  99.75  99.97  43111  839.352 14277 38397.220 39236.572
  0.00   0.00  29.83  99.75  99.96  43111  839.352 14278 38402.896 39242.249
  0.00   0.00  99.09  99.75  99.98  43111  839.352 14280 38405.740 39245.093
  0.00   0.00  99.80  99.70  99.97  43111  839.352 14281 38408.664 39248.016
  0.00   0.00  99.41  99.70  99.77  43111  839.352 14283 38414.305 39253.657
  0.00   0.00  99.82  99.70  99.79  43111  839.352 14284 38417.120 39256.473

When all is normal, its behaviour is much better :
/usr/java/jdk_5_64/jdk1.5.0_22/bin/jstat -gcutil 28174  5000 10
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
  0.00  96.31  17.02  50.89  99.06  28939  522.830  1777 3073.229 3596.059
 97.80   0.00  11.21  51.04  99.06  28942  522.874  1777 3073.229 3596.103
 90.52   0.00  87.36  51.19  99.06  28944  522.904  1777 3073.229 3596.133
  0.00  99.05  43.58  51.35  99.06  28947  522.951  1777 3073.229 3596.180
  0.00  97.12  95.89  51.47  99.06  28949  522.983  1777 3073.229 3596.211
  0.00  84.73  47.50  51.72  99.06  28953  523.045  1777 3073.229 3596.274
  0.00  98.39  68.94  51.87  99.06  28955  523.078  1777 3073.229 3596.307
  0.00  97.86  56.04  52.01  99.07  28957  523.111  1777 3073.229 3596.340
  0.00  99.08  63.76  52.15  99.07  28959  523.141  1777 3073.229 3596.370
  0.00  96.79   1.02  52.29  99.07  28961  523.171  1777 3073.229 3596.400

The whole Java heap space is all in RAM, i.e. the OS swap file is not used at all.  We suspected a memory leak in the application, since the old generation space becomes almost full, but the JVM does not throw an OutOfMemory error; instead, it seems to free very little memory each time, resulting in a full GC.

Any ideas?

Thanks.
Constantine
--------------------------------------------------------------

Reply to this message by going to Community
[http://community.jboss.org/message/571975#571975]

Start a new discussion in Performance Tuning at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20101119/39208049/attachment.html