Setup
8-core Xeon / 10 GB RAM
RedHat EL 5.3 (64-bit)
JDK 1.5.0._22 (64-bit)
JBoss AS 4.2.3.GA with JBoss Messaging installed
JAVA_OPTS=”-server -Xrs -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -XX:NewRatio=1 -XX:ThreadStackSize=160 -XX:+UseParallelGC -Djava.awt.headless=true -Djava.library.path=$JBOSS_NATIVE_DIR -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000”
Description
We are running an EE web application on JBoss with an Apache httpd as a reverse proxy (using mod_jk) on another machine. Our stress tests have shown that the server can handle up to 400 concurrent requests, and we have therefore set the Tomcat AJP pool to that size.
When usage is normal, we can see about 40 active threads out of 400 being used, and the server works fine. Sometimes (we have not been able to discover a pattern – it may happen anytime from every couple of days to every couple of weeks) the number of active AJP threads reaches the maximum (400) and from that time onwards, server performance degrades significantly and does not get back to normal, even after the active AJP thread count has fallen to 10-20. Eventually, performance reaches unacceptable levels and we have to restart JBoss.
We have observed that when the problem occurs, the garbage collector behaves like this (and never recovers) :
/usr/java/jdk_5_64/jdk1.5.0_22/bin/jstat -gcutil 21801 5000 10
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 0.00 46.39 99.81 99.97 43111 839.352 14271 38382.902 39222.254
0.00 0.00 99.81 99.78 99.97 43111 839.352 14273 38385.821 39225.173
0.00 0.00 99.98 99.78 99.99 43111 839.352 14274 38388.653 39228.005
0.00 0.00 99.96 99.78 99.98 43111 839.352 14276 38394.327 39233.679
0.00 0.00 99.80 99.75 99.97 43111 839.352 14277 38397.220 39236.572
0.00 0.00 29.83 99.75 99.96 43111 839.352 14278 38402.896 39242.249
0.00 0.00 99.09 99.75 99.98 43111 839.352 14280 38405.740 39245.093
0.00 0.00 99.80 99.70 99.97 43111 839.352 14281 38408.664 39248.016
0.00 0.00 99.41 99.70 99.77 43111 839.352 14283 38414.305 39253.657
0.00 0.00 99.82 99.70 99.79 43111 839.352 14284 38417.120 39256.473
When all is normal, its behaviour is much better :
/usr/java/jdk_5_64/jdk1.5.0_22/bin/jstat -gcutil 28174 5000 10
S0 S1 E O P YGC YGCT FGC FGCT GCT
0.00 96.31 17.02 50.89 99.06 28939 522.830 1777 3073.229 3596.059
97.80 0.00 11.21 51.04 99.06 28942 522.874 1777 3073.229 3596.103
90.52 0.00 87.36 51.19 99.06 28944 522.904 1777 3073.229 3596.133
0.00 99.05 43.58 51.35 99.06 28947 522.951 1777 3073.229 3596.180
0.00 97.12 95.89 51.47 99.06 28949 522.983 1777 3073.229 3596.211
0.00 84.73 47.50 51.72 99.06 28953 523.045 1777 3073.229 3596.274
0.00 98.39 68.94 51.87 99.06 28955 523.078 1777 3073.229 3596.307
0.00 97.86 56.04 52.01 99.07 28957 523.111 1777 3073.229 3596.340
0.00 99.08 63.76 52.15 99.07 28959 523.141 1777 3073.229 3596.370
0.00 96.79 1.02 52.29 99.07 28961 523.171 1777 3073.229 3596.400
The whole Java heap space is all in RAM, i.e. the OS swap file is not used at all. We suspected a memory leak in the application, since the old generation space becomes almost full, but the JVM does not throw an OutOfMemory error; instead, it seems to free very little memory each time, resulting in a full GC.
Any ideas?
Thanks.
Constantine