<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
Full Garbage Collection every few seconds – GC frees very little memory (JBoss 4.2.3 with JDK 1.5.0 64-bit)
</h3>
<span style="margin-bottom: 10px;">
created by <a href="http://community.jboss.org/people/kopin">Constantine Pinakoulakis</a> in <i>Performance Tuning</i> - <a href="http://community.jboss.org/message/571975#571975">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p class="MsoNormal"><span style="text-decoration: underline;">Setup</span></p><p class="MsoNormal">8-core Xeon / 10 GB RAM</p><p class="MsoNormal">RedHat EL 5.3 (64-bit)</p><p class="MsoNormal">JDK 1.5.0._22 (64-bit)</p><p class="MsoNormal">JBoss AS 4.2.3.GA with JBoss Messaging installed</p><p class="MsoNormal">JAVA_OPTS=”-server -Xrs -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -XX:NewRatio=1 -XX:ThreadStackSize=160 -XX:+UseParallelGC -Djava.awt.headless=true -Djava.library.path=$JBOSS_NATIVE_DIR -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000”</p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal"><span style="text-decoration: underline;">Description</span></p><p class="MsoNormal">We are running an EE web application on JBoss with an Apache httpd as a reverse proxy (using mod_jk) on another machine.  Our stress tests have shown that the server can handle up to 400 concurrent requests, and we have therefore set the Tomcat AJP pool to that size.</p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal">When usage is normal, we can see about 40 active threads out of 400 being used, and the server works fine.  Sometimes (we have not been able to discover a pattern – it may happen anytime from every couple of days to every couple of weeks) the number of active AJP threads reaches the maximum (400) and from that time onwards, server performance degrades significantly and does not get back to normal, even after the active AJP thread count has fallen to 10-20.  Eventually, performance reaches unacceptable levels and we have to restart JBoss.</p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal">We have observed that when the problem occurs, the garbage collector behaves like this (and never recovers) :</p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">/usr/java/jdk_5_64/jdk1.5.0_22/bin/jstat -gcutil 21801 5000 10</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  46.39  99.81  99.97  43111  839.352 14271 38382.902 39222.254</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.81  99.78  99.97  43111  839.352 14273 38385.821 39225.173</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.98  99.78  99.99  43111  839.352 14274 38388.653 39228.005</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.96  99.78  99.98  43111  839.352 14276 38394.327 39233.679</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.80  99.75  99.97  43111  839.352 14277 38397.220 39236.572</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  29.83  99.75  99.96  43111  839.352 14278 38402.896 39242.249</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.09  99.75  99.98  43111  839.352 14280 38405.740 39245.093</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.80  99.70  99.97  43111  839.352 14281 38408.664 39248.016</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00   0.00  99.41  99.70  99.77  43111  839.352 14283 38414.305 39253.657</span></p><p class="MsoNormal"><span style="font-family: courier new,courier;">  0.00   0.00  99.82  99.70  99.79  43111  839.352 14284 38417.120 39256.473</span></p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal">When all is normal, its behaviour is much better :</p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">/usr/java/jdk_5_64/jdk1.5.0_22/bin/jstat -gcutil 28174  5000 10</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  96.31  17.02  50.89  99.06  28939  522.830  1777 3073.229 3596.059</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;"> 97.80   0.00  11.21  51.04  99.06  28942  522.874  1777 3073.229 3596.103</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;"> 90.52   0.00  87.36  51.19  99.06  28944  522.904  1777 3073.229 3596.133</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  99.05  43.58  51.35  99.06  28947  522.951  1777 3073.229 3596.180</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  97.12  95.89  51.47  99.06  28949  522.983  1777 3073.229 3596.211</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  84.73  47.50  51.72  99.06  28953  523.045  1777 3073.229 3596.274</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  98.39  68.94  51.87  99.06  28955  523.078  1777 3073.229 3596.307</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  97.86  56.04  52.01  99.07  28957  523.111  1777 3073.229 3596.340</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  99.08  63.76  52.15  99.07  28959  523.141  1777 3073.229 3596.370</span></p><p class="MsoNormal" style="margin-bottom: 0.0001pt;"><span style="font-family: courier new,courier;">  0.00  96.79   1.02  52.29  99.07  28961  523.171  1777 3073.229 3596.400</span></p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal">The whole Java heap space is all in RAM, i.e. the OS swap file is not used at all.  We suspected a memory leak in the application, since the old generation space becomes almost full, but the JVM does not throw an OutOfMemory error; instead, it seems to free very little memory each time, resulting in a full GC.</p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal">Any ideas?</p><p class="MsoNormal" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p class="MsoNormal">Thanks.</p><p class="MsoNormal">Constantine</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="http://community.jboss.org/message/571975#571975">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in Performance Tuning at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>