<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="https://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
jboss 5.1.0: performance drops after one hour and strange GC behaviour
</h3>
<span style="margin-bottom: 10px;">
created by <a href="https://community.jboss.org/people/mattdarwin">Matt Darwin</a> in <i>Performance Tuning</i> - <a href="https://community.jboss.org/message/832880#832880">View the full discussion</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p id="yui_3_7_2_116_1376558508820_55"><span id="yui_3_7_2_116_1376558508820_71">We upgraded our software from jboss 4.0.5GA to 5.1.0GA and noticed that after one hour or so (or 90 minutes in some cases) performance drops dramatically.</span></p><p id="yui_3_7_2_116_1376558508820_97" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><br/><span id="yui_3_7_2_116_1376558508820_71"></span></p><p id="yui_3_7_2_116_1376558508820_100" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><span id="yui_3_7_2_116_1376558508820_71">At the same moment, the garbage collector logs show minor garbage collection times jumping from 0.01s to ~1.5s, with the amount of the heap being cleared each time reducing from ~400MB before to ~300MB after.  (see GC viewer graph 1)<br/></span></p><p id="yui_3_7_2_116_1376558508820_117" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><a href="https://community.jboss.org/servlet/JiveServlet/showImage/2-832880-21301/GC+viewer+graph+1.png"><span> https://community.jboss.org/servlet/JiveServlet/downloadImage/2-832880-21301/450-381/GC+viewer+graph+1.png </span></a></p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p id="yui_3_7_2_116_1376558508820_120" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><span id="yui_3_7_2_116_1376558508820_71">We think these are both symptoms of the the same underlying root cause</span>.<span class="tab" id="yui_3_7_2_116_1376558508820_127"><br/></span></p><p id="yui_3_7_2_116_1376558508820_78" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><br/><span id="yui_3_7_2_116_1376558508820_71"></span></p><p id="yui_3_7_2_116_1376558508820_81" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><span id="yui_3_7_2_116_1376558508820_71">jvm settings are:</span></p><p id="yui_3_7_2_116_1376558508820_85" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><span id="yui_3_7_2_116_1376558508820_71">-server -Xms2048m -Xmx2048m -XX:NewSize=384m -XX:MaxNewSize=384m -XX:SurvivorRatio=4 -XX:MinHeapFreeRatio=11 -XX:PermSize=80m -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+DisableExplicitGC -Djava.awt.headless=TRUE -DUseSunHttpHandler=TRUE -Dsun.net.client.defaultConnectTimeout=25000 -Dsun.net.client.defaultReadTimeout=50000 -Dfile.encoding=UTF-8 -Dvzzv.log.dir=${ercorebatch.log.dir} -Xloggc:${ercorebatch.log.dir}/gc.log  -Duser.language=it -Duser.region=IT -Duser.country=IT -DVFjavaWL=er.core.it<br/></span></p><p id="yui_3_7_2_116_1376558508820_57"><br/>The production environment is T5220 or T2000 hardware, with 32 bit SPARC, running a Solaris 10 virtual machine. jboss 5.0.1.GA, java 1.6.0_17  </p><p id="yui_3_7_2_116_1376558508820_91" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p id="yui_3_7_2_116_1376558508820_139">We set up a test environment consisting of 2 identical boxes, running the same software but one using jboss 4.0.5GA and one using jboss 5.0.1.GA.  They are VMWare VMs running on a HP ProLiant DL560 Gen8 with 4 x 2.2GHz Intel Xeon CPU E5-4620 and 64GB RAM.  Guest VMs are 4 vCPU, 4096MB RAM, CentOS 6.4.  </p><p id="yui_3_7_2_116_1376558508820_180"><br id="yui_3_7_2_116_1376558508820_185"/></p><p id="yui_3_7_2_116_1376558508820_182" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">We found that we could easily reproduce the problem in our environment.  The box which was running on 4.0.5 ran fine, but on jboss 5.0.1GA we saw the same strange GC behaviour.  Performance can't easily be tested in our environment since we don't have the same amount of load as production.</p><p id="yui_3_7_2_116_1376558508820_199" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><br id="yui_3_7_2_116_1376558508820_479"/></p><p id="yui_3_7_2_116_1376558508820_476" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">We don't think it's a memory leak, since after each major GC, the used heap size returns to the same size:</p><p style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><a href="https://community.jboss.org/servlet/JiveServlet/showImage/2-832880-21302/Screen+shot+2013-08-16+at+13.12.35.png"><span> https://community.jboss.org/servlet/JiveServlet/downloadImage/2-832880-21302/450-336/Screen+shot+2013-08-16+at+13.12.35.png </span></a></p><p id="yui_3_7_2_116_1376558508820_480" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><br id="yui_3_7_2_116_1376558508820_203"/></p><p id="yui_3_7_2_116_1376558508820_200" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">Analysing heap dumps taken pre- and post-apocalypse, we discovered the number of the following objects was different:</p><p id="yui_3_7_2_116_1376558508820_239" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">org.jboss.virtual.plugins.context.file.FileSystemContext</p><p id="yui_3_7_2_116_1376558508820_243" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">during the first hour, there are about 8 of them, and after the apocalypse hits, we see between 100 and 800.</p><p id="yui_3_7_2_116_1376558508820_265" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">Other than that, the heap dumps look quite similar, and the top objects are either java or jboss objects (ie no application classes)</p><p id="yui_3_7_2_116_1376558508820_289" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;"><br id="yui_3_7_2_116_1376558508820_294"/></p><p id="yui_3_7_2_116_1376558508820_291" style="color: #000000; font-size: 13.3333px; font-family: tahoma,new york,times,serif; font-style: normal;">Setting -Djboss.vfs.forceVfsJar=true on our test environment fixed the problem (ie the strange GC behaviour disappeared) but when applied in production, both the strange GC pattern and the performance problem remained - although the GC times didn't increase so much (to 0.3 seconds rather than 1.5 seconds).</p><p id="yui_3_7_2_116_1376558508820_149" style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p id="yui_3_7_2_116_1376558508820_410">In our test environment, we then deployed the same software in jboss 5.1.0 and found the same behaviour as with 5.0.1.  </p><p id="yui_3_7_2_116_1376558508820_448"><br id="yui_3_7_2_116_1376558508820_452"/></p><p>So the conclusions at this point are that there is something happening in jboss 5.x around the 60 / 90 minute mark which has an impact on both garbage collection and performance. </p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Reply to this message by <a href="https://community.jboss.org/message/832880#832880">going to Community</a></p>
<p style="margin: 0;">Start a new discussion in Performance Tuning at <a href="https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>