<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<body link="#355491" alink="#4262a1" vlink="#355491" style="background: #e2e2e2; margin: 0; padding: 20px;">
<div>
<table cellpadding="0" bgcolor="#FFFFFF" border="0" cellspacing="0" style="border: 1px solid #dadada; margin-bottom: 30px; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td>
<table border="0" cellpadding="0" cellspacing="0" bgcolor="#FFFFFF" style="border: solid 2px #ccc; background: #dadada; width: 100%; -moz-border-radius: 6px; -webkit-border-radius: 6px;">
<tbody>
<tr>
<td bgcolor="#000000" valign="middle" height="58px" style="border-bottom: 1px solid #ccc; padding: 20px; -moz-border-radius-topleft: 3px; -moz-border-radius-topright: 3px; -webkit-border-top-right-radius: 5px; -webkit-border-top-left-radius: 5px;">
<h1 style="color: #333333; font: bold 22px Arial, Helvetica, sans-serif; margin: 0; display: block !important;">
<!-- To have a header image/logo replace the name below with your img tag -->
<!-- Email clients will render the images when the message is read so any image -->
<!-- must be made available on a public server, so that all recipients can load the image. -->
<a href="http://community.jboss.org/index.jspa" style="text-decoration: none; color: #E1E1E1">JBoss Community</a></h1>
</td>
</tr>
<tr>
<td bgcolor="#FFFFFF" style="font: normal 12px Arial, Helvetica, sans-serif; color:#333333; padding: 20px; -moz-border-radius-bottomleft: 4px; -moz-border-radius-bottomright: 4px; -webkit-border-bottom-right-radius: 5px; -webkit-border-bottom-left-radius: 5px;"><h3 style="margin: 10px 0 5px; font-size: 17px; font-weight: normal;">
Profiling AS 7 using OProfile on Fedora
</h3>
<span style="margin-bottom: 10px;">
modified by <a href="http://community.jboss.org/people/trustin">Trustin Lee</a> in <i>JBoss AS7 Development</i> - <a href="http://community.jboss.org/docs/DOC-16814">View the full document</a>
</span>
<hr style="margin: 20px 0; border: none; background-color: #dadada; height: 1px;">
<div class="jive-rendered-content"><p>Several times over the last year I have experimented with various tools for profiling AS.  One of the frustrating things with that, is that the overhead of all the traditional Java profiling solutions is so high, that its hard to get meaningful results.  With that in mind, I started to turn my attention to system profiling solutions, of which there are two for Linux.  One is called perf, and the other oprofile.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p><strong>Perf</strong>, is easier to use, but it currently does not have the ability to give Java method level detailed information.  It also has some very interesting capabilities around locks that I would like to exploit, but without the Java method level information its a non-starter.  I will definitely keep an eye on perf, because if it can add the method level information, then it may end up being the more useful too.  <strong>OProfile</strong>, on the other hand, does have the capability to show method level information.  The other item of note, is that both of these tools measure CPU utililzation, and can only give the method level information for methods that have been compiled to native code via the JIT (Just-in-Time compiler).  So, methods that are run interpreted will not show up in the details, and just be lumped together with all other intrepreted methods.  So, let's take a look at how to setup OProfile, and use it to profile a running JBoss Application Server.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>First, we need to install OProfile.  To do this, you can use yum, and the command is as follows:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">yum install oprofile oprofile-jit
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>The second package (<span style="font-family: 'courier new', courier;">oprofile-jit</span>) is necessary to get the Java method level symbol information.  Next, depending on the JVM that you are using you may have to install the debug information package for the JVM.  I always use OpenJDK, so I have it installed along with its debug information package.  The Sun JDK has symbol information in it, so it does not need a separate installation.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>To install the debug information package for OpenJDK, you first have to enable the debug information repository.  List the available repositories:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">$ ls /etc/yum.repos.d
dropbox.repo                 rpmfusion-free.repo
fedora-rawhide.repo          rpmfusion-free-updates.repo
fedora.repo                  rpmfusion-free-updates-testing.repo
fedora-updates.repo          rpmfusion-nonfree-rawhide.repo
fedora-updates-testing.repo  rpmfusion-nonfree.repo
google-chrome.repo           rpmfusion-nonfree-updates.repo
rpmfusion-free-rawhide.repo  rpmfusion-nonfree-updates-testing.repo
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>If you look into the content of a .repo file, it will look like the following:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">$ cat /etc/yum.repos.d/fedora.repo
...
[fedora-debuginfo]
name=Fedora $releasever - $basearch - Debug
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/releases/$releasever/Everything/$basearch/debug/
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$basearch
...
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>To enable the debuginfo repository, you must replace '<strong style="font-family: 'courier new', courier;">enabled=0</strong>' with '<span style="font-family: 'courier new', courier;"><strong>enabled=1</strong></span>' for every repository you enabled.  For example, I am using <span style="font-family: 'courier new', courier;">dropbox</span>, <span style="font-family: 'courier new', courier;">fedora</span>, <span style="font-family: 'courier new', courier;">fedora-updates</span>, <span style="font-family: 'courier new', courier;">google-chrome</span>, <span style="font-family: 'courier new', courier;">rpmfusion-free</span>, <span style="font-family: 'courier new', courier;">rpmfusion-free-updates</span>, <span style="font-family: 'courier new', courier;">rpmfusion-nonfree</span>, <span style="font-family: 'courier new', courier;">rpmfusion-nonfree-updates</span>, so I updated them all.  If the <span style="font-family: 'courier new', courier;">.repo</span> file does not have a section for a debuginfo repository, you might have to add it by yourself to enable it.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Now, as it turns out, just adding the repository is not enough.  If you want to keep the debug information packages in sync with their respective non-debug packages you also need to install a yum plugin that will search the debug information reposiitory for corresponding updates.  Otherwise, you will get them out of sync.  You can install that plugin via the following command:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">yum install yum-plugin-auto-update-debug-info
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Now that yum is setup properly, you can install the OpenJDK debug information package via the following command:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">yum install java-1.6.0-openjdk-debuginfo
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Now you have both OProfile and the necessary symbol information for the JDK installed, and we can move on to profiling a running AS instance.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>So, to setup an AS 7 server to be profiled we first have to start the JVM up with the oprofile agent.  This is done very simply by adding the following to the JVM command line:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code"># 64-bit JVM
-agentpath:/usr/lib64/oprofile/libjvmti_oprofile.so
# 32-bit JVM
-agentpath:/usr/lib/oprofile/libjvmti_oprofile.so
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>This can be found in the standalone.conf or domain.conf in the bin directory for AS 7.  The following is an example of what it looks like:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">#
# Specify options to pass to the Java VM.
#
if [ "x$JAVA_OPTS" = "x" ]; then
   JAVA_OPTS="-Xms10240m -Xmx10240m -XX:+UseLargePages -XX:+UseParallelOldGC"
   JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true"
   JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
   JAVA_OPTS="$JAVA_OPTS -agentpath:/usr/lib64/oprofile/libjvmti_oprofile.so"
fi
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>That's all there is to the setup for the AS, but this by itself does not enable profiling.  There is a separate daemon process that controls the profiling that has to be started up, and controlled to turn profiling on and off.  To start the daemon, and control when to start and stop profiling, there is a control program called opcontrol.  The following commands will get things rolling for you.  By the way, you can start up the application server at any time, after getting the agent configured on the JVM command line.  Just don't start your workload that you want to profile until you are ready with the below opcontrol commands.</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">opcontrol --start-daemon   <-- Start the deamon, but does not start profiling.
opcontrol --start  <-- Starts profiling - you want to issue this command after you have started your workload running, or potentially just before.
opcontrol --dump  <-- dumps the profiling data out to the default file (this can be done at any time during the workload, or right after the workload completes).
opcontrol --stop  <-- Stops profiling, but leaves the daemon running.
opcontrol --shutdown  <-- Shuts the daemon down, and it will no longer be running.
</code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>Once you have captured profiling data, and have dumped it, you can generate a report.  The simpliest report, is done with the following:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><pre class="jive-pre"><code class="jive-code">opreport -l --output-file=<filename></code></pre><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>This will give you a list, starting with the highest percentage use of CPU cycles to the lowest.  Keep in mind that this is a system wide profiler, so it will show everything that was running on the server, not just the java process.  There are many other reporting options that can be played with, and for reference, see the <a class="jive-link-external-small" href="http://oprofile.sourceforge.net/doc/index.html">OProfile documentation</a>:</p><p style="min-height: 8pt; height: 8pt; padding: 0px;"> </p><p>So, if you are interested in helping out with performance tuning AS 7, this is a good place to start.</p></div>
<div style="background-color: #f4f4f4; padding: 10px; margin-top: 20px;">
<p style="margin: 0;">Comment by <a href="http://community.jboss.org/docs/DOC-16814">going to Community</a></p>
<p style="margin: 0;">Create a new document in JBoss AS7 Development at <a href="http://community.jboss.org/choose-container!input.jspa?contentType=102&containerType=14&container=2225">Community</a></p>
</div></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>