<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
h1
{mso-style-priority:9;
mso-style-link:"Heading 1 Char";
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:24.0pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
h3
{mso-style-priority:9;
mso-style-link:"Heading 3 Char";
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:13.5pt;
font-family:"Times New Roman","serif";
font-weight:bold;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#355491;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#355491;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.Heading1Char
{mso-style-name:"Heading 1 Char";
mso-style-priority:9;
mso-style-link:"Heading 1";
font-family:"Cambria","serif";
color:#365F91;
font-weight:bold;}
span.Heading3Char
{mso-style-name:"Heading 3 Char";
mso-style-priority:9;
mso-style-link:"Heading 3";
font-family:"Cambria","serif";
color:#4F81BD;
font-weight:bold;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:153493782;
mso-list-type:hybrid;
mso-list-template-ids:-148053278 233601510 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l0:level1
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:54.0pt;
text-indent:-36.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="#E2E2E2" lang="EN-GB" link="#355491" vlink="#355491">
<div class="Section1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Usually I do the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:54.0pt;text-indent:-36.0pt;
mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">i.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Check for full garbage collection events using ‘jstat –gcutil ‘<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:54.0pt;text-indent:-36.0pt;
mso-list:l0 level1 lfo1">
<![if !supportLists]><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">ii.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D">Take several thread dumps 15 seconds apart - use Thread Dump Analyser to look for locked threads and also look for busy threads top –H –p pid<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:
"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;
font-family:"Tahoma","sans-serif""> jboss-user-bounces@lists.jboss.org [mailto:jboss-user-bounces@lists.jboss.org]
<b>On Behalf Of </b>Amol Walanj<br>
<b>Sent:</b> 08 May 2012 10:03<br>
<b>To:</b> User development<br>
<b>Subject:</b> [jboss-user] [Performance Tuning] - Re: High CPU Utilization<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:white;border:solid #DADADA 1.0pt">
<tbody>
<tr>
<td style="border:none;padding:0cm 0cm 0cm 0cm">
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#DADADA;border:solid #CCCCCC 1.5pt">
<tbody>
<tr>
<td style="border:none;border-bottom:solid #CCCCCC 1.0pt;background:black;
padding:15.0pt 15.0pt 15.0pt 15.0pt">
<h1 style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:16.5pt;
font-family:"Arial","sans-serif";color:#333333"><a href="https://community.jboss.org/index.jspa"><span style="color:#E1E1E1;
text-decoration:none">JBoss Community</span></a><o:p></o:p></span></h1>
</td>
</tr>
<tr>
<td style="border:none;background:white;padding:15.0pt 15.0pt 15.0pt 15.0pt;
-moz-border-radius-bottomleft: 4px;-moz-border-radius-bottomright: 4px;
-webkit-border-bottom-right-radius: 5px;-webkit-border-bottom-left-radius: 5px">
<h3 style="mso-margin-top-alt:7.5pt;margin-right:0cm;margin-bottom:3.75pt;
margin-left:0cm">
<span style="font-size:13.0pt;font-family:"Arial","sans-serif";
color:#333333;font-weight:normal">Re: High CPU Utilization
<o:p></o:p></span></h3>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">created by
<a href="https://community.jboss.org/people/walanj.amol">Amol Walanj</a> in <i>Performance Tuning</i> -
<a href="https://community.jboss.org/message/734226#734226">View the full discussion</a>
<o:p></o:p></span></p>
<div class="MsoNormal" align="center" style="mso-margin-top-alt:15.0pt;
margin-right:0cm;margin-bottom:15.0pt;margin-left:0cm;text-align:center">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">
<hr size="1" width="100%" align="center">
</span></div>
<div>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">Hi , Thanks for quick reply.<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">Its during the runtime when we fired the bulk orders. (Client with 20 threads)<o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">What I had observed from jconsole is , the tomcat threads (http-0.0.0.0-8080-*) are taking more time to complete.<o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">I had tried to find the top 5 threads consuming the CPU during this run using the shell script.<o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">The Partial O/P is as follows :<o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"> PID USER <o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">PR NI VIRT RES SHR S %CPU %MEM<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">TIME+ COMMAND
<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">11935 sis <o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">15 0 6342m 3.7g 11m S<strong> 29.5</strong> 47.0 0:10.37 java
<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><strong>11681</strong> sis <o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">15 0 6342m 3.7g 11m R <strong>25.6</strong> 47.0 0:27.65 java
<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">11662 sis <o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">15 0 6342m 3.7g 11m S<strong> 23.6</strong> 47.0 0:31.89 java
<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">11682 sis <o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">15 0 6342m 3.7g 11m S <strong>21.6</strong> 47.0 0:31.34 java
<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">11663 sis <o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal">15 0 6342m 3.7g 11m S <strong>21.6</strong> 47.0 0:33.84 java
<o:p></o:p></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
</tbody>
</table>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">Pasting here the thread Dump for one of the PID.The others dump also shows the same dump.<o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">========> Java LWP:
<strong><span style="font-family:
"Arial","sans-serif"">11681 </span></strong>- Native Thread ID=2da1<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">"http-0.0.0.0-8080-15" daemon prio=10 tid=0x00002aaac0877000 nid=0x2da1 waiting on condition [0x0000000042e7f000]<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> java.lang.Thread.State: WAITING (parking)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at sun.misc.Unsafe.park(Native Method)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> - parking to wait for <0x0000000746df1920> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.security.plugins.JBossAuthorizationManager.internalAuthorization(JBossAuthorizationManager.java:470)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.security.plugins.JBossAuthorizationManager.authorize(JBossAuthorizationManager.java:125)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.security.plugins.javaee.WebAuthorizationHelper.hasUserDataPermission(WebAuthorizationHelper.java:201)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.web.tomcat.security.JBossWebRealm.hasUserDataPermission(JBossWebRealm.java:643)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:461)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:905)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:592)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2036)<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> at java.lang.Thread.run(Thread.java:662)<o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p style="min-height: 8pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333"> <o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">Thanks,<o:p></o:p></span></p>
<p><span style="font-size:9.0pt;font-family:"Arial","sans-serif";
color:#333333">Amol<o:p></o:p></span></p>
</div>
<div style="margin-top:15.0pt">
<p style="margin:0cm;margin-bottom:.0001pt;background:#F4F4F4"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Reply to this message by
<a href="https://community.jboss.org/message/734226#734226">going to Community</a><o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;background:#F4F4F4"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Start a new discussion in Performance Tuning at
<a href="https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2078">
Community</a><o:p></o:p></span></p>
</div>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br>
<hr>
<font face="Verdana" color="Gray" size="1"><br>
<br>
This e-mail from Practical Law Company (http://www.practicallaw.com) is subject to our terms of use (http://www.practicallaw.com/9-103-0884). Information about our companies: Practical Law Company Limited. Registered in England and Wales. Registered Number:
02889191. Registered Office: 19 Hatfields, London SE1 8DJ. Telephone: +44(0)20 7202 1200.<br>
Practical Law Company, Inc. Incorporated in Delaware, USA. Corporate Office: 747 Third Avenue, 36th Floor, New York, NY 10017. Telephone: +1(646) 562-3400.<br>
</font>
</body>
</html>