Hi,<div><br></div><div>I'm prototyping an HTTP service in Netty which accepts requests, and then proxies that request to several web applications and pulls their responses together into a single HTTP response containing an HTML payload.</div>
<div><br></div><div>The server channel factory uses the following thread pools:</div><div><br></div><div><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}
span.s1 {color: #bc319c}
span.s2 {color: #2834cf}
</style>
<p class="p1"><span class="s1">int</span> numCores = <span class="s2">4</span>*Runtime.getRuntime().availableProcessors();</p></div><div><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}
span.s1 {color: #bc319c}
span.s2 {color: #2834cf}
</style>
<p class="p1">Executor bossExecutor = <span class="s1">new</span> ThreadPoolExecutor(<span class="s2">0</span>, Integer.MAX_VALUE,</p>
<p class="p1"> <span class="s2">60L</span>, TimeUnit.MINUTES,</p>
<p class="p1"> <span class="s1">new</span> SynchronousQueue<Runnable>());</p><p class="p1"><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}
</style>
</p><p class="p1">Executor workerExecutor = Executors.newFixedThreadPool(numCores);</p><p></p></div><div><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}
span.s1 {color: #bc319c}
</style>
<p class="p1">bootstrap.setFactory(<span class="s1">new</span> NioServerSocketChannelFactory(bossExecutor, workerExecutor, numCores));</p><p class="p1"><br></p><p class="p1"><meta charset="utf-8"><span class="Apple-style-span" style="font-family: arial; font-size: small; ">The server pipeline contains an ExecutionHandler which has an OrderedMemoryAwareThreadPoolExecutor (the thread factory only changes the thread name):</span></p>
<p class="p1"><span class="Apple-style-span" style="font-family: arial; font-size: small; "><br></span></p><p class="p1"><span class="Apple-style-span" style="font-family: arial; font-size: small; ">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}
span.s1 {color: #bc319c}
span.s2 {color: #2834cf}
span.s3 {color: #d32d26}
</style>
</span></p><p class="p1"><span class="s1">final</span> ExecutionHandler embedRequestor = <span class="s1">new</span> ExecutionHandler(<span class="s1">new</span> OrderedMemoryAwareThreadPoolExecutor(numCores, </p>
<p class="p1"> memory, // this is 500 Kb</p>
<p class="p1"> memory,</p>
<p class="p1"> <span class="s2">60L</span>, TimeUnit.MINUTES,</p>
<p class="p1"> <span class="s1">new </span>FSThreadFactory(<span class="s3">"embed-requestor"</span>)));</p><p>
</p></div><div><br></div><div>The proxy HTTP clients all share a single client channel factory, which uses a cached thread pool whose threads have a long timeout (so that they're re-used):</div><div><br></div><div><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo}
span.s1 {color: #bc319c}
span.s2 {color: #2834cf}
</style>
<p class="p1"> <span class="s1">private</span> <span class="s1">final</span> <span class="s1">static</span> NioClientSocketChannelFactory clientChannelFactory =</p>
<p class="p1"> <span class="s1">new</span> NioClientSocketChannelFactory(<span class="s1">new</span> ThreadPoolExecutor(<span class="s2">0</span>, Integer.MAX_VALUE,</p>
<p class="p1"> <span class="s2">60L</span>, TimeUnit.MINUTES,</p>
<p class="p1"> <span class="s1">new</span> SynchronousQueue<Runnable>()), </p>
<p class="p1"> <span class="s1">new</span> ThreadPoolExecutor(<span class="s2">0</span>, Integer.MAX_VALUE,</p>
<p class="p1"> <span class="s2">60L</span>, TimeUnit.MINUTES,</p>
<p class="p1"> <span class="s1">new</span> SynchronousQueue<Runnable>()));</p><p class="p1"><br></p><p class="p1"><meta charset="utf-8"><font class="Apple-style-span" face="arial" size="2">When I run this through apache bench, with concurrency levels 1 to 100 in increments of 5, here's the performance (at 50th, 90th, 99th percentiles as well as request per second). This is with the following GC settings:</font></p>
<p class="p1"><font class="Apple-style-span" face="arial" size="2">java -server -Xms1024m -Xmx1024m -XX:MaxNewSize=768m -XX:NewSize=768m -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC (note that Old Gen never gets full enough to actually force CMS)</font></p>
<p class="p1"><font class="Apple-style-span" face="arial" size="2"><br></font></p><p class="p1"><font class="Apple-style-span" face="arial">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="ProgId" content="Excel.Sheet">
<meta name="Generator" content="Microsoft Excel 14">
<link id="Main-File" rel="Main-File" href="file://localhost/Users/bhaidri/Library/Caches/TemporaryItems/msoclip/0/clip.htm">
<link rel="File-List" href="file://localhost/Users/bhaidri/Library/Caches/TemporaryItems/msoclip/0/clip_filelist.xml">
<style>
<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:1.0in .75in 1.0in .75in;
mso-header-margin:.5in;
mso-footer-margin:.5in;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:12.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:Calibri, sans-serif;
mso-font-charset:0;
mso-number-format:General;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
.xl63
{font-size:10.0pt;
font-family:Calibri;
mso-generic-font-family:auto;
mso-font-charset:0;
text-align:right;}
.xl64
{font-size:10.0pt;
font-family:Calibri;
mso-generic-font-family:auto;
mso-font-charset:0;
mso-number-format:0%;}
.xl65
{font-size:10.0pt;
font-family:Calibri;
mso-generic-font-family:auto;
mso-font-charset:0;}
.xl66
{color:blue;
font-size:10.0pt;
font-weight:700;
font-family:Calibri;
mso-generic-font-family:auto;
mso-font-charset:0;}
.xl67
{color:green;
font-size:10.0pt;
font-weight:700;
font-family:Calibri;
mso-generic-font-family:auto;
mso-font-charset:0;}
-->
</style>
<table border="0" cellpadding="0" cellspacing="0" width="299" style="font-size: medium; border-collapse: collapse; width: 299pt; ">
<colgroup><col width="69" style="mso-width-source:userset;mso-width-alt:2944;width:69pt">
<col width="49" style="mso-width-source:userset;mso-width-alt:2090;width:49pt">
<col width="55" style="mso-width-source:userset;mso-width-alt:2346;width:55pt">
<col width="61" style="mso-width-source:userset;mso-width-alt:2602;width:61pt">
<col width="65" style="width:65pt">
</colgroup><tbody><tr height="15" style="height:15.0pt">
<td height="15" class="xl63" width="69" style="height:15.0pt;width:69pt">Concurrency</td>
<td class="xl64" align="right" width="49" style="width:49pt">50%</td>
<td class="xl64" align="right" width="55" style="width:55pt">90%</td>
<td class="xl64" align="right" width="61" style="width:61pt">99%</td>
<td class="xl63" width="65" style="width:65pt">RPS</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt"><a name="RANGE!A27:D47">1</a></td>
<td class="xl65" align="right">70.928</td>
<td class="xl65" align="right">84.199</td>
<td class="xl65" align="right">93.12</td>
<td class="xl65" align="right">13.7</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">5</td>
<td class="xl65" align="right">74.101</td>
<td class="xl65" align="right">99.795</td>
<td class="xl65" align="right">141.391</td>
<td class="xl65" align="right">62.91</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl66" align="right" style="height:15.0pt">10</td>
<td class="xl66" align="right">88.744</td>
<td class="xl66" align="right">146.416</td>
<td class="xl66" align="right">232.333</td>
<td class="xl66" align="right">99.22</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">15</td>
<td class="xl65" align="right">108.265</td>
<td class="xl65" align="right">188.855</td>
<td class="xl65" align="right">305.268</td>
<td class="xl65" align="right">121.44</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">20</td>
<td class="xl65" align="right">130.508</td>
<td class="xl65" align="right">231.05</td>
<td class="xl65" align="right">438.312</td>
<td class="xl65" align="right">134.31</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl67" align="right" style="height:15.0pt">25</td>
<td class="xl67" align="right">158.417</td>
<td class="xl67" align="right">297.67</td>
<td class="xl67" align="right">666.237</td>
<td class="xl67" align="right">132.42</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">30</td>
<td class="xl65" align="right">170.459</td>
<td class="xl65" align="right">349.15</td>
<td class="xl65" align="right">2541.908</td>
<td class="xl65" align="right">110.19</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">35</td>
<td class="xl65" align="right">197.119</td>
<td class="xl65" align="right">411.041</td>
<td class="xl65" align="right">4129.77</td>
<td class="xl65" align="right">110.64</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">40</td>
<td class="xl65" align="right">237.777</td>
<td class="xl65" align="right">493.632</td>
<td class="xl65" align="right">3091.222</td>
<td class="xl65" align="right">110.69</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">45</td>
<td class="xl65" align="right">268.308</td>
<td class="xl65" align="right">475.407</td>
<td class="xl65" align="right">8118.155</td>
<td class="xl65" align="right">98.13</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">50</td>
<td class="xl65" align="right">306.447</td>
<td class="xl65" align="right">553.581</td>
<td class="xl65" align="right">6455.245</td>
<td class="xl65" align="right">118.01</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">55</td>
<td class="xl65" align="right">340.214</td>
<td class="xl65" align="right">662.951</td>
<td class="xl65" align="right">3274.517</td>
<td class="xl65" align="right">110.55</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">60</td>
<td class="xl65" align="right">363.786</td>
<td class="xl65" align="right">779.651</td>
<td class="xl65" align="right">5793.395</td>
<td class="xl65" align="right">96.38</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">65</td>
<td class="xl65" align="right">404.513</td>
<td class="xl65" align="right">659.296</td>
<td class="xl65" align="right">3803.272</td>
<td class="xl65" align="right">119.49</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">70</td>
<td class="xl65" align="right">439.377</td>
<td class="xl65" align="right">800.518</td>
<td class="xl65" align="right">7408.627</td>
<td class="xl65" align="right">105.6</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">75</td>
<td class="xl65" align="right">462.924</td>
<td class="xl65" align="right">757.701</td>
<td class="xl65" align="right">2264.736</td>
<td class="xl65" align="right">137.37</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">80</td>
<td class="xl65" align="right">507.593</td>
<td class="xl65" align="right">1117.655</td>
<td class="xl65" align="right">11667.996</td>
<td class="xl65" align="right">82.76</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">85</td>
<td class="xl65" align="right">521.067</td>
<td class="xl65" align="right">968.124</td>
<td class="xl65" align="right">6895.998</td>
<td class="xl65" align="right">111.34</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">90</td>
<td class="xl65" align="right">545.382</td>
<td class="xl65" align="right">1713.862</td>
<td class="xl65" align="right">9650.341</td>
<td class="xl65" align="right">95.52</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">95</td>
<td class="xl65" align="right">613.355</td>
<td class="xl65" align="right">1016.572</td>
<td class="xl65" align="right">2502.987</td>
<td class="xl65" align="right">119.9</td>
</tr>
<tr height="15" style="height:15.0pt">
<td height="15" class="xl65" align="right" style="height:15.0pt">100</td>
<td class="xl65" align="right">636.438</td>
<td class="xl65" align="right">979.166</td>
<td class="xl65" align="right">5237.865</td>
<td class="xl65" align="right">129.07</td>
</tr>
</tbody></table><br><font class="Apple-style-span" size="2">My question is: why is Netty's performance degrading so badly at higher concurrencies? Why is the latency so unstable? Why does the CPU get pegged at higher concurrencies? I noticed when I profiled this code through JVisualVM that 1/3 of the CPU time is spent in </font></font>LinkedTransferQueue.awaitMatch.</p>
<p class="p1"><font class="Apple-style-span"><br></font></p><p class="p1"><font class="Apple-style-span">Any suggestions people might have would be greatly appreciated. We've implemented the same system in Node and as C++ module using ATS and are seeing far better performance numbers for both.<br>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta http-equiv="Content-Style-Type" content="text/css">
<title></title>
<meta name="Generator" content="Cocoa HTML Writer">
<meta name="CocoaVersion" content="1038.35">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Calibri}
</style>
<br></font></p><p class="p1"><font class="Apple-style-span">Thanks,</font></p><p class="p1"><font class="Apple-style-span">Baq</font></p><p class="p1"></p><p class="p1"></p></div>