<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Century Gothic";
panose-1:2 11 5 2 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Century Gothic","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Century Gothic","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif";color:#1F497D">Filed
<a href="https://issues.jboss.org/browse/XNIO-280">https://issues.jboss.org/browse/XNIO-280</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-family:"Century Gothic","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Bradley, Qian
<br>
<b>Sent:</b> Wednesday, October 19, 2016 11:27 AM<br>
<b>To:</b> 'undertow-dev@lists.jboss.org'<br>
<b>Subject:</b> undertow/xnio did not dispatch request<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">We are replacing tomcat connector with undertow in a spring-boot setting, for servlet 3.0 based container. During load performance test, see a possible issue that some request are
not dispatched from undertow, the count is very small (~ 0.01%), but highly repeatable.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Example test Setting:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Jmeter 1 client (keep-alive traffic)
</span><span style="font-family:Wingdings">à</span><span style="font-family:"Century Gothic","sans-serif""> undertow, traffic is on the same client thread, it is a blocking client, so all in-coming request to undertow are sequential. Request sent non-stopping
(as fast as possible).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(1) verified that it is not due to network packet, we used tcpdump to trace the missed call that client-side sent the request, and server “ack” the request. But the response is never
sent. See below shows when a call is successful vs. unsuccessful”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(successful)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(client send request)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.325799 IP 10.24.48.104.39687 > 10.25.171.155.pcsync-https: Flags [P.], seq 20304084:20304121, ack 100946003, win 501, options [nop,nop,TS val 3490584463 ecr 72906421], length
37<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.325850 IP 10.24.48.104.39687 > 10.25.171.155.pcsync-https: Flags [P.], seq 20304121:20304366, ack 100946003, win 501, options [nop,nop,TS val 3490584463 ecr 72906421], length
245<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(from server)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.325995 IP 10.25.171.155.pcsync-https > 10.24.48.104.39687: Flags [.], ack 20304366, win 1636, options [nop,nop,TS val 72906421 ecr 3490584463], length 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.327110 IP 10.25.171.155.pcsync-https > 10.24.48.104.39687: Flags [P.], seq 100946003:100947405, ack 20304366, win 1636, options [nop,nop,TS val 72906422 ecr 3490584463], length
1402<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(unsuccessful)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(client send request)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.327440 IP 10.24.48.104.39687 > 10.25.171.155.pcsync-https: Flags [P.], seq 20304366:20304403, ack 100947405, win 501, options [nop,nop,TS val 3490584465 ecr 72906422], length
37<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.327485 IP 10.24.48.104.39687 > 10.25.171.155.pcsync-https: Flags [P.], seq 20304403:20304648, ack 100947405, win 501, options [nop,nop,TS val 3490584465 ecr 72906422], length
245<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(from server)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">05:01:05.327628 IP 10.25.171.155.pcsync-https > 10.24.48.104.39687: Flags [.], ack 20304648, win 1636, options [nop,nop,TS val 72906422 ecr 3490584465], length 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">[response from server never received]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(2) From application logging, here is example when request is successful vs unsuccessful<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(successful response)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black">2016-10-19 04:05:23,268 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Selected on
<a href="mailto:sun.nio.ch.EPollSelectorImpl@4129489f">sun.nio.ch.EPollSelectorImpl@4129489f</a><br>
2016-10-19 04:05:23,268 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Beginning select on
<a href="mailto:sun.nio.ch.EPollSelectorImpl@4129489f">sun.nio.ch.EPollSelectorImpl@4129489f</a> (with timeout)<br>
2016-10-19 04:05:23,269 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Selected on
<a href="mailto:sun.nio.ch.EPollSelectorImpl@4129489f">sun.nio.ch.EPollSelectorImpl@4129489f</a><br>
2016-10-19 04:05:23,269 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Selected key
<a href="mailto:sun.nio.ch.SelectionKeyImpl@4218d66e">sun.nio.ch.SelectionKeyImpl@4218d66e</a> for java.nio.channels.SocketChannel[connected local=/127.0.0.1:8443 remote=/127.0.0.1:51626]<br>
2016-10-19 04:05:23,269 TRACE [XNIO-APPLICATION I/O-8] org.xnio.listener Invoking listener
<a href="mailto:io.undertow.server.protocol.http.HttpReadListener@192666c4">io.undertow.server.protocol.http.HttpReadListener@192666c4</a> on channel
<a href="mailto:org.xnio.conduits.ConduitStreamSourceChannel@5d338d2a">org.xnio.conduits.ConduitStreamSourceChannel@5d338d2a</a><br>
2016-10-19 04:05:23,270 WARN [XNIO-APPLICATION task-14] com.paypal.raptor.samples.impl.SampleResourceImpl 156539<br>
2016-10-19 04:05:23,270 TRACE [XNIO-APPLICATION task-14] io.undertow.server.HttpServerExchange Starting to write response for HttpServerExchange{ GET /v1/qbd1serv/lnp/hello request {Connection=[keep-alive], Accept=[text/html, image/gif, image/jpeg, *; q=.2,
*/*; q=.2], User-Agent=[Java/1.7.0_45], Host=[qbd1serv59287.qa.paypal.com:8443]} response {Connection=[keep-alive], SERVER_INFO=[qbd1serv:ppaas_1_2.v1.qbd1serv.lnp.hello.GET&CalThreadId=114&TopLevelTxnStartTime=157dc9dfa45&Host=kncntusrstage13875&pid=2901],
Content-Type=[text/plain; charset=UTF-8], Content-Length=[1024], CORRELATION-ID=[4dcef4a4f99c3], Date=[Wed, 19 Oct 2016 11:05:23 GMT]}}<br>
2016-10-19 04:05:23,270 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio Running task <a href="mailto:io.undertow.protocols.ssl.SslConduit$1@326a9cf4">
io.undertow.protocols.ssl.SslConduit$1@326a9cf4</a><br>
2016-10-19 04:05:23,270 TRACE [XNIO-APPLICATION task-14] org.xnio.safe-close Closing resource
<a href="mailto:io.undertow.servlet.core.ServletBlockingHttpExchange@d9add7a">io.undertow.servlet.core.ServletBlockingHttpExchange@d9add7a</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:13.5pt;color:black">(unsuccessful )<br>
2016-10-19 04:05:23,270 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Beginning select on
<a href="mailto:sun.nio.ch.EPollSelectorImpl@4129489f">sun.nio.ch.EPollSelectorImpl@4129489f</a> (with timeout)<br>
2016-10-19 04:05:24,001 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Selected on
<a href="mailto:sun.nio.ch.EPollSelectorImpl@4129489f">sun.nio.ch.EPollSelectorImpl@4129489f</a><br>
2016-10-19 04:05:24,001 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio Running task <a href="mailto:io.undertow.util.DateUtils$2@d27f1e9">
io.undertow.util.DateUtils$2@d27f1e9</a><br>
2016-10-19 04:05:24,001 TRACE [XNIO-APPLICATION I/O-8] org.xnio.nio.selector Beginning select on
<a href="mailto:sun.nio.ch.EPollSelectorImpl@4129489f">sun.nio.ch.EPollSelectorImpl@4129489f</a> (with timeout)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Note that rest server actions are missing, from server log, I used a counter in a servlet filter, also server counter < client count
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Note: the timestamp on the above might not perfect match on client vs server time stamp respectively.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(3) I am on spring-boot 1.4.1, also tested using 1.3.x same behavior. I also overrides xnio 3.4 vs 3.3.x, also same behavior. Changed configured the following other than default ..<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(3.a) Undertow io- 8 threads, task worker 25 core threads, max 64 threads. Server-idle timeout 20 sec, server-no request timeout 20 sec.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">(3.b) If I use concurrent jmeter client, same behavior is also observed, the ratio did not worse, actually, it might be slightly better than just single test client.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Please share insights if there is any configuration from my side, or else.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Thanks much,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Century Gothic","sans-serif"">Qian<o:p></o:p></span></p>
</div>
</body>
</html>