<div dir="ltr">Thanks Jonas!<div><br></div><div>I'll add this to the team's triage and see if we can reproduce it and, hopefully, find a solution.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 4, 2020 at 5:38 AM Jonas Zeiger <<a href="mailto:jonas.zeiger@talpidae.net">jonas.zeiger@talpidae.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">If this issue is similar to the one we encountered on our machines, this <br>
helps to reproduce it (HTTP2, TLSv1.3 is optional):<br>
<br>
<a href="https://gist.github.com/lyind/ae076548cafb2cb0b46a0819b749d6f4#file-curl-ssltest-sh" rel="noreferrer" target="_blank">https://gist.github.com/lyind/ae076548cafb2cb0b46a0819b749d6f4#file-curl-ssltest-sh</a><br>
<br>
The critical steps are:<br>
<br>
1. Attach debugger to your server process (normally via remote <br>
debugging)<br>
<br>
2. On a client at least one layer 3 network segment away (behind one or <br>
more IP routers):<br>
start >= 6 requests in a loop (increase the chance of "vulnerable" <br>
TCP connections at step 3)<br>
<br>
3. Sever the connection WITHOUT one of the two (client, server) <br>
operating system kernels being able to update TCP state:<br>
pull LAN cable of the client or shutdown router at the client-side<br>
<br>
4. Repeat 2) and 3) until:<br>
1. $ netstat -tnp | grep java<br>
-> some connections in CLOSE_WAIT (then look how the server <br>
handles these, play with operating system tunables)<br>
2. (optional) XNIO threads stuck at 100% CPU<br>
3. (optional) unexpected exceptions occur<br>
<br>
<br>
In theory one could also use some firewall or network emulator trickery <br>
to cut the connections but that would be harder to get right and still <br>
be quite complex for a unit or integration tests. Sounds more like a <br>
case for the physical network test lab.<br>
<br>
On 2020-03-04 06:24, Flavia Rainone wrote:<br>
> Can someone provide a reproducer for this error?<br>
> <br>
> As for the old version of XNIO, it will be upgraded in Undertow<br>
> 2.1.0.Final.<br>
> <br>
> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <<a href="mailto:sdouglas@redhat.com" target="_blank">sdouglas@redhat.com</a>><br>
> wrote:<br>
> <br>
>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow<br>
>> the connection is not being closed if the client hangs up abruptly.<br>
>> I had a quick look at the code though and I think it looks ok, but<br>
>> maybe some more investigation is needed.<br>
>> <br>
>> Stuart<br>
>> <br>
>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar<br>
>> <<a href="mailto:nishantkumar35@gmail.com" target="_blank">nishantkumar35@gmail.com</a>> wrote:<br>
>> <br>
>> Yes, i have no control on client side. I am using HTTP2. I have<br>
>> tried increasing open file limit to 400k but that consumes all<br>
>> memory and system hangs. I will probably try to put a nginx in front<br>
>> of Undertow and test.<br>
>> <br>
>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)<br>
>> <br>
>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <<a href="mailto:david.lloyd@redhat.com" target="_blank">david.lloyd@redhat.com</a>><br>
>> wrote:<br>
>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg<br>
>> <<a href="mailto:stan.rosenberg@acm.org" target="_blank">stan.rosenberg@acm.org</a>> wrote:<br>
>>> <br>
>>> Stuck in CLOSE_WAIT is a symptom of the client-side not properly<br>
>> shutting down [1].<br>
>> <br>
>> I would partially disagree. In the article you linked: "It all<br>
>> starts<br>
>> with a listening application that leaks sockets and forgets to call<br>
>> close(). This kind of bug does happen in complex applications."<br>
>> This<br>
>> seems to be essentially what's happening here: the server isn't<br>
>> completing the connection (for some reason), stranding the socket in<br>
>> `CLOSE_WAIT`.<br>
>> <br>
>> We can't assume that the client is abandoning the connection after<br>
>> `FIN_WAIT2` (the titular RFC violation); if the server stays in<br>
>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`<br>
>> forever, the resolving condition still needs to be that the server<br>
>> shuts down its side of the connection.<br>
>> <br>
>> This diagram is a useful visual aid, mapping TCP states to the XNIO<br>
>> API:<br>
>> <br>
> <a href="https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a1c0a0046e1/image.png" rel="noreferrer" target="_blank">https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a1c0a0046e1/image.png</a><br>
>> <br>
>> --<br>
>> - DML<br>
> _______________________________________________<br>
> undertow-dev mailing list<br>
> <a href="mailto:undertow-dev@lists.jboss.org" target="_blank">undertow-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/undertow-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/undertow-dev</a><br>
> <br>
> --<br>
> <br>
> Flavia Rainone<br>
> <br>
> Principal Software Engineer<br>
> <br>
> Red Hat [1]<br>
> <br>
> <a href="mailto:frainone@redhat.com" target="_blank">frainone@redhat.com</a><br>
> <br>
> [1]<br>
> <br>
> <br>
> <br>
> Links:<br>
> ------<br>
> [1] <a href="https://www.redhat.com" rel="noreferrer" target="_blank">https://www.redhat.com</a><br>
> _______________________________________________<br>
> undertow-dev mailing list<br>
> <a href="mailto:undertow-dev@lists.jboss.org" target="_blank">undertow-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/undertow-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/undertow-dev</a><br>
<br>
_______________________________________________<br>
undertow-dev mailing list<br>
<a href="mailto:undertow-dev@lists.jboss.org" target="_blank">undertow-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/undertow-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailman/listinfo/undertow-dev</a><br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>
<p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;font-family:RedHatText,sans-serif">
<span>Flavia</span> <span>Rainone</span><span style="color:rgb(170,170,170);margin:0px"></span>
</p>
<p style="font-size:12px;margin:0px;font-family:RedHatText,sans-serif">
<span>Principal Software Engineer</span>
</p>
<p style="margin:0px 0px 4px;font-size:12px;font-family:RedHatText,sans-serif">
<a style="color:rgb(0,136,206);margin:0px" href="https://www.redhat.com" target="_blank">Red Hat </a>
</p>
<div style="margin-bottom:4px">
</div>
<p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif">
<span style="margin:0px;padding:0px"><a style="color:rgb(0,0,0);margin:0px" href="mailto:frainone@redhat.com" target="_blank">frainone@redhat.com</a> </span>
</p>
<div style="margin-top:12px">
<table border="0">
<tbody><tr>
<td width="100px"><a href="https://www.redhat.com" target="_blank"> <img src="https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png" width="90" height="auto"></a> </td>
</tr>
</tbody></table>
</div></div></div></div></div></div></div></div>