<div dir="ltr">I tried disabling HTTP/2 but still too many CLOSE_WAIT connections. I tried putting Nginx in front of the Java server but seems the same issue with it. It seems Nginx is also creating the same number of connections with Java.<br><br><font face="monospace">server = Undertow.builder()<br> .addHttpListener(SERVER_LISTEN_PORT, SERVER_HOST)<br> .addHttpsListener(SERVER_SSL_LISTEN_PORT, SERVER_HOST, sslContext)<br> .setWorkerThreads(WORKER_THREAD)<br> <b><font color="#ff0000">.setServerOption(UndertowOptions.ENABLE_HTTP2, false)</font></b><br> .setServerOption(UndertowOptions.IDLE_TIMEOUT, 150000) // 150s<br> .setServerOption(UndertowOptions.NO_REQUEST_TIMEOUT, 150000) // 150s<br> .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024 * 20) // 20000 sessions<br> .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 1500) // 150s<br> .setIoThreads(IO_THREAD)<br> .setWorkerOption(org.xnio.Options.TCP_NODELAY, true)<br> .setSocketOption(org.xnio.Options.TCP_NODELAY, true)<br> .setSocketOption(org.xnio.Options.KEEP_ALIVE, true)<br> .setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)<br> .setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 100000)<br> .setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 100000)<br> .setHandler(Handlers.routing().post("/", new RequestHandler(appContext)))<br> .build();</font><br><br><i># netstat -nalp | grep -E ":80 |:443 " | awk '{split($4,a,":");print a[2] " " $6}'| sort | uniq -c<br> 85918 443 CLOSE_WAIT<br> 10279 443 ESTABLISHED<br> 67 443 LAST_ACK<br> 152 443 SYN_RECV<br> 505 443 TIME_WAIT<br> 31151 80 CLOSE_WAIT<br> 3747 80 ESTABLISHED<br> 108 80 LAST_ACK<br> 146 80 SYN_RECV<br> 2 LISTEN</i><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 3, 2020 at 5:17 AM Stuart Douglas <<a href="mailto:sdouglas@redhat.com">sdouglas@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow the connection is not being closed if the client hangs up abruptly. I had a quick look at the code though and I think it looks ok, but maybe some more investigation is needed.<div><br></div><div>Stuart</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <<a href="mailto:nishantkumar35@gmail.com" target="_blank">nishantkumar35@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Yes, i have no control on client side. I am using HTTP2. I have tried increasing open file limit to 400k but that consumes all memory and system hangs. I will probably try to put a nginx in front of Undertow and test.<div dir="auto"><br></div><div dir="auto"><span style="color:rgb(36,41,46);font-family:sfmono-regular,consolas,"liberation mono",menlo,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">setServerOption(</span><span style="color:rgb(36,41,46);font-family:sfmono-regular,consolas,"liberation mono",menlo,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">UndertowOptions</span><span style="color:rgb(0,92,197);font-family:sfmono-regular,consolas,"liberation mono",menlo,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)"><span style="color:rgb(215,58,73)">.</span>ENABLE_HTTP2</span><span style="color:rgb(36,41,46);font-family:sfmono-regular,consolas,"liberation mono",menlo,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">, </span><span style="color:rgb(0,92,197);font-family:sfmono-regular,consolas,"liberation mono",menlo,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">true</span><span style="color:rgb(36,41,46);font-family:sfmono-regular,consolas,"liberation mono",menlo,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255)">)</span><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 2, 2020, 7:48 PM David Lloyd <<a href="mailto:david.lloyd@redhat.com" rel="noreferrer noreferrer" target="_blank">david.lloyd@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <<a href="mailto:stan.rosenberg@acm.org" rel="noreferrer noreferrer noreferrer" target="_blank">stan.rosenberg@acm.org</a>> wrote:<br>
><br>
> Stuck in CLOSE_WAIT is a symptom of the client-side not properly shutting down [1].<br>
<br>
I would partially disagree. In the article you linked: "It all starts<br>
with a listening application that leaks sockets and forgets to call<br>
close(). This kind of bug does happen in complex applications." This<br>
seems to be essentially what's happening here: the server isn't<br>
completing the connection (for some reason), stranding the socket in<br>
`CLOSE_WAIT`.<br>
<br>
We can't assume that the client is abandoning the connection after<br>
`FIN_WAIT2` (the titular RFC violation); if the server stays in<br>
`CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`<br>
forever, the resolving condition still needs to be that the server<br>
shuts down its side of the connection.<br>
<br>
This diagram is a useful visual aid, mapping TCP states to the XNIO<br>
API: <a href="https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a1c0a0046e1/image.png" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a1c0a0046e1/image.png</a><br>
<br>
-- <br>
- DML<br>
<br>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Nishant Kumar<br>Bangalore, India<br>Mob: +91 80088 42030<br>Email: <a href="mailto:nishantkumar35@gmail.com" target="_blank">nishantkumar35@gmail.com</a></div></div>