Re: [undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

I have also observed that the CPU load increases significantly on 3.7.7-Final (load average 50 on 48 core machine) on moderate load but it's very low (load average 20) on 3.3.8-Final on the same QPS (20k Req/sec) and it (3.7.7-Final) starts refusing connection on high load. On Thu, Mar 12, 2020 at 7:20 AM Nishant Kumar <nishantkumar35(a)gmail.com&gt; wrote: > I didn't get time to write standalone program to reproduce it but with > new version 3.7.7-Final, i am not able to survive for more than an hour. > With the older default xnio version of Undertow (xnio 3.3.8-Final) i can > run server for few days until it stops working with error i mentioned in > first email. > > Just FYI, i am using OpenJDK 11 with OpenJ9 JVM. > > On Wed, Mar 4, 2020, 10:54 AM Flavia Rainone <frainone(a)redhat.com&gt; wrote: > >> Can someone provide a reproducer for this error? >> >> As for the old version of XNIO, it will be upgraded in Undertow >> 2.1.0.Final. >> >> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com&gt; >> wrote: >> >>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow the >>> connection is not being closed if the client hangs up abruptly. I had a >>> quick look at the code though and I think it looks ok, but maybe some more >>> investigation is needed. >>> >>> Stuart >>> >>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com&gt; >>> wrote: >>> >>>> Yes, i have no control on client side. I am using HTTP2. I have tried >>>> increasing open file limit to 400k but that consumes all memory and system >>>> hangs. I will probably try to put a nginx in front of Undertow and test. >>>> >>>> setServerOption(UndertowOptions.ENABLE_HTTP2, true) >>>> >>>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com&gt; >>>> wrote: >>>> >>>>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <stan.rosenberg(a)acm.org&gt; >>>>> wrote: >>>>> > >>>>> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly >>>>> shutting down [1]. >>>>> >>>>> I would partially disagree. In the article you linked: "It all starts >>>>> with a listening application that leaks sockets and forgets to call >>>>> close(). This kind of bug does happen in complex applications." This >>>>> seems to be essentially what's happening here: the server isn't >>>>> completing the connection (for some reason), stranding the socket in >>>>> `CLOSE_WAIT`. >>>>> >>>>> We can't assume that the client is abandoning the connection after >>>>> `FIN_WAIT2` (the titular RFC violation); if the server stays in >>>>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2` >>>>> forever, the resolving condition still needs to be that the server >>>>> shuts down its side of the connection. >>>>> >>>>> This diagram is a useful visual aid, mapping TCP states to the XNIO >>>>> API: >>>>> https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a... >>>>> >>>>> -- >>>>> - DML >>>>> >>>>> _______________________________________________ >>> undertow-dev mailing list >>> undertow-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/undertow-dev >> >> >> >> -- >> >> Flavia Rainone >> >> Principal Software Engineer >> >> Red Hat <https://www.redhat.com> >> >> frainone(a)redhat.com >> <https://www.redhat.com> >> > -- Nishant Kumar Bangalore, India Mob: +91 80088 42030 Email: nishantkumar35(a)gmail.com