[undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

[Richard Opalka]

Hi Nishant,

   thanks for the update - that is good news!

Rio

On Sun, Mar 15, 2020 at 6:00 AM Nishant Kumar <nishantkumar35 at gmail.com>
wrote:

> I have tried XNIO 3.7.7-Final  with jboss.threads 2.3.3.Final in Undertow 2.0.30.Final
> and it looks good so far. I am running it in production for the last 2 days
> and no issues so far :). I am monitoring it and will report if find
> anything.
>
> The reason I changed jboss thread version is that I observed a high CPU
> load in XNIO 3.7.7.Final comparing XNIO 3.3.8.Final which I reported in the
> last email. However, I have not done any extensive tests for it.
>
> On Thu, Mar 12, 2020 at 1:37 PM Nishant Kumar <nishantkumar35 at gmail.com>
> wrote:
>
>> I have also observed that the CPU load increases significantly on
>> 3.7.7-Final (load average 50 on 48 core machine) on moderate load but it's
>> very low (load average 20) on 3.3.8-Final on the same QPS (20k Req/sec) and
>> it (3.7.7-Final) starts refusing connection on high load.
>>
>> On Thu, Mar 12, 2020 at 7:20 AM Nishant Kumar <nishantkumar35 at gmail.com>
>> wrote:
>>
>>> I didn't get time to write standalone program to reproduce it but with
>>> new version 3.7.7-Final, i am not able to survive for more than an hour.
>>> With the older default xnio version of Undertow (xnio 3.3.8-Final) i can
>>> run server for few days until it stops working with error i mentioned in
>>> first email.
>>>
>>> Just FYI, i am using OpenJDK 11 with OpenJ9 JVM.
>>>
>>> On Wed, Mar 4, 2020, 10:54 AM Flavia Rainone <frainone at redhat.com>
>>> wrote:
>>>
>>>> Can someone provide a reproducer for this error?
>>>>
>>>> As for the old version of XNIO, it will be upgraded in Undertow
>>>> 2.1.0.Final.
>>>>
>>>> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas at redhat.com>
>>>> wrote:
>>>>
>>>>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
>>>>> the connection is not being closed if the client hangs up abruptly. I had a
>>>>> quick look at the code though and I think it looks ok, but maybe some more
>>>>> investigation is needed.
>>>>>
>>>>> Stuart
>>>>>
>>>>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, i have no control on client side. I am using HTTP2. I have tried
>>>>>> increasing open file limit to 400k but that consumes all memory and system
>>>>>> hangs. I will probably try to put a nginx in front of Undertow and test.
>>>>>>
>>>>>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>>>>>
>>>>>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <
>>>>>>> stan.rosenberg at acm.org> wrote:
>>>>>>> >
>>>>>>> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly
>>>>>>> shutting down [1].
>>>>>>>
>>>>>>> I would partially disagree.  In the article you linked: "It all
>>>>>>> starts
>>>>>>> with a listening application that leaks sockets and forgets to call
>>>>>>> close(). This kind of bug does happen in complex applications."  This
>>>>>>> seems to be essentially what's happening here: the server isn't
>>>>>>> completing the connection (for some reason), stranding the socket in
>>>>>>> `CLOSE_WAIT`.
>>>>>>>
>>>>>>> We can't assume that the client is abandoning the connection after
>>>>>>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>>>>>>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>>>>>>> forever, the resolving condition still needs to be that the server
>>>>>>> shuts down its side of the connection.
>>>>>>>
>>>>>>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>>>>>>> API:
>>>>>>> https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a1c0a0046e1/image.png
>>>>>>>
>>>>>>> --
>>>>>>> - DML
>>>>>>>
>>>>>>> _______________________________________________
>>>>> undertow-dev mailing list
>>>>> undertow-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Flavia Rainone
>>>>
>>>> Principal Software Engineer
>>>>
>>>> Red Hat <https://www.redhat.com>
>>>>
>>>> frainone at redhat.com
>>>> <https://www.redhat.com>
>>>>
>>>
>>
>> --
>> Nishant Kumar
>> Bangalore, India
>> Mob: +91 80088 42030
>> Email: nishantkumar35 at gmail.com
>>
>
>
> --
> Nishant Kumar
> Bangalore, India
> Mob: +91 80088 42030
> Email: nishantkumar35 at gmail.com
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev



-- 
Richard Opalka
Principal Software Engineer
Red Hat JBoss Middleware
Mobile: +420 731 186 942
E-mail: ropalka at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20200318/fcd90c20/attachment.html