[undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

[Nishant Kumar]

Thanks for the reply. I am running it under supervisord and i have updated
open file limit in supervisord config. The problem seems to be same as what
@Carter has mentioned. It happens mostly during sudden traffic spike and
then sudden increase (~30k-300k) of TIME_WAIT socket. I will try to build
Undertow with latest XNIO because latest Undertow release is still using
old version of XNIO. Any suggestions on the build?

On Sat, Feb 29, 2020, 11:34 PM Stan Rosenberg <stan.rosenberg at acm.org>
wrote:

> On Fri, Feb 28, 2020 at 11:08 PM Nishant Kumar <nishantkumar35 at gmail.com>
> wrote:
>
>> I am using undertow-core (2.0.29.Final) with Java 11. Every once in a
>> while (1-2 days) I got  Too many open files error on my high load server
>> (30k JSON Request/sec) and then it stops responding. I have noticed that it
>> happens during full GC (but not during all full GC). Could you please help
>> me to fix this?
>>
>
> The "too many open files" error is invariably related to exceeding the
> max. open file descriptors as enforced by the OS.  Note that if you're
> using systemd to start undertow, then you have to specify LimitNOFILE (it
> ignores /etc/security/limits*); see [1] for more details.  If you're not
> running systemd, then I'd double-check the file descriptor limit for the
> user running the undertow process.  As to why this happens once in a while,
> my best guess is either or both (a) there is a sudden, short-lived spike in
> new connections, (b) many connections are kept alive for over 24 hours.
>
> We've had similar issues while running undertow in high-throughput,
> low-latency environment (in front of GLB).  Even with the unreasonably high
> file descriptor limit (LimitNOFILE=795588), we've seen traffic bursts that
> would bring down undertow by triggering (linux) OOM killer (with that many
> connections, non-heap memory is exhausted).
>
> [1]
> https://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7
>
>
> Best,
>
> stan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20200301/5eb07a89/attachment.html