[undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

[Stan Rosenberg]

On Fri, Feb 28, 2020 at 11:08 PM Nishant Kumar <nishantkumar35 at gmail.com>
wrote:

> I am using undertow-core (2.0.29.Final) with Java 11. Every once in a
> while (1-2 days) I got  Too many open files error on my high load server
> (30k JSON Request/sec) and then it stops responding. I have noticed that it
> happens during full GC (but not during all full GC). Could you please help
> me to fix this?
>

The "too many open files" error is invariably related to exceeding the max.
open file descriptors as enforced by the OS.  Note that if you're using
systemd to start undertow, then you have to specify LimitNOFILE (it
ignores /etc/security/limits*); see [1] for more details.  If you're not
running systemd, then I'd double-check the file descriptor limit for the
user running the undertow process.  As to why this happens once in a while,
my best guess is either or both (a) there is a sudden, short-lived spike in
new connections, (b) many connections are kept alive for over 24 hours.

We've had similar issues while running undertow in high-throughput,
low-latency environment (in front of GLB).  Even with the unreasonably high
file descriptor limit (LimitNOFILE=795588), we've seen traffic bursts that
would bring down undertow by triggering (linux) OOM killer (with that many
connections, non-heap memory is exhausted).

[1]
https://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7


Best,

stan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20200229/888f7a41/attachment.html