[undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

[David Lloyd]

On Tue, Mar 3, 2020 at 4:05 AM Jonas Zeiger <jonas.zeiger at talpidae.net> wrote:
> [...]
> Thus a workaround is best applied on the layer actually handling the TCP
> connection state.

This is a really great point.  The fact is that in most cases, we
really have no way at the Java level to cope with a server request
that is taking too long due to overload in conjunction with an
impatient client.  Whether it's a thread pool queue that is overlong,
or the list of ready sockets takes too long to iterate, or any other
case, there's really no way we can know immediately when the client
has decided to give up on an HTTP request or connection.

> On Linux I successfully use the following sysctl.conf entries to keep
> the CLOSE_WAIT connections in check and avoid netfilter issues:
>
> # keep the number of TCP connections in CLOSE_WAIT low
> # by killing CLOSE_WAIT sockets after some time (will look like RESET to
> the server processes)
> net.ipv4.tcp_fin_timeout = 15
> net.ipv4.tcp_ecn = 1
> net.ipv4.tcp_synack_retries = 2
> net.ipv4.tcp_tw_recycle = 1
> net.ipv4.tcp_tw_reuse = 1
> # tune TCP keepalive to be a little more practical (2h -> 5 minutes
> timeout, kill connection after 2 failed probes)
> net.ipv4.tcp_keepalive_intvl = 2
> net.ipv4.tcp_keepalive_probes = 2
> net.ipv4.tcp_keepalive_time = 300
>
> # more contrack table entries (else we may get some "upstream connection
> timed out"
> # only needed when using netfilter connection tracking (for example NAT)
> # net.nf_conntrack_max = 6556000

This seems like a very pragmatic solution.  I'm not sure this would
lower the actual number of file descriptors but it would help with the
situation where clients are transitioning out of FIN_WAIT_2 too early.


-- 
- DML