On Tue, Mar 3, 2020 at 4:05 AM Jonas Zeiger <jonas.zeiger(a)talpidae.net> wrote:
[...]
Thus a workaround is best applied on the layer actually handling the TCP
connection state.
This is a really great point. The fact is that in most cases, we
really have no way at the Java level to cope with a server request
that is taking too long due to overload in conjunction with an
impatient client. Whether it's a thread pool queue that is overlong,
or the list of ready sockets takes too long to iterate, or any other
case, there's really no way we can know immediately when the client
has decided to give up on an HTTP request or connection.
On Linux I successfully use the following sysctl.conf entries to
keep
the CLOSE_WAIT connections in check and avoid netfilter issues:
# keep the number of TCP connections in CLOSE_WAIT low
# by killing CLOSE_WAIT sockets after some time (will look like RESET to
the server processes)
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
# tune TCP keepalive to be a little more practical (2h -> 5 minutes
timeout, kill connection after 2 failed probes)
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 300
# more contrack table entries (else we may get some "upstream connection
timed out"
# only needed when using netfilter connection tracking (for example NAT)
# net.nf_conntrack_max = 6556000
This seems like a very pragmatic solution. I'm not sure this would
lower the actual number of file descriptors but it would help with the
situation where clients are transitioning out of FIN_WAIT_2 too early.
--
- DML