Re: [undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)
9:29 p.m.
I agree that it's a load-balancing issue but we can't do much about it at
this moment.
I still see issues after using the latest XNIO (3.7.7) with Undertow. what
I have observed it that when there is a spike in request
and CONNECTION_HIGH_WATER is reached, the server stops accepting new
connection as expected and the client starts to close the connection
because of delay (we have strict low latency requirement < 100ms) and try
to create new connection again (which will also not be accepted) but server
has not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will
be high number of CLOSE_WAIT connections at this moment. The server is
considering CLOSE_WAIT + ESTABLISHED for CONNECTION_HIGH_WATER (my
understanding).
Is there a way that I can close all CLOSE_WAIT connection at this moment
so that connection counts drop under CONNECTION_HIGH_WATER and we start
responding to newly established connections? or any other suggestions? I
have tried removing CONNECTION_HIGH_WATER and relying on the FD limit but
that didn't work.
On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg <stan.rosenberg(a)gmail.com>
wrote:
On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar
<nishantkumar35(a)gmail.com>
wrote:
> Thanks for the reply. I am running it under supervisord and i have
> updated open file limit in supervisord config. The problem seems to be same
> as what @Carter has mentioned. It happens mostly during sudden traffic
> spike and then sudden increase (~30k-300k) of TIME_WAIT socket.
>
The changes in
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11...
are
likely to improve the exceptional case of exceeding the file descriptor
limit. However, if you're already setting the limit too high (e.g., in our
case it was 795588), then exceeding it is a symptom of not properly
load-balancing your traffic; with that many connections, you'd better have
a ton of free RAM available.
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
10:56 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
This sounds like a bug, when the client closes the connection it should
wake up the read listener, which will read -1 and then cleanly close the
socket.
Are the clients closing idle connections or connections processing a
request?
Stuart
On Mon, 2 Mar 2020 at 14:31, Nishant Kumar <nishantkumar35(a)gmail.com> wrote:
I agree that it's a load-balancing issue but we can't do much
about it at
this moment.
I still see issues after using the latest XNIO (3.7.7) with Undertow. what
I have observed it that when there is a spike in request
and CONNECTION_HIGH_WATER is reached, the server stops accepting new
connection as expected and the client starts to close the connection
because of delay (we have strict low latency requirement < 100ms) and try
to create new connection again (which will also not be accepted) but server
has not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will
be high number of CLOSE_WAIT connections at this moment. The server is
considering CLOSE_WAIT + ESTABLISHED for CONNECTION_HIGH_WATER (my
understanding).
Is there a way that I can close all CLOSE_WAIT connection at this moment
so that connection counts drop under CONNECTION_HIGH_WATER and we start
responding to newly established connections? or any other suggestions? I
have tried removing CONNECTION_HIGH_WATER and relying on the FD limit but
that didn't work.
On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg <stan.rosenberg(a)gmail.com>
wrote:
> On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar <nishantkumar35(a)gmail.com>
> wrote:
>
>> Thanks for the reply. I am running it under supervisord and i have
>> updated open file limit in supervisord config. The problem seems to be same
>> as what @Carter has mentioned. It happens mostly during sudden traffic
>> spike and then sudden increase (~30k-300k) of TIME_WAIT socket.
>>
>
> The changes in
> https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11...
are
> likely to improve the exceptional case of exceeding the file descriptor
> limit. However, if you're already setting the limit too high (e.g., in our
> case it was 795588), then exceeding it is a symptom of not properly
> load-balancing your traffic; with that many connections, you'd better have
> a ton of free RAM available.
>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
2:59 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Generally, clients also close the connection after a few thousand requests
other than normal fatal conditions. There might be other cases too but I am
not aware of it. They keep initiating new connections if we are not
responding within the threshold time frame. This is a server to server
communication system.
On Mon, Mar 2, 2020 at 10:26 AM Stuart Douglas <sdouglas(a)redhat.com> wrote:
This sounds like a bug, when the client closes the connection it
should
wake up the read listener, which will read -1 and then cleanly close the
socket.
Are the clients closing idle connections or connections processing a
request?
Stuart
On Mon, 2 Mar 2020 at 14:31, Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
> I agree that it's a load-balancing issue but we can't do much about it at
> this moment.
>
> I still see issues after using the latest XNIO (3.7.7) with Undertow.
> what I have observed it that when there is a spike in request
> and CONNECTION_HIGH_WATER is reached, the server stops accepting new
> connection as expected and the client starts to close the connection
> because of delay (we have strict low latency requirement < 100ms) and try
> to create new connection again (which will also not be accepted) but server
> has not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will
> be high number of CLOSE_WAIT connections at this moment. The server is
> considering CLOSE_WAIT + ESTABLISHED for CONNECTION_HIGH_WATER (my
> understanding).
>
> Is there a way that I can close all CLOSE_WAIT connection at this moment
> so that connection counts drop under CONNECTION_HIGH_WATER and we start
> responding to newly established connections? or any other suggestions? I
> have tried removing CONNECTION_HIGH_WATER and relying on the FD limit but
> that didn't work.
>
> On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg <stan.rosenberg(a)gmail.com>
> wrote:
>
>> On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar <nishantkumar35(a)gmail.com>
>> wrote:
>>
>>> Thanks for the reply. I am running it under supervisord and i have
>>> updated open file limit in supervisord config. The problem seems to be same
>>> as what @Carter has mentioned. It happens mostly during sudden traffic
>>> spike and then sudden increase (~30k-300k) of TIME_WAIT socket.
>>>
>>
>> The changes in
>>
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11... are
>> likely to improve the exceptional case of exceeding the file descriptor
>> limit. However, if you're already setting the limit too high (e.g., in our
>> case it was 795588), then exceeding it is a symptom of not properly
>> load-balancing your traffic; with that many connections, you'd better have
>> a ton of free RAM available.
>>
>
>
> --
> Nishant Kumar
> Bangalore, India
> Mob: +91 80088 42030
> Email: nishantkumar35(a)gmail.com
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
6:16 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
have you check the logs again?
Your log had these entries:
*/var/log/messages :*
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
What is the output of
|sysctl -a | grep conntrack | grep timeout ||Please read:
|https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet
||
On 2-3-2020 09:59, Nishant Kumar wrote:
Generally, clients also close the connection after a few thousand
requests other than normal fatal conditions. There might be other
cases too but I am not aware of it. They keep initiating new
connections if we are not responding within the threshold time frame.
This is a server to server communication system.
On Mon, Mar 2, 2020 at 10:26 AM Stuart Douglas <sdouglas(a)redhat.com
<mailto:sdouglas@redhat.com>> wrote:
This sounds like a bug, when the client closes the connection it
should wake up the read listener, which will read -1 and then
cleanly close the socket.
Are the clients closing idle connections or connections processing
a request?
Stuart
On Mon, 2 Mar 2020 at 14:31, Nishant Kumar
<nishantkumar35(a)gmail.com <mailto:nishantkumar35@gmail.com>> wrote:
I agree that it's a load-balancing issue but we can't do much
about it at this moment.
I still see issues after using the latest XNIO (3.7.7) with
Undertow. what I have observed it that when there is a spike
in request and CONNECTION_HIGH_WATER is reached, the server
stops accepting new connection as expected and the client
starts to close the connection because of delay (we have
strict low latency requirement < 100ms) and try to create new
connection again (which will also not be accepted) but server
has not closed those connections (NO_REQUEST_TIMEOUT = 6000)
and there will be high number of CLOSE_WAIT connections at
this moment. The server is considering CLOSE_WAIT +
ESTABLISHED for CONNECTION_HIGH_WATER (my understanding).
Is there a way that I can close all CLOSE_WAIT connection at
this moment so that connection counts drop
under CONNECTION_HIGH_WATER and we start responding to newly
established connections? or any other suggestions? I have
tried removing CONNECTION_HIGH_WATER and relying on the FD
limit but that didn't work.
On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg
<stan.rosenberg(a)gmail.com <mailto:stan.rosenberg@gmail.com>>
wrote:
On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar
<nishantkumar35(a)gmail.com
<mailto:nishantkumar35@gmail.com>> wrote:
Thanks for the reply. I am running it under
supervisord and i have updated open file limit in
supervisord config. The problem seems to be same as
what @Carter has mentioned. It happens mostly during
sudden traffic spike and then sudden increase
(~30k-300k) of TIME_WAIT socket.
The changes in
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11...
likely to improve the exceptional case of exceeding the
file descriptor limit. However, if you're already setting
the limit too high (e.g., in our case it was 795588), then
exceeding it is a symptom of not properly load-balancing
your traffic; with that many connections, you'd better
have a ton of free RAM available.
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com <mailto:nishantkumar35@gmail.com>
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com <mailto:nishantkumar35@gmail.com>
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
7:13 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
It feels like CLOSE_WAIT connection are not getting closed properly.
Although not sure. If we reduce NO_REQUEST_TIMEOUT to small value, i can
see that CLOSE_WAIT are comparatively (number decrease faster) low but
still very high overall.
On Mon, Mar 2, 2020, 2:29 PM Nishant Kumar <nishantkumar35(a)gmail.com> wrote:
Generally, clients also close the connection after a few thousand
requests
other than normal fatal conditions. There might be other cases too but I am
not aware of it. They keep initiating new connections if we are not
responding within the threshold time frame. This is a server to server
communication system.
On Mon, Mar 2, 2020 at 10:26 AM Stuart Douglas <sdouglas(a)redhat.com>
wrote:
> This sounds like a bug, when the client closes the connection it should
> wake up the read listener, which will read -1 and then cleanly close the
> socket.
>
> Are the clients closing idle connections or connections processing a
> request?
>
> Stuart
>
> On Mon, 2 Mar 2020 at 14:31, Nishant Kumar <nishantkumar35(a)gmail.com>
> wrote:
>
>> I agree that it's a load-balancing issue but we can't do much about it
>> at this moment.
>>
>> I still see issues after using the latest XNIO (3.7.7) with Undertow.
>> what I have observed it that when there is a spike in request
>> and CONNECTION_HIGH_WATER is reached, the server stops accepting new
>> connection as expected and the client starts to close the connection
>> because of delay (we have strict low latency requirement < 100ms) and try
>> to create new connection again (which will also not be accepted) but server
>> has not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will
>> be high number of CLOSE_WAIT connections at this moment. The server is
>> considering CLOSE_WAIT + ESTABLISHED for CONNECTION_HIGH_WATER (my
>> understanding).
>>
>> Is there a way that I can close all CLOSE_WAIT connection at this
>> moment so that connection counts drop under CONNECTION_HIGH_WATER and we
>> start responding to newly established connections? or any other
>> suggestions? I have tried removing CONNECTION_HIGH_WATER and relying on the
>> FD limit but that didn't work.
>>
>> On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg <stan.rosenberg(a)gmail.com>
>> wrote:
>>
>>> On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar
<nishantkumar35(a)gmail.com>
>>> wrote:
>>>
>>>> Thanks for the reply. I am running it under supervisord and i have
>>>> updated open file limit in supervisord config. The problem seems to be
same
>>>> as what @Carter has mentioned. It happens mostly during sudden traffic
>>>> spike and then sudden increase (~30k-300k) of TIME_WAIT socket.
>>>>
>>>
>>> The changes in
>>>
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11... are
>>> likely to improve the exceptional case of exceeding the file descriptor
>>> limit. However, if you're already setting the limit too high (e.g., in
our
>>> case it was 795588), then exceeding it is a symptom of not properly
>>> load-balancing your traffic; with that many connections, you'd better
have
>>> a ton of free RAM available.
>>>
>>
>>
>> --
>> Nishant Kumar
>> Bangalore, India
>> Mob: +91 80088 42030
>> Email: nishantkumar35(a)gmail.com
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
7:54 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Stuck in CLOSE_WAIT is a symptom of the client-side not properly shutting
down [1]. I assume you have no control of your client traffic. Thus, the
best you can do is (a) raise fd limit to an unreasonably high number and
make sure you have plenty of RAM, (b) put the LB in front of undertow.
Truth be told we faced the same issue in GCP; unfortunately, GLB still
doesn't support max. conn. limit per backend afaik. The next best thing is
implementing a custom netfilter using iptables or ebpf. The level of
effort is rather high. Note, we're essentially talking about mitigations
for a low-level DDoS (perhaps a non-malicious one).
If all the above fail, yet another option is to use http/2. I believe
undertow supports it although I have no experience using it.
Best,
stan
[1]
https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-speci...
On Mon, Mar 2, 2020, 8:13 AM Nishant Kumar <nishantkumar35(a)gmail.com> wrote:
It feels like CLOSE_WAIT connection are not getting closed properly.
Although not sure. If we reduce NO_REQUEST_TIMEOUT to small value, i can
see that CLOSE_WAIT are comparatively (number decrease faster) low but
still very high overall.
On Mon, Mar 2, 2020, 2:29 PM Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
> Generally, clients also close the connection after a few thousand
> requests other than normal fatal conditions. There might be other cases too
> but I am not aware of it. They keep initiating new connections if we are
> not responding within the threshold time frame. This is a server to server
> communication system.
>
> On Mon, Mar 2, 2020 at 10:26 AM Stuart Douglas <sdouglas(a)redhat.com>
> wrote:
>
>> This sounds like a bug, when the client closes the connection it should
>> wake up the read listener, which will read -1 and then cleanly close the
>> socket.
>>
>> Are the clients closing idle connections or connections processing a
>> request?
>>
>> Stuart
>>
>> On Mon, 2 Mar 2020 at 14:31, Nishant Kumar <nishantkumar35(a)gmail.com>
>> wrote:
>>
>>> I agree that it's a load-balancing issue but we can't do much about
it
>>> at this moment.
>>>
>>> I still see issues after using the latest XNIO (3.7.7) with Undertow.
>>> what I have observed it that when there is a spike in request
>>> and CONNECTION_HIGH_WATER is reached, the server stops accepting new
>>> connection as expected and the client starts to close the connection
>>> because of delay (we have strict low latency requirement < 100ms) and try
>>> to create new connection again (which will also not be accepted) but server
>>> has not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will
>>> be high number of CLOSE_WAIT connections at this moment. The server is
>>> considering CLOSE_WAIT + ESTABLISHED for CONNECTION_HIGH_WATER (my
>>> understanding).
>>>
>>> Is there a way that I can close all CLOSE_WAIT connection at this
>>> moment so that connection counts drop under CONNECTION_HIGH_WATER and we
>>> start responding to newly established connections? or any other
>>> suggestions? I have tried removing CONNECTION_HIGH_WATER and relying on the
>>> FD limit but that didn't work.
>>>
>>> On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg
<stan.rosenberg(a)gmail.com>
>>> wrote:
>>>
>>>> On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar <
>>>> nishantkumar35(a)gmail.com> wrote:
>>>>
>>>>> Thanks for the reply. I am running it under supervisord and i have
>>>>> updated open file limit in supervisord config. The problem seems to
be same
>>>>> as what @Carter has mentioned. It happens mostly during sudden
traffic
>>>>> spike and then sudden increase (~30k-300k) of TIME_WAIT socket.
>>>>>
>>>>
>>>> The changes in
>>>>
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11... are
>>>> likely to improve the exceptional case of exceeding the file descriptor
>>>> limit. However, if you're already setting the limit too high (e.g.,
in our
>>>> case it was 795588), then exceeding it is a symptom of not properly
>>>> load-balancing your traffic; with that many connections, you'd better
have
>>>> a ton of free RAM available.
>>>>
>>>
>>>
>>> --
>>> Nishant Kumar
>>> Bangalore, India
>>> Mob: +91 80088 42030
>>> Email: nishantkumar35(a)gmail.com
>>> _______________________________________________
>>> undertow-dev mailing list
>>> undertow-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>>
>
> --
> Nishant Kumar
> Bangalore, India
> Mob: +91 80088 42030
> Email: nishantkumar35(a)gmail.com
>
8:17 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <stan.rosenberg(a)acm.org> wrote:
Stuck in CLOSE_WAIT is a symptom of the client-side not properly shutting down [1].
I would partially disagree. In the article you linked: "It all starts
with a listening application that leaks sockets and forgets to call
close(). This kind of bug does happen in complex applications." This
seems to be essentially what's happening here: the server isn't
completing the connection (for some reason), stranding the socket in
`CLOSE_WAIT`.
We can't assume that the client is abandoning the connection after
`FIN_WAIT2` (the titular RFC violation); if the server stays in
`CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
forever, the resolving condition still needs to be that the server
shuts down its side of the connection.
This diagram is a useful visual aid, mapping TCP states to the XNIO
API:
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
--
- DML
10:40 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Yes, i have no control on client side. I am using HTTP2. I have tried
increasing open file limit to 400k but that consumes all memory and system
hangs. I will probably try to put a nginx in front of Undertow and test.
setServerOption(UndertowOptions.ENABLE_HTTP2, true)
On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com> wrote:
On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg
<stan.rosenberg(a)acm.org>
wrote:
>
> Stuck in CLOSE_WAIT is a symptom of the client-side not properly
shutting down [1].
I would partially disagree. In the article you linked: "It all starts
with a listening application that leaks sockets and forgets to call
close(). This kind of bug does happen in complex applications." This
seems to be essentially what's happening here: the server isn't
completing the connection (for some reason), stranding the socket in
`CLOSE_WAIT`.
We can't assume that the client is abandoning the connection after
`FIN_WAIT2` (the titular RFC violation); if the server stays in
`CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
forever, the resolving condition still needs to be that the server
shuts down its side of the connection.
This diagram is a useful visual aid, mapping TCP states to the XNIO
API:
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
--
- DML
5:47 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow the
connection is not being closed if the client hangs up abruptly. I had a
quick look at the code though and I think it looks ok, but maybe some more
investigation is needed.
Stuart
On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com> wrote:
Yes, i have no control on client side. I am using HTTP2. I have
tried
increasing open file limit to 400k but that consumes all memory and system
hangs. I will probably try to put a nginx in front of Undertow and test.
setServerOption(UndertowOptions.ENABLE_HTTP2, true)
On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com> wrote:
> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <stan.rosenberg(a)acm.org>
> wrote:
> >
> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly
> shutting down [1].
>
> I would partially disagree. In the article you linked: "It all starts
> with a listening application that leaks sockets and forgets to call
> close(). This kind of bug does happen in complex applications." This
> seems to be essentially what's happening here: the server isn't
> completing the connection (for some reason), stranding the socket in
> `CLOSE_WAIT`.
>
> We can't assume that the client is abandoning the connection after
> `FIN_WAIT2` (the titular RFC violation); if the server stays in
> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
> forever, the resolving condition still needs to be that the server
> shuts down its side of the connection.
>
> This diagram is a useful visual aid, mapping TCP states to the XNIO
> API:
>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>
> --
> - DML
>
>
3:24 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
I tried disabling HTTP/2 but still too many CLOSE_WAIT connections. I tried
putting Nginx in front of the Java server but seems the same issue with it.
It seems Nginx is also creating the same number of connections with Java.
server = Undertow.builder()
.addHttpListener(SERVER_LISTEN_PORT, SERVER_HOST)
.addHttpsListener(SERVER_SSL_LISTEN_PORT, SERVER_HOST, sslContext)
.setWorkerThreads(WORKER_THREAD)
*.setServerOption(UndertowOptions.ENABLE_HTTP2, false)*
.setServerOption(UndertowOptions.IDLE_TIMEOUT, 150000) // 150s
.setServerOption(UndertowOptions.NO_REQUEST_TIMEOUT, 150000) //
150s
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE,
1024 * 20) // 20000 sessions
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT,
1500) // 150s
.setIoThreads(IO_THREAD)
.setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
.setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
.setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 100000)
.setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 100000)
.setHandler(Handlers.routing().post("/", new
RequestHandler(appContext)))
.build();
*# netstat -nalp | grep -E ":80 |:443 " | awk
'{split($4,a,":");print
a[2] " " $6}'| sort | uniq -c 85918 443 CLOSE_WAIT 10279 443 ESTABLISHED
67 443 LAST_ACK 152 443 SYN_RECV 505 443 TIME_WAIT 31151 80
CLOSE_WAIT 3747 80 ESTABLISHED 108 80 LAST_ACK 146 80 SYN_RECV
2 LISTEN*
On Tue, Mar 3, 2020 at 5:17 AM Stuart Douglas <sdouglas(a)redhat.com> wrote:
Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
the
connection is not being closed if the client hangs up abruptly. I had a
quick look at the code though and I think it looks ok, but maybe some more
investigation is needed.
Stuart
On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
> Yes, i have no control on client side. I am using HTTP2. I have tried
> increasing open file limit to 400k but that consumes all memory and system
> hangs. I will probably try to put a nginx in front of Undertow and test.
>
> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>
> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com> wrote:
>
>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <stan.rosenberg(a)acm.org>
>> wrote:
>> >
>> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly
>> shutting down [1].
>>
>> I would partially disagree. In the article you linked: "It all starts
>> with a listening application that leaks sockets and forgets to call
>> close(). This kind of bug does happen in complex applications." This
>> seems to be essentially what's happening here: the server isn't
>> completing the connection (for some reason), stranding the socket in
>> `CLOSE_WAIT`.
>>
>> We can't assume that the client is abandoning the connection after
>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>> forever, the resolving condition still needs to be that the server
>> shuts down its side of the connection.
>>
>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>> API:
>>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>
>> --
>> - DML
>>
>>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
4:03 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Hi,
I also encountered the "too many TCP connections in CLOSE_WAIT" issue
with multiple web servers (also Undertow).
We see this happening with mobile clients a lot as they often loose
connection in the most weird moments (railway tunnel, person walks into
building, ...).
As far as I'm aware this is an issue with TCP in general, when the
remote client just goes away (network route severed/power lost, not just
process terminated).
Thus a workaround is best applied on the layer actually handling the TCP
connection state. This is OSI layer 4 and handled by the operating
system kernel network stack.
On Linux I successfully use the following sysctl.conf entries to keep
the CLOSE_WAIT connections in check and avoid netfilter issues:
# keep the number of TCP connections in CLOSE_WAIT low
# by killing CLOSE_WAIT sockets after some time (will look like RESET to
the server processes)
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
# tune TCP keepalive to be a little more practical (2h -> 5 minutes
timeout, kill connection after 2 failed probes)
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 300
# more contrack table entries (else we may get some "upstream connection
timed out"
# only needed when using netfilter connection tracking (for example NAT)
# net.nf_conntrack_max = 6556000
Undertow seems to work OK and behaves as an application on layer 7
should. There are open issues when using TLSv1.3 (100% CPU XNIO I/O
threads), but that seems off-topic here.
note: The inherent issues of TCP will likely go away when HTTP3 is in
wide use, as it replaces layer 4 TCP with layer 4+ QUIC.
On 2020-03-03 10:24, Nishant Kumar wrote:
I tried disabling HTTP/2 but still too many CLOSE_WAIT connections.
I
tried putting Nginx in front of the Java server but seems the same
issue with it. It seems Nginx is also creating the same number of
connections with Java.
server = Undertow.builder()
.addHttpListener(SERVER_LISTEN_PORT, SERVER_HOST)
.addHttpsListener(SERVER_SSL_LISTEN_PORT, SERVER_HOST,
sslContext)
.setWorkerThreads(WORKER_THREAD)
.SETSERVEROPTION(UNDERTOWOPTIONS.ENABLE_HTTP2, FALSE)
.setServerOption(UndertowOptions.IDLE_TIMEOUT, 150000) //
150s
.setServerOption(UndertowOptions.NO_REQUEST_TIMEOUT, 150000)
// 150s
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024
* 20) // 20000 sessions
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 1500) //
150s
.setIoThreads(IO_THREAD)
.setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
.setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
.setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER,
100000)
.setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER,
100000)
.setHandler(Handlers.routing().post("/", new
RequestHandler(appContext)))
.build();
# netstat -nalp | grep -E ":80 |:443 " | awk
'{split($4,a,":");print
a[2] " " $6}'| sort | uniq -c
85918 443 CLOSE_WAIT
10279 443 ESTABLISHED
67 443 LAST_ACK
152 443 SYN_RECV
505 443 TIME_WAIT
31151 80 CLOSE_WAIT
3747 80 ESTABLISHED
108 80 LAST_ACK
146 80 SYN_RECV
2 LISTEN
On Tue, Mar 3, 2020 at 5:17 AM Stuart Douglas <sdouglas(a)redhat.com>
wrote:
> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
> the connection is not being closed if the client hangs up abruptly.
> I had a quick look at the code though and I think it looks ok, but
> maybe some more investigation is needed.
>
> Stuart
>
> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar
> <nishantkumar35(a)gmail.com> wrote:
>
> Yes, i have no control on client side. I am using HTTP2. I have
> tried increasing open file limit to 400k but that consumes all
> memory and system hangs. I will probably try to put a nginx in front
> of Undertow and test.
>
> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>
> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com>
> wrote:
> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg
> <stan.rosenberg(a)acm.org> wrote:
>>
>> Stuck in CLOSE_WAIT is a symptom of the client-side not properly
> shutting down [1].
>
> I would partially disagree. In the article you linked: "It all
> starts
> with a listening application that leaks sockets and forgets to call
> close(). This kind of bug does happen in complex applications."
> This
> seems to be essentially what's happening here: the server isn't
> completing the connection (for some reason), stranding the socket in
> `CLOSE_WAIT`.
>
> We can't assume that the client is abandoning the connection after
> `FIN_WAIT2` (the titular RFC violation); if the server stays in
> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
> forever, the resolving condition still needs to be that the server
> shuts down its side of the connection.
>
> This diagram is a useful visual aid, mapping TCP states to the XNIO
> API:
>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>
> --
> - DML
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
8 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
On Tue, Mar 3, 2020 at 4:05 AM Jonas Zeiger <jonas.zeiger(a)talpidae.net> wrote:
[...]
Thus a workaround is best applied on the layer actually handling the TCP
connection state.
This is a really great point. The fact is that in most cases, we
really have no way at the Java level to cope with a server request
that is taking too long due to overload in conjunction with an
impatient client. Whether it's a thread pool queue that is overlong,
or the list of ready sockets takes too long to iterate, or any other
case, there's really no way we can know immediately when the client
has decided to give up on an HTTP request or connection.
On Linux I successfully use the following sysctl.conf entries to
keep
the CLOSE_WAIT connections in check and avoid netfilter issues:
# keep the number of TCP connections in CLOSE_WAIT low
# by killing CLOSE_WAIT sockets after some time (will look like RESET to
the server processes)
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
# tune TCP keepalive to be a little more practical (2h -> 5 minutes
timeout, kill connection after 2 failed probes)
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 300
# more contrack table entries (else we may get some "upstream connection
timed out"
# only needed when using netfilter connection tracking (for example NAT)
# net.nf_conntrack_max = 6556000
This seems like a very pragmatic solution. I'm not sure this would
lower the actual number of file descriptors but it would help with the
situation where clients are transitioning out of FIN_WAIT_2 too early.
--
- DML
11:24 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Can someone provide a reproducer for this error?
As for the old version of XNIO, it will be upgraded in Undertow 2.1.0.Final.
On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com> wrote:
Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
the
connection is not being closed if the client hangs up abruptly. I had a
quick look at the code though and I think it looks ok, but maybe some more
investigation is needed.
Stuart
On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
> Yes, i have no control on client side. I am using HTTP2. I have tried
> increasing open file limit to 400k but that consumes all memory and system
> hangs. I will probably try to put a nginx in front of Undertow and test.
>
> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>
> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com> wrote:
>
>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <stan.rosenberg(a)acm.org>
>> wrote:
>> >
>> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly
>> shutting down [1].
>>
>> I would partially disagree. In the article you linked: "It all starts
>> with a listening application that leaks sockets and forgets to call
>> close(). This kind of bug does happen in complex applications." This
>> seems to be essentially what's happening here: the server isn't
>> completing the connection (for some reason), stranding the socket in
>> `CLOSE_WAIT`.
>>
>> We can't assume that the client is abandoning the connection after
>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>> forever, the resolving condition still needs to be that the server
>> shuts down its side of the connection.
>>
>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>> API:
>>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>
>> --
>> - DML
>>
>> _______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Flavia Rainone
Principal Software Engineer
Red Hat <https://www.redhat.com>
frainone(a)redhat.com
<https://www.redhat.com>
2:38 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
If this issue is similar to the one we encountered on our machines, this
helps to reproduce it (HTTP2, TLSv1.3 is optional):
https://gist.github.com/lyind/ae076548cafb2cb0b46a0819b749d6f4#file-curl-...
The critical steps are:
1. Attach debugger to your server process (normally via remote
debugging)
2. On a client at least one layer 3 network segment away (behind one or
more IP routers):
start >= 6 requests in a loop (increase the chance of "vulnerable"
TCP connections at step 3)
3. Sever the connection WITHOUT one of the two (client, server)
operating system kernels being able to update TCP state:
pull LAN cable of the client or shutdown router at the client-side
4. Repeat 2) and 3) until:
1. $ netstat -tnp | grep java
-> some connections in CLOSE_WAIT (then look how the server
handles these, play with operating system tunables)
2. (optional) XNIO threads stuck at 100% CPU
3. (optional) unexpected exceptions occur
In theory one could also use some firewall or network emulator trickery
to cut the connections but that would be harder to get right and still
be quite complex for a unit or integration tests. Sounds more like a
case for the physical network test lab.
On 2020-03-04 06:24, Flavia Rainone wrote:
Can someone provide a reproducer for this error?
As for the old version of XNIO, it will be upgraded in Undertow
2.1.0.Final.
On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com>
wrote:
> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
> the connection is not being closed if the client hangs up abruptly.
> I had a quick look at the code though and I think it looks ok, but
> maybe some more investigation is needed.
>
> Stuart
>
> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar
> <nishantkumar35(a)gmail.com> wrote:
>
> Yes, i have no control on client side. I am using HTTP2. I have
> tried increasing open file limit to 400k but that consumes all
> memory and system hangs. I will probably try to put a nginx in front
> of Undertow and test.
>
> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>
> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com>
> wrote:
> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg
> <stan.rosenberg(a)acm.org> wrote:
>>
>> Stuck in CLOSE_WAIT is a symptom of the client-side not properly
> shutting down [1].
>
> I would partially disagree. In the article you linked: "It all
> starts
> with a listening application that leaks sockets and forgets to call
> close(). This kind of bug does happen in complex applications."
> This
> seems to be essentially what's happening here: the server isn't
> completing the connection (for some reason), stranding the socket in
> `CLOSE_WAIT`.
>
> We can't assume that the client is abandoning the connection after
> `FIN_WAIT2` (the titular RFC violation); if the server stays in
> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
> forever, the resolving condition still needs to be that the server
> shuts down its side of the connection.
>
> This diagram is a useful visual aid, mapping TCP states to the XNIO
> API:
>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>
> --
> - DML
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Flavia Rainone
Principal Software Engineer
Red Hat [1]
frainone(a)redhat.com
[1]
Links:
------
[1] https://www.redhat.com
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
3:20 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Thanks Jonas!
I'll add this to the team's triage and see if we can reproduce it and,
hopefully, find a solution.
On Wed, Mar 4, 2020 at 5:38 AM Jonas Zeiger <jonas.zeiger(a)talpidae.net>
wrote:
If this issue is similar to the one we encountered on our machines,
this
helps to reproduce it (HTTP2, TLSv1.3 is optional):
https://gist.github.com/lyind/ae076548cafb2cb0b46a0819b749d6f4#file-curl-...
The critical steps are:
1. Attach debugger to your server process (normally via remote
debugging)
2. On a client at least one layer 3 network segment away (behind one or
more IP routers):
start >= 6 requests in a loop (increase the chance of "vulnerable"
TCP connections at step 3)
3. Sever the connection WITHOUT one of the two (client, server)
operating system kernels being able to update TCP state:
pull LAN cable of the client or shutdown router at the client-side
4. Repeat 2) and 3) until:
1. $ netstat -tnp | grep java
-> some connections in CLOSE_WAIT (then look how the server
handles these, play with operating system tunables)
2. (optional) XNIO threads stuck at 100% CPU
3. (optional) unexpected exceptions occur
In theory one could also use some firewall or network emulator trickery
to cut the connections but that would be harder to get right and still
be quite complex for a unit or integration tests. Sounds more like a
case for the physical network test lab.
On 2020-03-04 06:24, Flavia Rainone wrote:
> Can someone provide a reproducer for this error?
>
> As for the old version of XNIO, it will be upgraded in Undertow
> 2.1.0.Final.
>
> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com>
> wrote:
>
>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
>> the connection is not being closed if the client hangs up abruptly.
>> I had a quick look at the code though and I think it looks ok, but
>> maybe some more investigation is needed.
>>
>> Stuart
>>
>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar
>> <nishantkumar35(a)gmail.com> wrote:
>>
>> Yes, i have no control on client side. I am using HTTP2. I have
>> tried increasing open file limit to 400k but that consumes all
>> memory and system hangs. I will probably try to put a nginx in front
>> of Undertow and test.
>>
>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>
>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com>
>> wrote:
>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg
>> <stan.rosenberg(a)acm.org> wrote:
>>>
>>> Stuck in CLOSE_WAIT is a symptom of the client-side not properly
>> shutting down [1].
>>
>> I would partially disagree. In the article you linked: "It all
>> starts
>> with a listening application that leaks sockets and forgets to call
>> close(). This kind of bug does happen in complex applications."
>> This
>> seems to be essentially what's happening here: the server isn't
>> completing the connection (for some reason), stranding the socket in
>> `CLOSE_WAIT`.
>>
>> We can't assume that the client is abandoning the connection after
>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>> forever, the resolving condition still needs to be that the server
>> shuts down its side of the connection.
>>
>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>> API:
>>
>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>
>> --
>> - DML
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
> --
>
> Flavia Rainone
>
> Principal Software Engineer
>
> Red Hat [1]
>
> frainone(a)redhat.com
>
> [1]
>
>
>
> Links:
> ------
> [1] https://www.redhat.com
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Flavia Rainone
Principal Software Engineer
Red Hat <https://www.redhat.com>
frainone(a)redhat.com
<https://www.redhat.com>
8:50 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
I didn't get time to write standalone program to reproduce it but with new
version 3.7.7-Final, i am not able to survive for more than an hour. With
the older default xnio version of Undertow (xnio 3.3.8-Final) i can run
server for few days until it stops working with error i mentioned in first
email.
Just FYI, i am using OpenJDK 11 with OpenJ9 JVM.
On Wed, Mar 4, 2020, 10:54 AM Flavia Rainone <frainone(a)redhat.com> wrote:
Can someone provide a reproducer for this error?
As for the old version of XNIO, it will be upgraded in Undertow
2.1.0.Final.
On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com> wrote:
> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow the
> connection is not being closed if the client hangs up abruptly. I had a
> quick look at the code though and I think it looks ok, but maybe some more
> investigation is needed.
>
> Stuart
>
> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com>
> wrote:
>
>> Yes, i have no control on client side. I am using HTTP2. I have tried
>> increasing open file limit to 400k but that consumes all memory and system
>> hangs. I will probably try to put a nginx in front of Undertow and test.
>>
>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>
>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com> wrote:
>>
>>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <stan.rosenberg(a)acm.org>
>>> wrote:
>>> >
>>> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly
>>> shutting down [1].
>>>
>>> I would partially disagree. In the article you linked: "It all starts
>>> with a listening application that leaks sockets and forgets to call
>>> close(). This kind of bug does happen in complex applications." This
>>> seems to be essentially what's happening here: the server isn't
>>> completing the connection (for some reason), stranding the socket in
>>> `CLOSE_WAIT`.
>>>
>>> We can't assume that the client is abandoning the connection after
>>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>>> forever, the resolving condition still needs to be that the server
>>> shuts down its side of the connection.
>>>
>>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>>> API:
>>>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>>
>>> --
>>> - DML
>>>
>>> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Flavia Rainone
Principal Software Engineer
Red Hat <https://www.redhat.com>
frainone(a)redhat.com
<https://www.redhat.com>
3:07 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
I have also observed that the CPU load increases significantly on
3.7.7-Final (load average 50 on 48 core machine) on moderate load but it's
very low (load average 20) on 3.3.8-Final on the same QPS (20k Req/sec) and
it (3.7.7-Final) starts refusing connection on high load.
On Thu, Mar 12, 2020 at 7:20 AM Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
I didn't get time to write standalone program to reproduce it but
with new
version 3.7.7-Final, i am not able to survive for more than an hour. With
the older default xnio version of Undertow (xnio 3.3.8-Final) i can run
server for few days until it stops working with error i mentioned in first
email.
Just FYI, i am using OpenJDK 11 with OpenJ9 JVM.
On Wed, Mar 4, 2020, 10:54 AM Flavia Rainone <frainone(a)redhat.com> wrote:
> Can someone provide a reproducer for this error?
>
> As for the old version of XNIO, it will be upgraded in Undertow
> 2.1.0.Final.
>
> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com>
> wrote:
>
>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow the
>> connection is not being closed if the client hangs up abruptly. I had a
>> quick look at the code though and I think it looks ok, but maybe some more
>> investigation is needed.
>>
>> Stuart
>>
>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com>
>> wrote:
>>
>>> Yes, i have no control on client side. I am using HTTP2. I have tried
>>> increasing open file limit to 400k but that consumes all memory and system
>>> hangs. I will probably try to put a nginx in front of Undertow and test.
>>>
>>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>>
>>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com>
>>> wrote:
>>>
>>>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg
<stan.rosenberg(a)acm.org>
>>>> wrote:
>>>> >
>>>> > Stuck in CLOSE_WAIT is a symptom of the client-side not properly
>>>> shutting down [1].
>>>>
>>>> I would partially disagree. In the article you linked: "It all
starts
>>>> with a listening application that leaks sockets and forgets to call
>>>> close(). This kind of bug does happen in complex applications."
This
>>>> seems to be essentially what's happening here: the server isn't
>>>> completing the connection (for some reason), stranding the socket in
>>>> `CLOSE_WAIT`.
>>>>
>>>> We can't assume that the client is abandoning the connection after
>>>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>>>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>>>> forever, the resolving condition still needs to be that the server
>>>> shuts down its side of the connection.
>>>>
>>>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>>>> API:
>>>>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>>>
>>>> --
>>>> - DML
>>>>
>>>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
>
> --
>
> Flavia Rainone
>
> Principal Software Engineer
>
> Red Hat <https://www.redhat.com>
>
> frainone(a)redhat.com
> <https://www.redhat.com>
>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
11:56 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
I have tried XNIO 3.7.7-Final with jboss.threads 2.3.3.Final in
Undertow 2.0.30.Final
and it looks good so far. I am running it in production for the last 2 days
and no issues so far :). I am monitoring it and will report if find
anything.
The reason I changed jboss thread version is that I observed a high CPU
load in XNIO 3.7.7.Final comparing XNIO 3.3.8.Final which I reported in the
last email. However, I have not done any extensive tests for it.
On Thu, Mar 12, 2020 at 1:37 PM Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
I have also observed that the CPU load increases significantly on
3.7.7-Final (load average 50 on 48 core machine) on moderate load but it's
very low (load average 20) on 3.3.8-Final on the same QPS (20k Req/sec) and
it (3.7.7-Final) starts refusing connection on high load.
On Thu, Mar 12, 2020 at 7:20 AM Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
> I didn't get time to write standalone program to reproduce it but with
> new version 3.7.7-Final, i am not able to survive for more than an hour.
> With the older default xnio version of Undertow (xnio 3.3.8-Final) i can
> run server for few days until it stops working with error i mentioned in
> first email.
>
> Just FYI, i am using OpenJDK 11 with OpenJ9 JVM.
>
> On Wed, Mar 4, 2020, 10:54 AM Flavia Rainone <frainone(a)redhat.com> wrote:
>
>> Can someone provide a reproducer for this error?
>>
>> As for the old version of XNIO, it will be upgraded in Undertow
>> 2.1.0.Final.
>>
>> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com>
>> wrote:
>>
>>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow the
>>> connection is not being closed if the client hangs up abruptly. I had a
>>> quick look at the code though and I think it looks ok, but maybe some more
>>> investigation is needed.
>>>
>>> Stuart
>>>
>>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar <nishantkumar35(a)gmail.com>
>>> wrote:
>>>
>>>> Yes, i have no control on client side. I am using HTTP2. I have tried
>>>> increasing open file limit to 400k but that consumes all memory and
system
>>>> hangs. I will probably try to put a nginx in front of Undertow and test.
>>>>
>>>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>>>
>>>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd <david.lloyd(a)redhat.com>
>>>> wrote:
>>>>
>>>>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg
<stan.rosenberg(a)acm.org>
>>>>> wrote:
>>>>> >
>>>>> > Stuck in CLOSE_WAIT is a symptom of the client-side not
properly
>>>>> shutting down [1].
>>>>>
>>>>> I would partially disagree. In the article you linked: "It all
starts
>>>>> with a listening application that leaks sockets and forgets to call
>>>>> close(). This kind of bug does happen in complex applications."
This
>>>>> seems to be essentially what's happening here: the server
isn't
>>>>> completing the connection (for some reason), stranding the socket in
>>>>> `CLOSE_WAIT`.
>>>>>
>>>>> We can't assume that the client is abandoning the connection
after
>>>>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>>>>> `CLOSE_WAIT`, then even if the client dutifully stays in `FIN_WAIT2`
>>>>> forever, the resolving condition still needs to be that the server
>>>>> shuts down its side of the connection.
>>>>>
>>>>> This diagram is a useful visual aid, mapping TCP states to the XNIO
>>>>> API:
>>>>>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>>>>
>>>>> --
>>>>> - DML
>>>>>
>>>>> _______________________________________________
>>> undertow-dev mailing list
>>> undertow-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>>
>>
>> --
>>
>> Flavia Rainone
>>
>> Principal Software Engineer
>>
>> Red Hat <https://www.redhat.com>
>>
>> frainone(a)redhat.com
>> <https://www.redhat.com>
>>
>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
6:42 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Hi Nishant,
thanks for the update - that is good news!
Rio
On Sun, Mar 15, 2020 at 6:00 AM Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
I have tried XNIO 3.7.7-Final with jboss.threads 2.3.3.Final in
Undertow 2.0.30.Final
and it looks good so far. I am running it in production for the last 2 days
and no issues so far :). I am monitoring it and will report if find
anything.
The reason I changed jboss thread version is that I observed a high CPU
load in XNIO 3.7.7.Final comparing XNIO 3.3.8.Final which I reported in the
last email. However, I have not done any extensive tests for it.
On Thu, Mar 12, 2020 at 1:37 PM Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
> I have also observed that the CPU load increases significantly on
> 3.7.7-Final (load average 50 on 48 core machine) on moderate load but it's
> very low (load average 20) on 3.3.8-Final on the same QPS (20k Req/sec) and
> it (3.7.7-Final) starts refusing connection on high load.
>
> On Thu, Mar 12, 2020 at 7:20 AM Nishant Kumar <nishantkumar35(a)gmail.com>
> wrote:
>
>> I didn't get time to write standalone program to reproduce it but with
>> new version 3.7.7-Final, i am not able to survive for more than an hour.
>> With the older default xnio version of Undertow (xnio 3.3.8-Final) i can
>> run server for few days until it stops working with error i mentioned in
>> first email.
>>
>> Just FYI, i am using OpenJDK 11 with OpenJ9 JVM.
>>
>> On Wed, Mar 4, 2020, 10:54 AM Flavia Rainone <frainone(a)redhat.com>
>> wrote:
>>
>>> Can someone provide a reproducer for this error?
>>>
>>> As for the old version of XNIO, it will be upgraded in Undertow
>>> 2.1.0.Final.
>>>
>>> On Mon, Mar 2, 2020 at 8:47 PM Stuart Douglas <sdouglas(a)redhat.com>
>>> wrote:
>>>
>>>> Hmm, maybe this is a bug in the HTTP/2 close code then, and somehow
>>>> the connection is not being closed if the client hangs up abruptly. I had
a
>>>> quick look at the code though and I think it looks ok, but maybe some
more
>>>> investigation is needed.
>>>>
>>>> Stuart
>>>>
>>>> On Tue, 3 Mar 2020 at 03:41, Nishant Kumar
<nishantkumar35(a)gmail.com>
>>>> wrote:
>>>>
>>>>> Yes, i have no control on client side. I am using HTTP2. I have
tried
>>>>> increasing open file limit to 400k but that consumes all memory and
system
>>>>> hangs. I will probably try to put a nginx in front of Undertow and
test.
>>>>>
>>>>> setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>>>>
>>>>> On Mon, Mar 2, 2020, 7:48 PM David Lloyd
<david.lloyd(a)redhat.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Mar 2, 2020 at 7:56 AM Stan Rosenberg <
>>>>>> stan.rosenberg(a)acm.org> wrote:
>>>>>> >
>>>>>> > Stuck in CLOSE_WAIT is a symptom of the client-side not
properly
>>>>>> shutting down [1].
>>>>>>
>>>>>> I would partially disagree. In the article you linked: "It
all
>>>>>> starts
>>>>>> with a listening application that leaks sockets and forgets to
call
>>>>>> close(). This kind of bug does happen in complex
applications." This
>>>>>> seems to be essentially what's happening here: the server
isn't
>>>>>> completing the connection (for some reason), stranding the socket
in
>>>>>> `CLOSE_WAIT`.
>>>>>>
>>>>>> We can't assume that the client is abandoning the connection
after
>>>>>> `FIN_WAIT2` (the titular RFC violation); if the server stays in
>>>>>> `CLOSE_WAIT`, then even if the client dutifully stays in
`FIN_WAIT2`
>>>>>> forever, the resolving condition still needs to be that the
server
>>>>>> shuts down its side of the connection.
>>>>>>
>>>>>> This diagram is a useful visual aid, mapping TCP states to the
XNIO
>>>>>> API:
>>>>>>
https://www.lucidchart.com/publicSegments/view/524ec20a-5c40-4fd0-8bde-0a...
>>>>>>
>>>>>> --
>>>>>> - DML
>>>>>>
>>>>>> _______________________________________________
>>>> undertow-dev mailing list
>>>> undertow-dev(a)lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>
>>>
>>>
>>> --
>>>
>>> Flavia Rainone
>>>
>>> Principal Software Engineer
>>>
>>> Red Hat <https://www.redhat.com>
>>>
>>> frainone(a)redhat.com
>>> <https://www.redhat.com>
>>>
>>
>
> --
> Nishant Kumar
> Bangalore, India
> Mob: +91 80088 42030
> Email: nishantkumar35(a)gmail.com
>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Richard Opalka
Principal Software Engineer
Red Hat JBoss Middleware
Mobile: +420 731 186 942
E-mail: ropalka(a)redhat.com
8:11 a.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
Once you exceed the high water mark, what you should be seeing in
netstat on the client side are connections waiting in SYN_SENT. They
shouldn't even appear on the server's netstat, even if the client
closes and re-opens the connection (I've verified this experimentally
at the socket level). You are correct that connections in CLOSE_WAIT
indicate that somehow the HTTP server is not correctly terminating the
connection; under most normal HTTP 1.1 operational scenarios, there
should be very few CLOSE_WAIT connections at any given time.
The next step would be diagnosing why the server is not closing
connections promptly.
On Mon, Mar 2, 2020 at 7:19 AM Nishant Kumar <nishantkumar35(a)gmail.com> wrote:
It feels like CLOSE_WAIT connection are not getting closed properly. Although not sure.
If we reduce NO_REQUEST_TIMEOUT to small value, i can see that CLOSE_WAIT are
comparatively (number decrease faster) low but still very high overall.
On Mon, Mar 2, 2020, 2:29 PM Nishant Kumar <nishantkumar35(a)gmail.com> wrote:
>
> Generally, clients also close the connection after a few thousand requests other than
normal fatal conditions. There might be other cases too but I am not aware of it. They
keep initiating new connections if we are not responding within the threshold time frame.
This is a server to server communication system.
>
> On Mon, Mar 2, 2020 at 10:26 AM Stuart Douglas <sdouglas(a)redhat.com> wrote:
>>
>> This sounds like a bug, when the client closes the connection it should wake up
the read listener, which will read -1 and then cleanly close the socket.
>>
>> Are the clients closing idle connections or connections processing a request?
>>
>> Stuart
>>
>> On Mon, 2 Mar 2020 at 14:31, Nishant Kumar <nishantkumar35(a)gmail.com>
wrote:
>>>
>>> I agree that it's a load-balancing issue but we can't do much about
it at this moment.
>>>
>>> I still see issues after using the latest XNIO (3.7.7) with Undertow. what I
have observed it that when there is a spike in request and CONNECTION_HIGH_WATER is
reached, the server stops accepting new connection as expected and the client starts to
close the connection because of delay (we have strict low latency requirement < 100ms)
and try to create new connection again (which will also not be accepted) but server has
not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will be high number of
CLOSE_WAIT connections at this moment. The server is considering CLOSE_WAIT + ESTABLISHED
for CONNECTION_HIGH_WATER (my understanding).
>>>
>>> Is there a way that I can close all CLOSE_WAIT connection at this moment so
that connection counts drop under CONNECTION_HIGH_WATER and we start responding to newly
established connections? or any other suggestions? I have tried removing
CONNECTION_HIGH_WATER and relying on the FD limit but that didn't work.
>>>
>>> On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg
<stan.rosenberg(a)gmail.com> wrote:
>>>>
>>>> On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar
<nishantkumar35(a)gmail.com> wrote:
>>>>>
>>>>> Thanks for the reply. I am running it under supervisord and i have
updated open file limit in supervisord config. The problem seems to be same as what
@Carter has mentioned. It happens mostly during sudden traffic spike and then sudden
increase (~30k-300k) of TIME_WAIT socket.
>>>>
>>>>
>>>> The changes in
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11... are
likely to improve the exceptional case of exceeding the file descriptor limit. However, if
you're already setting the limit too high (e.g., in our case it was 795588), then
exceeding it is a symptom of not properly load-balancing your traffic; with that many
connections, you'd better have a ton of free RAM available.
>>>
>>>
>>>
>>> --
>>> Nishant Kumar
>>> Bangalore, India
>>> Mob: +91 80088 42030
>>> Email: nishantkumar35(a)gmail.com
>>> _______________________________________________
>>> undertow-dev mailing list
>>> undertow-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
>
> --
> Nishant Kumar
> Bangalore, India
> Mob: +91 80088 42030
> Email: nishantkumar35(a)gmail.com
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
--
- DML
12:49 p.m.
New subject: Too many open files: Exception accepting request, closing server channel TCP server (NIO)
That sounds like the same issue as mine (see other thread this month) - sockets not being
closed while waiting for a worker.
On 2 Mar 2020, at 04:57, Stuart Douglas
<sdouglas@redhat.com<mailto:sdouglas@redhat.com>> wrote:
This sounds like a bug, when the client closes the connection it should wake up the read
listener, which will read -1 and then cleanly close the socket.
Are the clients closing idle connections or connections processing a request?
Stuart
On Mon, 2 Mar 2020 at 14:31, Nishant Kumar
<nishantkumar35@gmail.com<mailto:nishantkumar35@gmail.com>> wrote:
I agree that it's a load-balancing issue but we can't do much about it at this
moment.
I still see issues after using the latest XNIO (3.7.7) with Undertow. what I have observed
it that when there is a spike in request and CONNECTION_HIGH_WATER is reached, the server
stops accepting new connection as expected and the client starts to close the connection
because of delay (we have strict low latency requirement < 100ms) and try to create new
connection again (which will also not be accepted) but server has not closed those
connections (NO_REQUEST_TIMEOUT = 6000) and there will be high number of CLOSE_WAIT
connections at this moment. The server is considering CLOSE_WAIT + ESTABLISHED for
CONNECTION_HIGH_WATER (my understanding).
Is there a way that I can close all CLOSE_WAIT connection at this moment so that
connection counts drop under CONNECTION_HIGH_WATER and we start responding to newly
established connections? or any other suggestions? I have tried removing
CONNECTION_HIGH_WATER and relying on the FD limit but that didn't work.
On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg
<stan.rosenberg@gmail.com<mailto:stan.rosenberg@gmail.com>> wrote:
On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar
<nishantkumar35@gmail.com<mailto:nishantkumar35@gmail.com>> wrote:
Thanks for the reply. I am running it under supervisord and i have updated open file limit
in supervisord config. The problem seems to be same as what @Carter has mentioned. It
happens mostly during sudden traffic spike and then sudden increase (~30k-300k) of
TIME_WAIT socket.
The changes in
https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11... are
likely to improve the exceptional case of exceeding the file descriptor limit. However, if
you're already setting the limit too high (e.g., in our case it was 795588), then
exceeding it is a symptom of not properly load-balancing your traffic; with that many
connections, you'd better have a ton of free RAM available.
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35@gmail.com<mailto:nishantkumar35@gmail.com>
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
1755
days inactive
1771
days old
20 comments
9 participants
participants (9)
-
David Lloyd -
Flavia Rainone -
James Howe -
Jonas Zeiger -
Nishant Kumar -
openindiana -
Richard Opalka -
Stan Rosenberg -
Stuart Douglas