<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><font size="+1"><font face="Calibri">have you check the logs
again? <br>
</font></font></p>
<p><font size="+1"><font face="Calibri">Your log had these entries:</font></font></p>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;word-break:normal;border-radius:4px"><font face="arial, sans-serif"><b>/var/log/messages :</b></font>
</pre>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;word-break:normal;border-radius:4px"><font face="Monaco, Menlo, Consolas, Courier New, monospace" color="#1d1c1d"><span style="font-variant-ligatures:none;white-space:pre-wrap">Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet</span></font></pre>
<p><font size="+1"><font face="Calibri"></font></font>What is the
output of <br>
</p>
<pre style="margin: 0px 0px 1em; padding: 12px 8px; border: 0px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-numeric: inherit; font-variant-east-asian: inherit; font-weight: 400; font-stretch: inherit; line-height: inherit; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; vertical-align: baseline; box-sizing: inherit; width: auto; max-height: 600px; overflow: auto; background-color: var(--black-050); border-radius: 3px; overflow-wrap: normal; color: rgb(36, 39, 41); letter-spacing: normal; orphans: 2; text-align: left; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"><code style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; vertical-align: baseline; box-sizing: inherit; background-color: var(--black-050); white-space: inherit;">sysctl -a | grep conntrack | grep timeout
</code><code style="margin: 0px; padding: 0px; border: 0px; font-style: inherit; font-variant: inherit; font-weight: inherit; font-stretch: inherit; line-height: inherit; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; font-size: 13px; vertical-align: baseline; box-sizing: inherit; background-color: var(--black-050); white-space: inherit;">
Please read: </code><a href="https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet">https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet</a>
</pre>
<code style="margin: 0px; padding: 0px; border: 0px; font-style:
inherit; font-variant: inherit; font-weight: inherit;
font-stretch: inherit; line-height: inherit; font-family:
Consolas, Menlo, Monaco, "Lucida Console",
"Liberation Mono", "DejaVu Sans Mono",
"Bitstream Vera Sans Mono", "Courier New",
monospace, sans-serif; font-size: 13px; vertical-align: baseline;
box-sizing: inherit; background-color: var(--black-050);
white-space: inherit;"></code>
<div class="moz-cite-prefix">On 2-3-2020 09:59, Nishant Kumar wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMDUb2CcVXBursznTZCo2auUCp=SRsNa_mpJgxhCAOk3UKLe5w@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Generally, clients also close the connection after
a few thousand requests other than normal fatal conditions.
There might be other cases too but I am not aware of it. They
keep initiating new connections if we are not responding within
the threshold time frame. This is a server to server
communication system. </div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Mar 2, 2020 at 10:26
AM Stuart Douglas <<a href="mailto:sdouglas@redhat.com"
moz-do-not-send="true">sdouglas@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>This sounds like a bug, when the client closes the
connection it should wake up the read listener, which will
read -1 and then cleanly close the socket.</div>
<div><br>
</div>
<div>Are the clients closing idle connections or connections
processing a request?</div>
<div><br>
</div>
<div>Stuart</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, 2 Mar 2020 at
14:31, Nishant Kumar <<a
href="mailto:nishantkumar35@gmail.com" target="_blank"
moz-do-not-send="true">nishantkumar35@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">I agree that it's a load-balancing issue
but we can't do much about it at this moment.<br>
<br>
I still see issues after using the latest XNIO (3.7.7)
with Undertow. what I have observed it that when there
is a spike in request and CONNECTION_HIGH_WATER is
reached, the server stops accepting new connection as
expected and the client starts to close the connection
because of delay (we have strict low latency
requirement < 100ms) and try to create new
connection again (which will also not be accepted) but
server has not closed those connections
(NO_REQUEST_TIMEOUT = 6000) and there will be high
number of CLOSE_WAIT connections at this moment. The
server is considering CLOSE_WAIT + ESTABLISHED
for CONNECTION_HIGH_WATER (my understanding).
<div><br>
</div>
<div>Is there a way that I can close all CLOSE_WAIT
connection at this moment so that connection counts
drop under CONNECTION_HIGH_WATER and we start
responding to newly established connections? or any
other suggestions? I have tried
removing CONNECTION_HIGH_WATER and relying on the FD
limit but that didn't work.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Mar 1, 2020
at 7:47 AM Stan Rosenberg <<a
href="mailto:stan.rosenberg@gmail.com"
target="_blank" moz-do-not-send="true">stan.rosenberg@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">
<div style="font-size:small">On Sat, Feb 29,
2020 at 8:18 PM Nishant Kumar <<a
href="mailto:nishantkumar35@gmail.com"
target="_blank" moz-do-not-send="true">nishantkumar35@gmail.com</a>>
wrote:<br>
</div>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="auto">Thanks for the reply. I am
running it under supervisord and i have
updated open file limit in supervisord
config. The problem seems to be same as what
@Carter has mentioned. It happens mostly
during sudden traffic spike and then sudden
increase (~30k-300k) of TIME_WAIT socket. </div>
</blockquote>
<div><br>
</div>
<div style="font-size:small">The changes in <a
href="https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11bf9d214bR453"
target="_blank" moz-do-not-send="true">https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11bf9d214bR453</a> are
likely to improve the exceptional case of
exceeding the file descriptor limit. However,
if you're already setting the limit too high
(e.g., in our case it was 795588), then
exceeding it is a symptom of not properly
load-balancing your traffic; with that many
connections, you'd better have a ton of free
RAM available. </div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">Nishant Kumar<br>
Bangalore, India<br>
Mob: +91 80088 42030<br>
Email: <a href="mailto:nishantkumar35@gmail.com"
target="_blank" moz-do-not-send="true">nishantkumar35@gmail.com</a></div>
</div>
_______________________________________________<br>
undertow-dev mailing list<br>
<a href="mailto:undertow-dev@lists.jboss.org"
target="_blank" moz-do-not-send="true">undertow-dev@lists.jboss.org</a><br>
<a
href="https://lists.jboss.org/mailman/listinfo/undertow-dev"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.jboss.org/mailman/listinfo/undertow-dev</a></blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">Nishant Kumar<br>
Bangalore, India<br>
Mob: +91 80088 42030<br>
Email: <a href="mailto:nishantkumar35@gmail.com"
target="_blank" moz-do-not-send="true">nishantkumar35@gmail.com</a></div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
undertow-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:undertow-dev@lists.jboss.org">undertow-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/undertow-dev">https://lists.jboss.org/mailman/listinfo/undertow-dev</a></pre>
</blockquote>
</body>
</html>