<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><font size="+1"><font face="Calibri">Are your sure it is only
XNIO related? <br>
</font></font></p>
<p><a
href="https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet">https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet</a></p>
<p>Look sto me that there's a problem with your systems netfilter,
which can result in hanging sockets (and not closing files) in
XNIO as a result. <br>
</p>
<div class="moz-cite-prefix">On 29-2-2020 18:25, Carter Kozak wrote:<br>
</div>
<blockquote type="cite"
cite="mid:61421C04-E5CA-439B-8B8B-9F101B6F6D42@ckozak.net">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div>Have you tried using the latest xnio release with the new
queued acceptor? I’ve been using it in production successfully
for a while.</div>
<a
href="https://github.com/xnio/xnio/pull/206#issuecomment-537597439"
moz-do-not-send="true">https://github.com/xnio/xnio/pull/206#issuecomment-537597439</a><br>
<br>
<div dir="ltr">-ck</div>
<div dir="ltr"><br>
<blockquote type="cite">On Feb 28, 2020, at 11:08 PM, Nishant
Kumar <a class="moz-txt-link-rfc2396E" href="mailto:nishantkumar35@gmail.com"><nishantkumar35@gmail.com></a> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I am using <span
style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">undertow-core
(</span><span
style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">2.0.29.Final) </span>with
Java 11. Every once in a while (1-2 days) I got <span style="color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap">Too many open files</span> error
on my high load server (30k JSON Request/sec) and then it
stops responding. I have noticed that it happens during
full GC (but not during all full GC). Could you please
help me to fix this? I have also noticed that Undertow is
using a very old version of XNIO (<span
style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">3.3.8.Final).
Any particular reason for that?</span></div>
<div><br>
<b>System info:</b></div>
<div><b><br>
RAM:</b> 64 GB<br>
<b>Core:</b> 24 with hyperthreading (2 Thread per core)<br>
<b>Java version: </b>11<br>
<b>VM option: </b>-server -Xms12g -Xmx16g -verbose:gc
-Xloggc:/home/platform/platform-java/logs/platform-gc.log
-Dorg.wildfly.openssl.path=/usr/local/ssl/lib
-Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml<br>
<br>
<b>$] netstat -nalp | grep java | grep -E ":80 |:443 " |
awk '{print $6}'| sort | uniq -c</b></div>
7916 ESTABLISHED<br>
<div> 2 LISTEN <br>
<b>$] ulimit -a</b><br>
core file size (blocks, -c) 0<br>
data seg size (kbytes, -d) unlimited<br>
scheduling priority (-e) 0<br>
file size (blocks, -f) unlimited<br>
pending signals (-i) 256172<br>
max locked memory (kbytes, -l) 64<br>
max memory size (kbytes, -m) unlimited<br>
open files (-n) 800000<br>
pipe size (512 bytes, -p) 8<br>
POSIX message queues (bytes, -q) 819200<br>
real-time priority (-r) 0<br>
stack size (kbytes, -s) 15360<br>
cpu time (seconds, -t) unlimited<br>
max user processes (-u) 65535<br>
virtual memory (kbytes, -v) unlimited<br>
file locks (-x) unlimited<br>
<b>$] openssl version</b><br>
OpenSSL 1.1.1d 10 Sep 2019<br>
<i><b>$] cat /etc/redhat-release</b></i><br>
CentOS Linux release 7.6.1810 (Core)<br>
<i><b>$] uname -a</b></i><br>
Linux <a href="http://web2.ams.bidstalk.com"
target="_blank" moz-do-not-send="true">web2.ams.bidstalk.com</a>
5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31 11:04:58 EDT
2019 x86_64 x86_64 x86_64 GNU/Linux<br>
<br>
<b>Undertow code:</b></div>
<div><b><br>
</b></div>
<div><font face="monospace">server = Undertow.builder()<br>
.addHttpListener(80, "0.0.0.0")<br>
.addHttpsListener(443, "0.0.0.0", sslContext)<br>
.setWorkerThreads(32)<br>
.setServerOption(UndertowOptions.ENABLE_HTTP2,
true)<br>
.setServerOption(UndertowOptions.IDLE_TIMEOUT,
120000) // 12000ms<br>
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE,
1024 * 20) // 20000 sessions<br>
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT,
3000) // 5m<br>
.setIoThreads(20)<br>
.setWorkerOption(org.xnio.Options.TCP_NODELAY,
true)<br>
.setSocketOption(org.xnio.Options.TCP_NODELAY,
true)<br>
.setSocketOption(org.xnio.Options.KEEP_ALIVE,
true)<br>
.setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)<br>
.setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER,
20000)<br>
.setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER,
20000)<br>
.setWorkerOption(org.xnio.Options.THREAD_AFFINITY,
false)<br>
.setHandler(Handlers.routing().post("/", new
RequestHandler(appContext)))<br>
.build();<br>
server.start();</font><b><br>
</b><br>
</div>
<div><b>error.log</b></div>
<div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;word-break:normal;border-radius:4px"><font face="Monaco, Menlo, Consolas, Courier New, monospace" color="#1d1c1d"><span style="font-variant-ligatures:none;white-space:pre-wrap">2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <910f2428></span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span></font><a href="http://java.io/" rel="noopener noreferrer" style="color:inherit;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit;text-decoration-line:none" target="_blank" moz-do-not-send="true">java.io</a><font face="Monaco, Menlo, Consolas, Courier New, monospace" color="#1d1c1d"><span style="font-variant-ligatures:none;white-space:pre-wrap">.IOException: Too many open files</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap">2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <9834817a></span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span></font><a href="http://java.io/" rel="noopener noreferrer" style="color:inherit;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit;text-decoration-line:none" target="_blank" moz-do-not-send="true">java.io</a><font face="Monaco, Menlo, Consolas, Courier New, monospace" color="#1d1c1d"><span style="font-variant-ligatures:none;white-space:pre-wrap">.IOException: Too many open files</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit">
</span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
</span></font><font face="Monaco, Menlo, Consolas, Courier New, monospace" color="#1d1c1d"><span style="font-variant-ligatures:none;white-space:pre-wrap">
</span></font><font face="arial, sans-serif"><b>/var/log/messages :</b></font>
</pre>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;word-break:normal;border-radius:4px"><font face="Monaco, Menlo, Consolas, Courier New, monospace" color="#1d1c1d"><span style="font-variant-ligatures:none;white-space:pre-wrap">Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
..
..</span></font></pre>
</div>
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Nishant Kumar<br>
Bangalore, India<br>
Mob: +91 80088 42030<br>
Email: <a href="mailto:nishantkumar35@gmail.com"
target="_blank" moz-do-not-send="true">nishantkumar35@gmail.com</a></div>
</div>
</div>
<span>_______________________________________________</span><br>
<span>undertow-dev mailing list</span><br>
<span><a class="moz-txt-link-abbreviated" href="mailto:undertow-dev@lists.jboss.org">undertow-dev@lists.jboss.org</a></span><br>
<span><a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/undertow-dev">https://lists.jboss.org/mailman/listinfo/undertow-dev</a></span></div>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
undertow-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:undertow-dev@lists.jboss.org">undertow-dev@lists.jboss.org</a>
<a class="moz-txt-link-freetext" href="https://lists.jboss.org/mailman/listinfo/undertow-dev">https://lists.jboss.org/mailman/listinfo/undertow-dev</a></pre>
</blockquote>
</body>
</html>