<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>Have you tried using the latest xnio release with the new queued acceptor? I’ve been using it in production successfully for a while.</div><a href="https://github.com/xnio/xnio/pull/206#issuecomment-537597439">https://github.com/xnio/xnio/pull/206#issuecomment-537597439</a><br><br><div dir="ltr">-ck</div><div dir="ltr"><br><blockquote type="cite">On Feb 28, 2020, at 11:08 PM, Nishant Kumar <nishantkumar35@gmail.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>I am using <span style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">undertow-core (</span><span style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">2.0.29.Final) </span>with Java 11. Every once in a while (1-2 days) I got <span style="color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap">Too many open files</span> error on my high load server (30k JSON Request/sec) and then it stops responding. I have noticed that it happens during full GC (but not during all full GC). Could you please help me to fix this? I have also noticed that Undertow is using a very old version of XNIO (<span style="color:rgb(0,0,0);font-family:Menlo;font-size:9pt">3.3.8.Final). Any particular reason for that?</span></div><div><br><b>System info:</b></div><div><b><br>RAM:</b> 64 GB<br><b>Core:</b> 24 with hyperthreading (2 Thread per core)<br><b>Java version: </b>11<br><b>VM option: </b>-server -Xms12g -Xmx16g -verbose:gc -Xloggc:/home/platform/platform-java/logs/platform-gc.log -Dorg.wildfly.openssl.path=/usr/local/ssl/lib -Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml<br><br><b>$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print $6}'| sort | uniq -c</b></div> 7916 ESTABLISHED<br><div> 2 LISTEN <br><b>$] ulimit -a</b><br>core file size (blocks, -c) 0<br>data seg size (kbytes, -d) unlimited<br>scheduling priority (-e) 0<br>file size (blocks, -f) unlimited<br>pending signals (-i) 256172<br>max locked memory (kbytes, -l) 64<br>max memory size (kbytes, -m) unlimited<br>open files (-n) 800000<br>pipe size (512 bytes, -p) 8<br>POSIX message queues (bytes, -q) 819200<br>real-time priority (-r) 0<br>stack size (kbytes, -s) 15360<br>cpu time (seconds, -t) unlimited<br>max user processes (-u) 65535<br>virtual memory (kbytes, -v) unlimited<br>file locks (-x) unlimited<br><b>$] openssl version</b><br>OpenSSL 1.1.1d 10 Sep 2019<br><i><b>$] cat /etc/redhat-release</b></i><br>CentOS Linux release 7.6.1810 (Core)<br><i><b>$] uname -a</b></i><br>Linux <a href="http://web2.ams.bidstalk.com" target="_blank">web2.ams.bidstalk.com</a> 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31 11:04:58 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux<br><br><b>Undertow code:</b></div><div><b><br></b></div><div><font face="monospace">server = Undertow.builder()<br> .addHttpListener(80, "0.0.0.0")<br> .addHttpsListener(443, "0.0.0.0", sslContext)<br> .setWorkerThreads(32)<br> .setServerOption(UndertowOptions.ENABLE_HTTP2, true)<br> .setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 12000ms<br> .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024 * 20) // 20000 sessions<br> .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 3000) // 5m<br> .setIoThreads(20)<br> .setWorkerOption(org.xnio.Options.TCP_NODELAY, true)<br> .setSocketOption(org.xnio.Options.TCP_NODELAY, true)<br> .setSocketOption(org.xnio.Options.KEEP_ALIVE, true)<br> .setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)<br> .setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000)<br> .setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000)<br> .setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false)<br> .setHandler(Handlers.routing().post("/", new RequestHandler(appContext)))<br> .build();<br> server.start();</font><b><br></b><br></div><div><b>error.log</b></div><div><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;word-break:normal;border-radius:4px"><font color="#1d1c1d" face="Monaco, Menlo, Consolas, Courier New, monospace"><span style="font-variant-ligatures:none;white-space:pre-wrap">2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <910f2428></span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span></font><a href="http://java.io/" rel="noopener noreferrer" style="color:inherit;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit;text-decoration-line:none" target="_blank">java.io</a><font color="#1d1c1d" face="Monaco, Menlo, Consolas, Courier New, monospace"><span style="font-variant-ligatures:none;white-space:pre-wrap">.IOException: Too many open files</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap">2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <9834817a></span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span></font><a href="http://java.io/" rel="noopener noreferrer" style="color:inherit;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit;text-decoration-line:none" target="_blank">java.io</a><font color="#1d1c1d" face="Monaco, Menlo, Consolas, Courier New, monospace"><span style="font-variant-ligatures:none;white-space:pre-wrap">.IOException: Too many open files</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]</span><span style="font-variant-ligatures:none;white-space:pre-wrap;box-sizing:inherit"><br style="box-sizing:inherit"></span><span style="font-variant-ligatures:none;white-space:pre-wrap"> at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
</span></font><font color="#1d1c1d" face="Monaco, Menlo, Consolas, Courier New, monospace"><span style="font-variant-ligatures:none;white-space:pre-wrap">
</span></font><font face="arial, sans-serif"><b>/var/log/messages :</b></font><br></pre><pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;line-height:1.50001;word-break:normal;border-radius:4px"><font color="#1d1c1d" face="Monaco, Menlo, Consolas, Courier New, monospace"><span style="font-variant-ligatures:none;white-space:pre-wrap">Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed<br>Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet<br>..<br>..</span></font></pre></div><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Nishant Kumar<br>Bangalore, India<br>Mob: +91 80088 42030<br>Email: <a href="mailto:nishantkumar35@gmail.com" target="_blank">nishantkumar35@gmail.com</a></div></div></div>
<span>_______________________________________________</span><br><span>undertow-dev mailing list</span><br><span>undertow-dev@lists.jboss.org</span><br><span>https://lists.jboss.org/mailman/listinfo/undertow-dev</span></div></blockquote></body></html>