Hi, I am using undertow-core (2.0.29.Final) with Java 11. Every once in a while (1-2 days) I got Too many open files error on my high load server (30k JSON Request/sec) and then it stops responding. I have noticed that it happens during full GC (but not during all full GC). Could you please help me to fix this? I have also noticed that Undertow is using a very old version of XNIO (3.3.8.Final). Any particular reason for that? *System info:* *RAM:* 64 GB *Core:* 24 with hyperthreading (2 Thread per core) *Java version: *11 *VM option: *-server -Xms12g -Xmx16g -verbose:gc -Xloggc:/home/platform/platform-java/logs/platform-gc.log -Dorg.wildfly.openssl.path=/usr/local/ssl/lib -Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml *$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print $6}'| sort | uniq -c* 7916 ESTABLISHED 2 LISTEN *$] ulimit -a* core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256172 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 800000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 15360 cpu time (seconds, -t) unlimited max user processes (-u) 65535 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited *$] openssl version* OpenSSL 1.1.1d 10 Sep 2019 *$] cat /etc/redhat-release* CentOS Linux release 7.6.1810 (Core) *$] uname -a* Linux web2.ams.bidstalk.com 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31 11:04:58 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux *Undertow code:* server = Undertow.builder() .addHttpListener(80, "0.0.0.0") .addHttpsListener(443, "0.0.0.0", sslContext) .setWorkerThreads(32) .setServerOption(UndertowOptions.ENABLE_HTTP2, true) .setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 12000ms .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024 * 20) // 20000 sessions .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 3000) // 5m .setIoThreads(20) .setWorkerOption(org.xnio.Options.TCP_NODELAY, true) .setSocketOption(org.xnio.Options.TCP_NODELAY, true) .setSocketOption(org.xnio.Options.KEEP_ALIVE, true) .setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true) .setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000) .setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000) .setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false) .setHandler(Handlers.routing().post("/", new RequestHandler(appContext))) .build(); server.start(); *error.log* 2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <910f2428> java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?] at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final] at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final] at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final] 2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <9834817a> java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?] at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final] at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final] at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]*/var/log/messages :* Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet .. .. -- Nishant Kumar Bangalore, India Mob: +91 80088 42030 Email: nishantkumar35(a)gmail.com