Have you tried using the latest xnio release with the new queued acceptor? I’ve been using it in production successfully for a while. https://github.com/xnio/xnio/pull/206#issuecomment-537597439 -ck > On Feb 28, 2020, at 11:08 PM, Nishant Kumar > <nishantkumar35(a)gmail.com&gt; wrote: > >  > Hi, > > I am using undertow-core (2.0.29.Final) with Java 11. Every once in a > while (1-2 days) I got Too many open files error on my high load > server (30k JSON Request/sec) and then it stops responding. I have > noticed that it happens during full GC (but not during all full GC). > Could you please help me to fix this? I have also noticed that > Undertow is using a very old version of XNIO (3.3.8.Final). Any > particular reason for that? > > *System info:* > * > RAM:* 64 GB > *Core:* 24 with hyperthreading (2 Thread per core) > *Java version: *11 > *VM option: *-server -Xms12g -Xmx16g -verbose:gc > -Xloggc:/home/platform/platform-java/logs/platform-gc.log > -Dorg.wildfly.openssl.path=/usr/local/ssl/lib > -Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml > > *$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print > $6}'| sort | uniq -c* >    7916 ESTABLISHED >       2 LISTEN > *$] ulimit -a* > core file size          (blocks, -c) 0 > data seg size           (kbytes, -d) unlimited > scheduling priority             (-e) 0 > file size               (blocks, -f) unlimited > pending signals                 (-i) 256172 > max locked memory       (kbytes, -l) 64 > max memory size         (kbytes, -m) unlimited > open files                      (-n) 800000 > pipe size            (512 bytes, -p) 8 > POSIX message queues     (bytes, -q) 819200 > real-time priority              (-r) 0 > stack size              (kbytes, -s) 15360 > cpu time               (seconds, -t) unlimited > max user processes              (-u) 65535 > virtual memory          (kbytes, -v) unlimited > file locks                      (-x) unlimited > *$] openssl version* > OpenSSL 1.1.1d  10 Sep 2019 > /*$] cat /etc/redhat-release*/ > CentOS Linux release 7.6.1810 (Core) > /*$] uname -a*/ > Linux web2.ams.bidstalk.com <http://web2.ams.bidstalk.com> > 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31 11:04:58 EDT 2019 x86_64 > x86_64 x86_64 GNU/Linux > > *Undertow code:* > * > * > server = Undertow.builder() >           .addHttpListener(80, "0.0.0.0") >           .addHttpsListener(443, "0.0.0.0", sslContext) >           .setWorkerThreads(32) >           .setServerOption(UndertowOptions.ENABLE_HTTP2, true) >           .setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // > 12000ms > .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024 > * 20) // 20000 sessions > .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 3000) // 5m >           .setIoThreads(20) >           .setWorkerOption(org.xnio.Options.TCP_NODELAY, true) >           .setSocketOption(org.xnio.Options.TCP_NODELAY, true) >           .setSocketOption(org.xnio.Options.KEEP_ALIVE, true) > .setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true) > .setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000) > .setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000) > .setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false) >           .setHandler(Handlers.routing().post("/", new > RequestHandler(appContext))) >           .build(); >       server.start();* > * > *error.log* > 2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept] > org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception > accepting request, closing server channel TCP server (NIO) > <910f2428>java.io <http://java.io/>.IOException: Too many open filesat > sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) > ~[?:?]at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) > ~[?:?]at > org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) > [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at > org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) > [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at > org.xnio.nio.WorkerThread.run(WorkerThread.java:561) > [xnio-nio-3.3.8.Final.jar:3.3.8.Final]2020-02-28 23:19:06.201 [ERROR] > [XNIO-1 Accept] > org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception > accepting request, closing server channel TCP server (NIO) > <9834817a>java.io <http://java.io/>.IOException: Too many open filesat > sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) > ~[?:?]at > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) > ~[?:?]at > org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) > [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at > org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) > [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at > org.xnio.nio.WorkerThread.run(WorkerThread.java:561) > [xnio-nio-3.3.8.Final.jar:3.3.8.Final] */var/log/messages :* > Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 > kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb > 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 > kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb > 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 > kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb > 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 > callbacks suppressed Feb 28 23:18:52 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 > kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb > 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 > kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb > 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 > kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb > 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet Feb 28 23:18:57 web2 > kernel: net_ratelimit: 60720 callbacks suppressed Feb 28 23:18:57 > web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet > Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, > dropping packet Feb 28 23:18:57 web2 kernel: nf_conntrack: > nf_conntrack: table full, dropping packet .. .. > > -- > Nishant Kumar > Bangalore, India > Mob: +91 80088 42030 > Email: nishantkumar35(a)gmail.com <mailto:nishantkumar35@gmail.com> > _______________________________________________ > undertow-dev mailing list > undertow-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/undertow-dev _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

On Fri, Feb 28, 2020 at 11:08 PM Nishant Kumar <nishantkumar35(a)gmail.com&gt; wrote: > I am using undertow-core (2.0.29.Final) with Java 11. Every once in a > while (1-2 days) I got Too many open files error on my high load server > (30k JSON Request/sec) and then it stops responding. I have noticed that it > happens during full GC (but not during all full GC). Could you please help > me to fix this? > The "too many open files" error is invariably related to exceeding the max. open file descriptors as enforced by the OS. Note that if you're using systemd to start undertow, then you have to specify LimitNOFILE (it ignores /etc/security/limits*); see [1] for more details. If you're not running systemd, then I'd double-check the file descriptor limit for the user running the undertow process. As to why this happens once in a while, my best guess is either or both (a) there is a sudden, short-lived spike in new connections, (b) many connections are kept alive for over 24 hours. We've had similar issues while running undertow in high-throughput, low-latency environment (in front of GLB). Even with the unreasonably high file descriptor limit (LimitNOFILE=795588), we've seen traffic bursts that would bring down undertow by triggering (linux) OOM killer (with that many connections, non-heap memory is exhausted). [1] https://serverfault.com/questions/628610/increasing-nproc-for-processes-l... Best, stan