[undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

Carter Kozak ckozak at ckozak.net
Sat Feb 29 12:25:38 EST 2020


Have you tried using the latest xnio release with the new queued acceptor? I’ve been using it in production successfully for a while.
https://github.com/xnio/xnio/pull/206#issuecomment-537597439

-ck

> On Feb 28, 2020, at 11:08 PM, Nishant Kumar <nishantkumar35 at gmail.com> wrote:
> 
> 
> Hi,
> 
> I am using undertow-core (2.0.29.Final) with Java 11. Every once in a while (1-2 days) I got  Too many open files error on my high load server (30k JSON Request/sec) and then it stops responding. I have noticed that it happens during full GC (but not during all full GC). Could you please help me to fix this? I have also noticed that Undertow is using a very old version of XNIO (3.3.8.Final). Any particular reason for that?
> 
> System info:
> 
> RAM: 64 GB
> Core: 24 with hyperthreading (2 Thread per core)
> Java version: 11
> VM option: -server -Xms12g -Xmx16g -verbose:gc -Xloggc:/home/platform/platform-java/logs/platform-gc.log -Dorg.wildfly.openssl.path=/usr/local/ssl/lib -Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml
> 
> $] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print $6}'| sort | uniq -c
>    7916 ESTABLISHED
>       2 LISTEN 
> $] ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 256172
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 800000
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 15360
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 65535
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> $] openssl version
> OpenSSL 1.1.1d  10 Sep 2019
> $] cat /etc/redhat-release
> CentOS Linux release 7.6.1810 (Core)
> $] uname -a
> Linux web2.ams.bidstalk.com 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31 11:04:58 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
> 
> Undertow code:
> 
> server = Undertow.builder()
>           .addHttpListener(80, "0.0.0.0")
>           .addHttpsListener(443, "0.0.0.0", sslContext)
>           .setWorkerThreads(32)
>           .setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>           .setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 12000ms
>           .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024 * 20) // 20000 sessions
>           .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 3000) // 5m
>           .setIoThreads(20)
>           .setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
>           .setSocketOption(org.xnio.Options.TCP_NODELAY, true)
>           .setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
>           .setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
>           .setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000)
>           .setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000)
>           .setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false)
>           .setHandler(Handlers.routing().post("/", new RequestHandler(appContext)))
>           .build();
>       server.start();
> 
> error.log
> 2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <910f2428>
> java.io.IOException: Too many open files
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]
>         at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
>         at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
>         at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
> 2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept] org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception accepting request, closing server channel TCP server (NIO) <9834817a>
> java.io.IOException: Too many open files
>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) ~[?:?]
>         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) ~[?:?]
>         at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
>         at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
>         at org.xnio.nio.WorkerThread.run(WorkerThread.java:561) [xnio-nio-3.3.8.Final.jar:3.3.8.Final]
> 
> /var/log/messages :
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed
> Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet
> ..
> ..
> 
> -- 
> Nishant Kumar
> Bangalore, India
> Mob: +91 80088 42030
> Email: nishantkumar35 at gmail.com
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20200229/88a20c45/attachment.html 


More information about the undertow-dev mailing list