[undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)

Sat Feb 29 12:34:57 EST 2020

Are your sure it is only XNIO related?

https://security.stackexchange.com/questions/43205/nf-conntrack-table-full-dropping-packet

Look sto me that there's a problem with your systems netfilter, which 
can result in hanging sockets (and not closing files) in XNIO as a result.

On 29-2-2020 18:25, Carter Kozak wrote:
> Have you tried using the latest xnio release with the new queued 
> acceptor? I’ve been using it in production successfully for a while.
> https://github.com/xnio/xnio/pull/206#issuecomment-537597439
>
> -ck
>
>> On Feb 28, 2020, at 11:08 PM, Nishant Kumar 
>> <nishantkumar35 at gmail.com> wrote:
>>
>> 
>> Hi,
>>
>> I am using undertow-core (2.0.29.Final) with Java 11. Every once in a 
>> while (1-2 days) I got Too many open files error on my high load 
>> server (30k JSON Request/sec) and then it stops responding. I have 
>> noticed that it happens during full GC (but not during all full GC). 
>> Could you please help me to fix this? I have also noticed that 
>> Undertow is using a very old version of XNIO (3.3.8.Final). Any 
>> particular reason for that?
>>
>> *System info:*
>> *
>> RAM:* 64 GB
>> *Core:* 24 with hyperthreading (2 Thread per core)
>> *Java version: *11
>> *VM option: *-server -Xms12g -Xmx16g -verbose:gc 
>> -Xloggc:/home/platform/platform-java/logs/platform-gc.log 
>> -Dorg.wildfly.openssl.path=/usr/local/ssl/lib 
>> -Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml
>>
>> *$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print 
>> $6}'| sort | uniq -c*
>>    7916 ESTABLISHED
>>       2 LISTEN
>> *$] ulimit -a*
>> core file size          (blocks, -c) 0
>> data seg size           (kbytes, -d) unlimited
>> scheduling priority             (-e) 0
>> file size               (blocks, -f) unlimited
>> pending signals                 (-i) 256172
>> max locked memory       (kbytes, -l) 64
>> max memory size         (kbytes, -m) unlimited
>> open files                      (-n) 800000
>> pipe size            (512 bytes, -p) 8
>> POSIX message queues     (bytes, -q) 819200
>> real-time priority              (-r) 0
>> stack size              (kbytes, -s) 15360
>> cpu time               (seconds, -t) unlimited
>> max user processes              (-u) 65535
>> virtual memory          (kbytes, -v) unlimited
>> file locks                      (-x) unlimited
>> *$] openssl version*
>> OpenSSL 1.1.1d  10 Sep 2019
>> /*$] cat /etc/redhat-release*/
>> CentOS Linux release 7.6.1810 (Core)
>> /*$] uname -a*/
>> Linux web2.ams.bidstalk.com <http://web2.ams.bidstalk.com> 
>> 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31 11:04:58 EDT 2019 x86_64 
>> x86_64 x86_64 GNU/Linux
>>
>> *Undertow code:*
>> *
>> *
>> server = Undertow.builder()
>>           .addHttpListener(80, "0.0.0.0")
>>           .addHttpsListener(443, "0.0.0.0", sslContext)
>>           .setWorkerThreads(32)
>>           .setServerOption(UndertowOptions.ENABLE_HTTP2, true)
>>           .setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 
>> 12000ms
>> .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE, 1024 
>> * 20) // 20000 sessions
>> .setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT, 3000) // 5m
>>           .setIoThreads(20)
>>           .setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
>>           .setSocketOption(org.xnio.Options.TCP_NODELAY, true)
>>           .setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
>> .setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
>> .setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000)
>> .setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000)
>> .setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false)
>>           .setHandler(Handlers.routing().post("/", new 
>> RequestHandler(appContext)))
>>           .build();
>>       server.start();*
>> *
>> *error.log*
>> 2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept] 
>> org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception 
>> accepting request, closing server channel TCP server (NIO) 
>> <910f2428>java.io  <http://java.io/>.IOException: Too many open filesat 
>> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]at 
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) 
>> ~[?:?]at 
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) 
>> ~[?:?]at 
>> org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) 
>> [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at 
>> org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) 
>> [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at 
>> org.xnio.nio.WorkerThread.run(WorkerThread.java:561) 
>> [xnio-nio-3.3.8.Final.jar:3.3.8.Final]2020-02-28 23:19:06.201 [ERROR] 
>> [XNIO-1 Accept] 
>> org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception 
>> accepting request, closing server channel TCP server (NIO) 
>> <9834817a>java.io  <http://java.io/>.IOException: Too many open filesat 
>> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]at 
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) 
>> ~[?:?]at 
>> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) 
>> ~[?:?]at 
>> org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467) 
>> [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at 
>> org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38) 
>> [xnio-nio-3.3.8.Final.jar:3.3.8.Final]at 
>> org.xnio.nio.WorkerThread.run(WorkerThread.java:561) 
>> [xnio-nio-3.3.8.Final.jar:3.3.8.Final] */var/log/messages :*
>> Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 
>> kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 
>> 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 
>> kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 
>> 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:47 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:47 web2 
>> kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 
>> 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 
>> callbacks suppressed Feb 28 23:18:52 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 
>> kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 
>> 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 
>> kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 
>> 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:52 web2 
>> kernel: nf_conntrack: nf_conntrack: table full, dropping packet Feb 
>> 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:52 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet Feb 28 23:18:57 web2 
>> kernel: net_ratelimit: 60720 callbacks suppressed Feb 28 23:18:57 
>> web2 kernel: nf_conntrack: nf_conntrack: table full, dropping packet 
>> Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full, 
>> dropping packet Feb 28 23:18:57 web2 kernel: nf_conntrack: 
>> nf_conntrack: table full, dropping packet .. ..
>>
>> -- 
>> Nishant Kumar
>> Bangalore, India
>> Mob: +91 80088 42030
>> Email: nishantkumar35 at gmail.com <mailto:nishantkumar35 at gmail.com>
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20200229/d7234a8b/attachment-0001.html 