Worker queue timeout
by James Howe
Under load, when the worker threads are saturated, new requests will be queued.
Is there a way to timeout these requests, so that it doesn't waste time serving them later when the client has already given up?
I saw a reference to RequestLimitingHandler having a configurable timeout, but that doesn't appear to be true.
I also don't want any limit to how many request that can be waiting, only how long they can wait.
This doesn't seem to be a feature offered by any of the standard Java ExecutorService implementations,
but perhaps Undertow has added this (or an appropriate extension point for this) somewhere?
Thanks
James
1 year, 5 months
Too many open files: Exception accepting request, closing server channel TCP server (NIO)
by Nishant Kumar
Hi,
I am using undertow-core (2.0.29.Final) with Java 11. Every once in a while
(1-2 days) I got Too many open files error on my high load server (30k
JSON Request/sec) and then it stops responding. I have noticed that it
happens during full GC (but not during all full GC). Could you please help
me to fix this? I have also noticed that Undertow is using a very old
version of XNIO (3.3.8.Final). Any particular reason for that?
*System info:*
*RAM:* 64 GB
*Core:* 24 with hyperthreading (2 Thread per core)
*Java version: *11
*VM option: *-server -Xms12g -Xmx16g -verbose:gc
-Xloggc:/home/platform/platform-java/logs/platform-gc.log
-Dorg.wildfly.openssl.path=/usr/local/ssl/lib
-Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml
*$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print $6}'|
sort | uniq -c*
7916 ESTABLISHED
2 LISTEN
*$] ulimit -a*
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256172
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 800000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 15360
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*$] openssl version*
OpenSSL 1.1.1d 10 Sep 2019
*$] cat /etc/redhat-release*
CentOS Linux release 7.6.1810 (Core)
*$] uname -a*
Linux web2.ams.bidstalk.com 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31
11:04:58 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
*Undertow code:*
server = Undertow.builder()
.addHttpListener(80, "0.0.0.0")
.addHttpsListener(443, "0.0.0.0", sslContext)
.setWorkerThreads(32)
.setServerOption(UndertowOptions.ENABLE_HTTP2, true)
.setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 12000ms
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE,
1024 * 20) // 20000 sessions
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT,
3000) // 5m
.setIoThreads(20)
.setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
.setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
.setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000)
.setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000)
.setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false)
.setHandler(Handlers.routing().post("/", new
RequestHandler(appContext)))
.build();
server.start();
*error.log*
2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept]
org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception
accepting request, closing server channel TCP server (NIO) <910f2428>
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
~[?:?]
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept]
org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception
accepting request, closing server channel TCP server (NIO) <9834817a>
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
~[?:?]
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]*/var/log/messages :*
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
..
..
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
1 year, 5 months
