Worker queue timeout
by James Howe
Under load, when the worker threads are saturated, new requests will be queued.
Is there a way to timeout these requests, so that it doesn't waste time serving them later when the client has already given up?
I saw a reference to RequestLimitingHandler having a configurable timeout, but that doesn't appear to be true.
I also don't want any limit to how many request that can be waiting, only how long they can wait.
This doesn't seem to be a feature offered by any of the standard Java ExecutorService implementations,
but perhaps Undertow has added this (or an appropriate extension point for this) somewhere?
Thanks
James
12 months
Too many open files: Exception accepting request, closing server channel TCP server (NIO)
by Nishant Kumar
Hi,
I am using undertow-core (2.0.29.Final) with Java 11. Every once in a while
(1-2 days) I got Too many open files error on my high load server (30k
JSON Request/sec) and then it stops responding. I have noticed that it
happens during full GC (but not during all full GC). Could you please help
me to fix this? I have also noticed that Undertow is using a very old
version of XNIO (3.3.8.Final). Any particular reason for that?
*System info:*
*RAM:* 64 GB
*Core:* 24 with hyperthreading (2 Thread per core)
*Java version: *11
*VM option: *-server -Xms12g -Xmx16g -verbose:gc
-Xloggc:/home/platform/platform-java/logs/platform-gc.log
-Dorg.wildfly.openssl.path=/usr/local/ssl/lib
-Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml
*$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print $6}'|
sort | uniq -c*
7916 ESTABLISHED
2 LISTEN
*$] ulimit -a*
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256172
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 800000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 15360
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*$] openssl version*
OpenSSL 1.1.1d 10 Sep 2019
*$] cat /etc/redhat-release*
CentOS Linux release 7.6.1810 (Core)
*$] uname -a*
Linux web2.ams.bidstalk.com 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31
11:04:58 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
*Undertow code:*
server = Undertow.builder()
.addHttpListener(80, "0.0.0.0")
.addHttpsListener(443, "0.0.0.0", sslContext)
.setWorkerThreads(32)
.setServerOption(UndertowOptions.ENABLE_HTTP2, true)
.setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 12000ms
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE,
1024 * 20) // 20000 sessions
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT,
3000) // 5m
.setIoThreads(20)
.setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
.setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
.setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000)
.setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000)
.setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false)
.setHandler(Handlers.routing().post("/", new
RequestHandler(appContext)))
.build();
server.start();
*error.log*
2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept]
org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception
accepting request, closing server channel TCP server (NIO) <910f2428>
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
~[?:?]
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept]
org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception
accepting request, closing server channel TCP server (NIO) <9834817a>
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
~[?:?]
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]*/var/log/messages :*
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
..
..
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
12 months
