Undertow proxy handler + RequestLimit handler issue
by paroczizs
Hi Undertow Dev,
we built a custom integration component based on Wildfly 11 with Undertow
1.4.18. The application contains a couple of custom and built in handlers.
We found the following:
There is a RequestLimit handler right at the beginning and finally the
proxy handler. When the backend getting to be slow and the load is
relatively high the RequestLimit handler does its job queues the request
then throws the request above the queue. When the proxy times out it starts
to consume the queue, in some cases the exception below occurs and we found
that in this case the counter in the RequestLimit handler is not reduced
and the it can stuck the way to the backend.
Are there any possibility to handle the exception in a custom handler?
Thank you in advance, Zsolt
2018-11-28 12:02:19,518 ERROR [io.undertow.proxy] (default I/O-2) UT005028:
Proxy request to /lwi/optinus/getentityanswersservice failed:
java.nio.channels.ClosedChannelException
at
io.undertow.client.http.HttpClientConnection$5.handleEvent(HttpClientConnection.java:194)
at
io.undertow.client.http.HttpClientConnection$5.handleEvent(HttpClientConnection.java:173)
at
org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at
org.xnio.StreamConnection.invokeCloseListener(StreamConnection.java:80)
at org.xnio.Connection.close(Connection.java:142)
at org.xnio.IoUtils.safeClose(IoUtils.java:152)
at io.undertow.util.ConnectionUtils$3.run(ConnectionUtils.java:102)
at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
2 months, 2 weeks
Creating a websocket proxy with undertow
by Marc Boorshtein
Using the 1.4 series, I'm trying to handle upgrade and websockets. Looking
at a few examples and the code I think what I need to do is:
1. add the WebSocketServlet to the right path
2. implement WebSocketConnectionCallback that will call my remote ws://
server
Am I on the right track? I haven't found an example of doing this with
undertow but found something similar with jetty (
https://github.com/chipster/chipster-web-server/blob/master/src/main/java...)
but I havne't found the same thing for undertow so any pointers would be
appreciated.
Thanks
Marc Boorshtein
3 months, 2 weeks
Issue with Undertow Client and SSL
by Thomas Segismont
Hi,
Openshift Origin uses Hawkular Metrics to store node and container data. In
this scenario, Hawkular Metrics calls the Kubernetes master server over
HTTPS to validate the client identity. This is implemented with Undertow
Client, as part of a ServletExtension (inside Wildfly 10) [1]. Works fine
in development and testing.
Now the Openshit team sees errors in the logs [2][3]. I couldn't reproduce
yet. Errors come in pair, first the "UT005001: An exception occurred
processing the request: java.lang.IllegalStateException: XNIO000017: Buffer
was already freed", and just after "XNIO001007: A channel event listener
threw an exception: java.lang.NullPointerException".
Does that ring a bell? I haven't been able to find a starting point by
looking into the source the code.
Thanks
--
Thomas Segismont
JBoss ON Engineering Team
[1] https://git.io/vrNyP
[2] https://issues.jboss.org/browse/HWKMETRICS-408
[3] https://issues.jboss.org/secure/attachment/12405779/hawkular.log
4 months, 2 weeks
Correctly shutting down a websocket handler
by Robin Anil
When a client disconnects, I see that onClose is not being fired. The only
way this seems to be firing if client sents a close frame.
Is there any way to detect disconnection and immediately close all the
opened resources.
Robin
Robin Anil | Software Engineer
2 years, 10 months
dynamic URIs and overlapping method requests on URIs
by David Robinson
I believe these two questions are very basic based on the posts I've read
so far on the mailing list. I appreciate your insights into how to handle
these correctly using undertow handlers (was thinking custom handlers).
First topic: a portion of the URI for my app is "dynamic" and changes as my
program runs.
For example, with this URI:
/xd/cohort1/healthevents/examdata
the "cohort1" portion "appears" after my program starts.
Each cohort may have distinct processing - so this URI:
/xd/cohort0/healthevents/examdata
needs to be handled by code that may be entirely different from the first
URI.
In a first approach, a single custom handler for the path /sd/* was
configured:
rh0 = new RoutingHandler().post("/xd/*", new
UndertowEventHttpHandler(config, posthandlers));
Undertow server = Undertow.builder().addHttpListener(ipPort,
ipAddress).setHandler(rh0).build();
That examined the URI, validated the "cohort" piece, and then called a
separate helper class directly (no dispatch, or async) to process the
request.
Is this the best approach or is there a better way to handle this with
Undertow?
(The out-of-the-box Path Template handler, for example, did not seem
dynamic enough to handle this sort of URI.)
Second topic: the application can receive "GETS" and "POSTS" to the same
URI.
A "GET" to /xd/cohort1/healthevents/examdata would return data where a
POST to /xd/cohort1/healthevents/examdata would expect the body of the
message to have JSON to be stored.
How to best handle both GET and POST messages coming to the same URI with
Undertow? I started to add an extra layer of abstraction - a generic
handler that looked at the request method
HttpString hString = exchange.getRequestMethod();
but was not sure where to go from there...do I then call the specialized
POST and GET handlers with .dispatch (deprecated except if I provide an
Executor?) or is there a way to register two handlers for the same
path...just different request methods?
Thank you,
6 years, 3 months
Re: [undertow-dev] Too many open files: Exception accepting request, closing server channel TCP server (NIO)
by Nishant Kumar
I agree that it's a load-balancing issue but we can't do much about it at
this moment.
I still see issues after using the latest XNIO (3.7.7) with Undertow. what
I have observed it that when there is a spike in request
and CONNECTION_HIGH_WATER is reached, the server stops accepting new
connection as expected and the client starts to close the connection
because of delay (we have strict low latency requirement < 100ms) and try
to create new connection again (which will also not be accepted) but server
has not closed those connections (NO_REQUEST_TIMEOUT = 6000) and there will
be high number of CLOSE_WAIT connections at this moment. The server is
considering CLOSE_WAIT + ESTABLISHED for CONNECTION_HIGH_WATER (my
understanding).
Is there a way that I can close all CLOSE_WAIT connection at this moment
so that connection counts drop under CONNECTION_HIGH_WATER and we start
responding to newly established connections? or any other suggestions? I
have tried removing CONNECTION_HIGH_WATER and relying on the FD limit but
that didn't work.
On Sun, Mar 1, 2020 at 7:47 AM Stan Rosenberg <stan.rosenberg(a)gmail.com>
wrote:
> On Sat, Feb 29, 2020 at 8:18 PM Nishant Kumar <nishantkumar35(a)gmail.com>
> wrote:
>
>> Thanks for the reply. I am running it under supervisord and i have
>> updated open file limit in supervisord config. The problem seems to be same
>> as what @Carter has mentioned. It happens mostly during sudden traffic
>> spike and then sudden increase (~30k-300k) of TIME_WAIT socket.
>>
>
> The changes in
> https://github.com/xnio/xnio/pull/206/files#diff-23a6a7997705ea72e4016c11... are
> likely to improve the exceptional case of exceeding the file descriptor
> limit. However, if you're already setting the limit too high (e.g., in our
> case it was 795588), then exceeding it is a symptom of not properly
> load-balancing your traffic; with that many connections, you'd better have
> a ton of free RAM available.
>
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
6 years, 3 months
Proxy and chunking.
by Jocke Eriksson
Hi devs and followers we have a problem and maybe someone on this list have some input.
We use undertow as an API gateway in front of JBoss application servers.
One API we have is responding with a chunked response, and the receiving side is expecting that each chunk is a valid JSon object.
When we analyze the TCP dumps it looks like some chunks are not the full JSon response we are expecting.
So the question is could this be a problem in the gateway?
I found a flag called PRE_CHUNKED_RESPONSE could this be the answer.
If so, how do we apply it in a proxy handler?
Regards Joakim.
6 years, 4 months
Worker queue timeout
by James Howe
Under load, when the worker threads are saturated, new requests will be queued.
Is there a way to timeout these requests, so that it doesn't waste time serving them later when the client has already given up?
I saw a reference to RequestLimitingHandler having a configurable timeout, but that doesn't appear to be true.
I also don't want any limit to how many request that can be waiting, only how long they can wait.
This doesn't seem to be a feature offered by any of the standard Java ExecutorService implementations,
but perhaps Undertow has added this (or an appropriate extension point for this) somewhere?
Thanks
James
6 years, 4 months
Too many open files: Exception accepting request, closing server channel TCP server (NIO)
by Nishant Kumar
Hi,
I am using undertow-core (2.0.29.Final) with Java 11. Every once in a while
(1-2 days) I got Too many open files error on my high load server (30k
JSON Request/sec) and then it stops responding. I have noticed that it
happens during full GC (but not during all full GC). Could you please help
me to fix this? I have also noticed that Undertow is using a very old
version of XNIO (3.3.8.Final). Any particular reason for that?
*System info:*
*RAM:* 64 GB
*Core:* 24 with hyperthreading (2 Thread per core)
*Java version: *11
*VM option: *-server -Xms12g -Xmx16g -verbose:gc
-Xloggc:/home/platform/platform-java/logs/platform-gc.log
-Dorg.wildfly.openssl.path=/usr/local/ssl/lib
-Dlog4j.configurationFile=/home/platform/platform-java/src/main/resources/log4j2.xml
*$] netstat -nalp | grep java | grep -E ":80 |:443 " | awk '{print $6}'|
sort | uniq -c*
7916 ESTABLISHED
2 LISTEN
*$] ulimit -a*
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256172
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 800000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 15360
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*$] openssl version*
OpenSSL 1.1.1d 10 Sep 2019
*$] cat /etc/redhat-release*
CentOS Linux release 7.6.1810 (Core)
*$] uname -a*
Linux web2.ams.bidstalk.com 5.1.6-1.el7.elrepo.x86_64 #1 SMP Fri May 31
11:04:58 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
*Undertow code:*
server = Undertow.builder()
.addHttpListener(80, "0.0.0.0")
.addHttpsListener(443, "0.0.0.0", sslContext)
.setWorkerThreads(32)
.setServerOption(UndertowOptions.ENABLE_HTTP2, true)
.setServerOption(UndertowOptions.IDLE_TIMEOUT, 120000) // 12000ms
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_CACHE_SIZE,
1024 * 20) // 20000 sessions
.setServerOption(org.xnio.Options.SSL_SERVER_SESSION_TIMEOUT,
3000) // 5m
.setIoThreads(20)
.setWorkerOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.TCP_NODELAY, true)
.setSocketOption(org.xnio.Options.KEEP_ALIVE, true)
.setSocketOption(org.xnio.Options.REUSE_ADDRESSES, true)
.setSocketOption(org.xnio.Options.CONNECTION_HIGH_WATER, 20000)
.setSocketOption(org.xnio.Options.CONNECTION_LOW_WATER, 20000)
.setWorkerOption(org.xnio.Options.THREAD_AFFINITY, false)
.setHandler(Handlers.routing().post("/", new
RequestHandler(appContext)))
.build();
server.start();
*error.log*
2020-02-28 23:19:05.997 [ERROR] [XNIO-1 Accept]
org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception
accepting request, closing server channel TCP server (NIO) <910f2428>
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
~[?:?]
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
2020-02-28 23:19:06.201 [ERROR] [XNIO-1 Accept]
org.xnio.nio.QueuedNioTcpServerHandle:handleReady(-1): Exception
accepting request, closing server channel TCP server (NIO) <9834817a>
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
~[?:?]
at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
[xnio-nio-3.3.8.Final.jar:3.3.8.Final]*/var/log/messages :*
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:47 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: net_ratelimit: 21918 callbacks suppressed
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:52 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: net_ratelimit: 60720 callbacks suppressed
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
Feb 28 23:18:57 web2 kernel: nf_conntrack: nf_conntrack: table full,
dropping packet
..
..
--
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkumar35(a)gmail.com
6 years, 4 months
