9:11 a.m.
I have undertow embedded into my application and for the most part its
working great, but unfortunately I've found a memory leak. The application
its self creates an Undertow server using SSL and then adds servlets and
filters. There are no custom handlers.
Throughout the logs I get the following NPEs:
2016-01-14 09:50:04,681][XNIO-2 task-28] ERROR request - UT005071: Undertow
request failed HttpServerExchange{ GET /wp-content/themes/Nexus/style.css
request {Accept=[text/css, */*], Connection=[Keep-Alive],
Accept-Language=[en-US,en;q=0.5], Accept-Encoding=[gzip, deflate],
Cookie=[nowwwsession=eyJpdiI6Ik5hV0xCWWR5VExzUEEzRTBVNUFKUkFcdTAwM2RcdTAwM2QiLCJlbmNyeXB0ZWRSZXF1ZXN0IjoiS2t0TENaVGZFb1NWNjlyVXB4aUxuQ2RHRmlTZzVDWm0veTFXLytXWlBQMEhpMDZKK3FaQWpQYnBGY0RnMW15RCJ9;
nvers-prod-session=eyJpdiI6Ikd2Wk5oTDVGbjdSTElGejlQSzFsalFcdTAwM2RcdTAwM2QiLCJlbmNyeXB0ZWRSZXF1ZXN0IjoiN2VldzBiT1lIWkZNaVgwTzV5b2tUSVNnaWwzTC9tRHpmeEdDOFZiQzh2a1RZN29ScElqbHFyZys3dmZMWUNhRSJ9],
Referer=[https://www.nvers.org/the-team/], User-Agent=[Mozilla/5.0 (Windows
NT 10.0; Trident/7.0; MALC; rv:11.0) like Gecko], Host=[www.nvers.org]}
response {Connection=[close], Last-Modified=[Thu, 26 Mar 2015 18:45:16
GMT], Server=[Apache/2.2.15 (CentOS)], Content-Type=[text/css],
Content-Language=[en-US], Accept-Ranges=[bytes], Date=[Thu, 14 Jan 2016
14:37:10 GMT]}}
java.lang.NullPointerException
at
io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:713)
at
io.undertow.conduits.FinishableStreamSinkConduit.flush(FinishableStreamSinkConduit.java:83)
at
org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
at
io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)
at org.xnio.channels.Channels.flushBlocking(Channels.java:63)
at
io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:609)
at
io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:476)
at
io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:560)
at
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)
at
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Also, I see the following in System.out:
Exception in thread "XNIO-2 task-18" java.lang.NullPointerException
at
io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:605)
at
io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:267)
at
org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
at
io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)
at
io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1652)
at
io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1630)
at
io.undertow.server.Connectors.executeRootHandler(Connectors.java:226)
at
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
After taking a heap dump and analyzing, I'm finding that a single thread is
holding all the objects:
The thread *org.xnio.nio.WorkerThread @ 0xc48cac70 XNIO-2 I/O-4* keeps
local variables with total size *578,543,888 (95.77%)* bytes.
The memory is accumulated in one instance of *"java.lang.Object[]"* loaded
by *"<system class loader>"*.
XNIO-2 I/O-4
at
sun.security.ssl.SSLEngineImpl.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult;
(SSLEngineImpl.java:781)
at
io.undertow.protocols.ssl.SslConduit.doUnwrap([Ljava/nio/ByteBuffer;II)J
(SslConduit.java:682)
at io.undertow.protocols.ssl.SslConduit.read(Ljava/nio/ByteBuffer;)I
(SslConduit.java:525)
at
org.xnio.conduits.ConduitStreamSourceChannel.read(Ljava/nio/ByteBuffer;)I
(ConduitStreamSourceChannel.java:127)
at
io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(Lorg/xnio/conduits/ConduitStreamSourceChannel;)V
(HttpReadListener.java:149)
at
io.undertow.server.protocol.http.HttpReadListener.handleEvent(Lorg/xnio/conduits/ConduitStreamSourceChannel;)V
(HttpReadListener.java:127)
at
io.undertow.server.protocol.http.HttpReadListener.handleEvent(Ljava/nio/channels/Channel;)V
(HttpReadListener.java:56)
at
org.xnio.ChannelListeners.invokeChannelListener(Ljava/nio/channels/Channel;Lorg/xnio/ChannelListener;)Z
(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady()V
(ReadReadyHandler.java:66)
at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady()V
(SslConduit.java:1054)
at io.undertow.protocols.ssl.SslConduit$1.run()V (SslConduit.java:225)
at org.xnio.nio.WorkerThread.safeRun(Ljava/lang/Runnable;)V
(WorkerThread.java:580)
at org.xnio.nio.WorkerThread.run()V (WorkerThread.java:464)
Any help would be appreciated. We didn't have any memory leak issues with
Jetty which is what we used prior so I don't think its an issue with our
internal code triggering this.
Thanks
Marc Boorshtein
CTO Tremolo Security
marc.boorshtein(a)tremolosecurity.com
<marc.boorshtein(a)tremolosecurity.com>