Re: [undertow-dev] Memory leak in undertow 1.3.11.Final

From: "Marc Boorshtein" <marc.boorshtein(a)tremolosecurity.com&gt; To: "undertow-dev@lists jboss. org" <undertow-dev(a)lists.jboss.org&gt; Sent: Friday, 15 January, 2016 12:11:35 AM Subject: [undertow-dev] Memory leak in undertow 1.3.11.Final I have undertow embedded into my application and for the most part its working great, but unfortunately I've found a memory leak. The application its self creates an Undertow server using SSL and then adds servlets and filters. There are no custom handlers. Throughout the logs I get the following NPEs: 2016-01-14 09:50:04,681][XNIO-2 task-28] ERROR request - UT005071: Undertow request failed HttpServerExchange{ GET /wp-content/themes/Nexus/style.css request {Accept=[text/css, */*], Connection=[Keep-Alive], Accept-Language=[en-US,en;q=0.5], Accept-Encoding=[gzip, deflate], Cookie=[nowwwsession=eyJpdiI6Ik5hV0xCWWR5VExzUEEzRTBVNUFKUkFcdTAwM2RcdTAwM2QiLCJlbmNyeXB0ZWRSZXF1ZXN0IjoiS2t0TENaVGZFb1NWNjlyVXB4aUxuQ2RHRmlTZzVDWm0veTFXLytXWlBQMEhpMDZKK3FaQWpQYnBGY0RnMW15RCJ9; nvers-prod-session=eyJpdiI6Ikd2Wk5oTDVGbjdSTElGejlQSzFsalFcdTAwM2RcdTAwM2QiLCJlbmNyeXB0ZWRSZXF1ZXN0IjoiN2VldzBiT1lIWkZNaVgwTzV5b2tUSVNnaWwzTC9tRHpmeEdDOFZiQzh2a1RZN29ScElqbHFyZys3dmZMWUNhRSJ9], Referer=[ https://www.nvers.org/the-team/ ], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Trident/7.0; MALC; rv:11.0) like Gecko], Host=[ www.nvers.org ]} response {Connection=[close], Last-Modified=[Thu, 26 Mar 2015 18:45:16 GMT], Server=[Apache/2.2.15 (CentOS)], Content-Type=[text/css], Content-Language=[en-US], Accept-Ranges=[bytes], Date=[Thu, 14 Jan 2016 14:37:10 GMT]}} java.lang.NullPointerException at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:713) at io.undertow.conduits.FinishableStreamSinkConduit.flush(FinishableStreamSinkConduit.java:83) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119) at org.xnio.channels.Channels.flushBlocking(Channels.java:63) at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:609) at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:476) at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:560) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Also, I see the following in System.out: Exception in thread "XNIO-2 task-18" java.lang.NullPointerException at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:605) at io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:267) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1652) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1630) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:226) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) After taking a heap dump and analyzing, I'm finding that a single thread is holding all the objects: The thread org.xnio.nio.WorkerThread @ 0xc48cac70 XNIO-2 I/O-4 keeps local variables with total size 578,543,888 (95.77%) bytes. The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>" . XNIO-2 I/O-4 at sun.security.ssl.SSLEngineImpl.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (SSLEngineImpl.java:781) at io.undertow.protocols.ssl.SslConduit.doUnwrap([Ljava/nio/ByteBuffer;II)J (SslConduit.java:682) at io.undertow.protocols.ssl.SslConduit.read(Ljava/nio/ByteBuffer;)I (SslConduit.java:525) at org.xnio.conduits.ConduitStreamSourceChannel.read(Ljava/nio/ByteBuffer;)I (ConduitStreamSourceChannel.java:127) at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(Lorg/xnio/conduits/ConduitStreamSourceChannel;)V (HttpReadListener.java:149) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(Lorg/xnio/conduits/ConduitStreamSourceChannel;)V (HttpReadListener.java:127) at io.undertow.server.protocol.http.HttpReadListener.handleEvent(Ljava/nio/channels/Channel;)V (HttpReadListener.java:56) at org.xnio.ChannelListeners.invokeChannelListener(Ljava/nio/channels/Channel;Lorg/xnio/ChannelListener;)Z (ChannelListeners.java:92) at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady()V (ReadReadyHandler.java:66) at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady()V (SslConduit.java:1054) at io.undertow.protocols.ssl.SslConduit$1.run()V (SslConduit.java:225) at org.xnio.nio.WorkerThread.safeRun(Ljava/lang/Runnable;)V (WorkerThread.java:580) at org.xnio.nio.WorkerThread.run()V (WorkerThread.java:464) Any help would be appreciated. We didn't have any memory leak issues with Jetty which is what we used prior so I don't think its an issue with our internal code triggering this. Thanks Marc Boorshtein CTO Tremolo Security marc.boorshtein(a)tremolosecurity.com _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev