[JBoss JIRA] (WFLY-4696) OutOfMemory DirectByteBuffer XNIO
by Carlos Rodríguez Aguado (JIRA)
[ https://issues.jboss.org/browse/WFLY-4696?page=com.atlassian.jira.plugin.... ]
Carlos Rodríguez Aguado commented on WFLY-4696:
-----------------------------------------------
We have been investigating the source code of class HttpResponseConduit in line 166 where java.nio.BufferOverflowException is thrown.
Just a few lines above an allocate() is executed. If java.no.BufferOverflowException is thrown in line 166 we think that memory allocated could not be free properly because method bufferDone() is never executed. Could it be the reason of the OutOfMemory?
159 if(pooledBuffer == null) {
160 pooledBuffer = {color:red}pool.allocate();{color}
161 }
162 ByteBuffer buffer = pooledBuffer.getResource();
163
164
165 assert buffer.remaining() >= 0x100;
166 {color:red}exchange.getProtocol().appendTo(buffer);{color}
.
.
.
255 bufferDone();
.
.
private void bufferDone() {
HttpServerConnection connection = (HttpServerConnection)exchange.getConnection();
if(connection.getExtraBytes() != null && connection.isOpen() && exchange.isRequestComplete()) {
//if we are pipelining we hold onto the buffer
pooledBuffer.getResource().clear();
} else {
pooledBuffer.free();
pooledBuffer = null;
}
}
> OutOfMemory DirectByteBuffer XNIO
> ---------------------------------
>
> Key: WFLY-4696
> URL: https://issues.jboss.org/browse/WFLY-4696
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 8.1.0.Final, 8.2.0.Final
> Reporter: Carlos Rodríguez Aguado
> Assignee: Stuart Douglas
> Priority: Blocker
>
> I get this errors constantly in my server when a web connection is interrupted from the browser for instance:
> 11:50:45,301 ERROR [stderr] (default task-339) Exception in thread "default task-339" java.nio.BufferOverflowException
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.ByteBuffer.put(ByteBuffer.java:859)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
> 11:50:45,301 ERROR [stderr] (default task-339) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.lang.Thread.run(Thread.java:745)
> And then, I think this errors lead to a OutOfMemory crash:
> 14:23:09,592 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> I also have found that if I perform a full GC manually the server recovers, but it can not recover by itself, by performing other types of GCs.
> This is the trace for version 8.2 of WildFly:
> 17:17:16,957 ERROR [io.undertow.request] (default task-49) Undertow request failed HttpServerExchange{ GET /modulab/servlet/ShowPDFReportServlet}: java.nio.BufferOverflowException
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.put(ByteBuffer.java:859) [rt.jar:1.8.0_20]
> at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166)
> at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:564)
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:106)
> at org.xnio.conduits.Conduits.writeFinalBasic(Conduits.java:132) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.writeFinal(AbstractFixedLengthStreamSinkConduit.java:175)
> at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:194)
> at io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1829)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:565)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:600)
> at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:497)
> at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:581)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:308)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]
> 10:57:12,389 ERROR [org.xnio.listener] (default I/O-4) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:143) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:146) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFLY-4696) OutOfMemory DirectByteBuffer XNIO
by Carlos Rodríguez Aguado (JIRA)
[ https://issues.jboss.org/browse/WFLY-4696?page=com.atlassian.jira.plugin.... ]
Carlos Rodríguez Aguado commented on WFLY-4696:
-----------------------------------------------
I suppose that you are referring to the parameter buffer-size that can be customized in io subsystem:
<subsystem xmlns="urn:jboss:domain:io:1.1">
<worker name="default"/>
<buffer-pool name="default" {color:red}buffer-size="value"{color}/>
</subsystem>
With the default value (we read that is 16384) the problem is reproduced. We tried to decrease the value to 8192 but the effect was exactly the same. Must we increase it to 32768 or 65536?
The protocol is https.
> OutOfMemory DirectByteBuffer XNIO
> ---------------------------------
>
> Key: WFLY-4696
> URL: https://issues.jboss.org/browse/WFLY-4696
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 8.1.0.Final, 8.2.0.Final
> Reporter: Carlos Rodríguez Aguado
> Assignee: Stuart Douglas
> Priority: Blocker
>
> I get this errors constantly in my server when a web connection is interrupted from the browser for instance:
> 11:50:45,301 ERROR [stderr] (default task-339) Exception in thread "default task-339" java.nio.BufferOverflowException
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.ByteBuffer.put(ByteBuffer.java:859)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
> 11:50:45,301 ERROR [stderr] (default task-339) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.lang.Thread.run(Thread.java:745)
> And then, I think this errors lead to a OutOfMemory crash:
> 14:23:09,592 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> I also have found that if I perform a full GC manually the server recovers, but it can not recover by itself, by performing other types of GCs.
> This is the trace for version 8.2 of WildFly:
> 17:17:16,957 ERROR [io.undertow.request] (default task-49) Undertow request failed HttpServerExchange{ GET /modulab/servlet/ShowPDFReportServlet}: java.nio.BufferOverflowException
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.put(ByteBuffer.java:859) [rt.jar:1.8.0_20]
> at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166)
> at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:564)
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:106)
> at org.xnio.conduits.Conduits.writeFinalBasic(Conduits.java:132) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.writeFinal(AbstractFixedLengthStreamSinkConduit.java:175)
> at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:194)
> at io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1829)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:565)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:600)
> at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:497)
> at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:581)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:308)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]
> 10:57:12,389 ERROR [org.xnio.listener] (default I/O-4) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:143) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:146) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (DROOLS-766) RuleNetworkEvaluator infinite loop when doUpdatesReorderLeftMemory is called
by Dmitry Toptygin (JIRA)
[ https://issues.jboss.org/browse/DROOLS-766?page=com.atlassian.jira.plugin... ]
Dmitry Toptygin commented on DROOLS-766:
----------------------------------------
Maybe this will help with finding the root cause:
When TimeFact condition in the rule1 is modified to contain "always true clause", like "id == (123 - 123)" - then the problem does not happen. This is the workaround we're currently using.
Otherwise problem happens when TimeFact conditions in rule1 and rule2 are the same.
Note:
Problem happens after a couple of minutes of executing the reproducer code.
AlarmRecord fact and ClearEvent event are inserted externally into the rule engine working memory while rule engine executes fireUntilHalt in streaming mode.
TimeFact has id of 0, it is updated externally to the rule engine via its FactHandle using kSession.update
Below are rules with the workaround:
===
package rules
import org.example.infiniteloop.ClearEvent
import org.example.infiniteloop.TimeFact
import org.example.infiniteloop.AlarmRecord
declare org.example.infiniteloop.ClearEvent
@role( event )
@timestamp( eventTimestamp )
@expires( 20s )
end
rule "rule1"
when
TimeFact($currentTime : value, id == (123 - 123) ) //remove "id == (123 - 123)" to reproduce the infinite loop problem
ClearEvent()
$clientAlarm: AlarmRecord(createdTimestamp < $currentTime - 15000)
then
//System.out.println("rule 1 fired");
delete($clientAlarm);
end
rule "rule2"
when
TimeFact($currentTime : value)
$clientAlarm: AlarmRecord(createdTimestamp < $currentTime - 15000)
//any other condition will do here
Number(intValue == 42)
then
//not important
end
> RuleNetworkEvaluator infinite loop when doUpdatesReorderLeftMemory is called
> ----------------------------------------------------------------------------
>
> Key: DROOLS-766
> URL: https://issues.jboss.org/browse/DROOLS-766
> Project: Drools
> Issue Type: Bug
> Components: core engine
> Affects Versions: 6.2.0.Final
> Reporter: Juan Carlos Garcia
> Assignee: Mario Fusco
> Attachments: infiniteLoopReproducer.jar, infiniteLoopReproducer_src.jar
>
>
> We are migrating our system to use drools 6.2.0.Final from 6.0.1.Final and one of our testcase which actually simulate several complex scenario is making drools (6.2.0.Final) going into an endless loop, while the same works fine in 6.0.1.Final.
> I took a thread dump and it seems stuck:
> {code}
> at org.drools.core.phreak.RuleNetworkEvaluator.doUpdatesReorderLeftMemory(RuleNetworkEvaluator.java:795)
> {code}
> While debugging the source code this is what i got:
> # LeftTupleMemory ltm = bm.getLeftTupleMemory();
> ** LeftTupleMemory is a LeftTupleList
> # leftTuple.getStagedNext();
> ** Always return the same object instance(leftTuple is a JoinNodeLeftTuple type), hence the loop never ends.
> Unfortunately i cannot provide the DRL files + domain classes involve and don't know if it would be even possible for me right now to recreate the same testcase without leaking internal information from the company i work for.
> ThreadDump
> {code}
> "Attach Listener" daemon prio=10 tid=0x00007fdb74001000 nid=0x88c waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "drools-worker-2" daemon prio=10 tid=0x00007fdb54003000 nid=0x803 waiting on condition [0x00007fdb9b7cc000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000783f213f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "pool-3-thread-1" prio=10 tid=0x00007fdba46de000 nid=0x764 waiting on condition [0x00007fdb9bbd6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0000000783eba160> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "Service Thread" daemon prio=10 tid=0x00007fdba40ad800 nid=0x754 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007fdba40ab000 nid=0x753 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007fdba40a8000 nid=0x752 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "JDWP Command Reader" daemon prio=10 tid=0x00007fdb68001000 nid=0x74e runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "JDWP Event Helper Thread" daemon prio=10 tid=0x00007fdba40a6000 nid=0x74d runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x00007fdba40a2800 nid=0x74c runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007fdba4094800 nid=0x74b runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x00007fdba4074800 nid=0x74a in Object.wait() [0x00007fdba0cfb000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007838824e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> - locked <0x00000007838824e0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007fdba4072800 nid=0x749 in Object.wait() [0x00007fdba0dfc000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000007838820a0> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:503)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
> - locked <0x00000007838820a0> (a java.lang.ref.Reference$Lock)
> "main" prio=10 tid=0x00007fdba400e800 nid=0x743 runnable [0x00007fdbad9a8000]
> java.lang.Thread.State: RUNNABLE
> at org.drools.core.phreak.RuleNetworkEvaluator.doUpdatesReorderLeftMemory(RuleNetworkEvaluator.java:795)
> at org.drools.core.phreak.PhreakJoinNode.doNode(PhreakJoinNode.java:38)
> at org.drools.core.phreak.RuleNetworkEvaluator.switchOnDoBetaNode(RuleNetworkEvaluator.java:547)
> at org.drools.core.phreak.RuleNetworkEvaluator.evalBetaNode(RuleNetworkEvaluator.java:533)
> at org.drools.core.phreak.RuleNetworkEvaluator.innerEval(RuleNetworkEvaluator.java:334)
> at org.drools.core.phreak.RuleNetworkEvaluator.outerEval(RuleNetworkEvaluator.java:161)
> at org.drools.core.phreak.RuleNetworkEvaluator.evaluateNetwork(RuleNetworkEvaluator.java:116)
> at org.drools.core.phreak.RuleExecutor.reEvaluateNetwork(RuleExecutor.java:235)
> - locked <0x0000000783f2da90> (a org.drools.core.phreak.RuleExecutor)
> at org.drools.core.phreak.RuleExecutor.evaluateNetworkAndFire(RuleExecutor.java:106)
> - locked <0x0000000783f2da90> (a org.drools.core.phreak.RuleExecutor)
> at org.drools.core.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1016)
> at org.drools.core.common.DefaultAgenda.fireAllRules(DefaultAgenda.java:1302)
> at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1289)
> at org.drools.core.impl.StatefulKnowledgeSessionImpl.fireAllRules(StatefulKnowledgeSessionImpl.java:1262)
> at org.drools.core.command.runtime.rule.FireAllRulesCommand.execute(FireAllRulesCommand.java:109)
> at org.drools.core.command.runtime.rule.FireAllRulesCommand.execute(FireAllRulesCommand.java:34)
> at org.drools.core.command.runtime.BatchExecutionCommandImpl.execute(BatchExecutionCommandImpl.java:155)
> at org.drools.core.command.runtime.BatchExecutionCommandImpl.execute(BatchExecutionCommandImpl.java:76)
> at org.drools.core.impl.StatefulKnowledgeSessionImpl.execute(StatefulKnowledgeSessionImpl.java:723)
> at org.drools.core.impl.StatefulKnowledgeSessionImpl.execute(StatefulKnowledgeSessionImpl.java:697)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFCORE-66) Provide an operation to request service status report after server has been started
by ehsavoie Hugonnet (JIRA)
[ https://issues.jboss.org/browse/WFCORE-66?page=com.atlassian.jira.plugin.... ]
ehsavoie Hugonnet commented on WFCORE-66:
-----------------------------------------
I don't think so
> Provide an operation to request service status report after server has been started
> -------------------------------------------------------------------------------------
>
> Key: WFCORE-66
> URL: https://issues.jboss.org/browse/WFCORE-66
> Project: WildFly Core
> Issue Type: Feature Request
> Components: Domain Management
> Reporter: Heiko Braun
> Assignee: ehsavoie Hugonnet
> Labels: console_prio
> Fix For: 1.0.0.Alpha6
>
>
> For proper healthcare it would beneficial if I could verify the server status after it has been started. I.e. to catch these, which currently only appear in the server log:
> [Server:server-three] 17:48:26,636 INFO [org.jboss.as.server] (MSC service thread 1-2) Service status report
> [Server:server-three] New missing/unsatisfied dependencies:
> [Server:server-three] service jboss.binding.http2 (missing)
> [Server:server-three]
> [Server:server-three] 17:48:26,636 ERROR [org.jboss.as] (MSC service thread 1-4) JBoss AS 7.0.0.Beta4-SNAPSHOT "(TBD)" started (with errors) in 3893ms - Started 101 of 149 services (1 services failed or missing dependencies, 47 services are passive or on-demand)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFCORE-66) Provide an operation to request service status report after server has been started
by Shaun Appleton (JIRA)
[ https://issues.jboss.org/browse/WFCORE-66?page=com.atlassian.jira.plugin.... ]
Shaun Appleton commented on WFCORE-66:
--------------------------------------
Has a notification mechanism been included?
> Provide an operation to request service status report after server has been started
> -------------------------------------------------------------------------------------
>
> Key: WFCORE-66
> URL: https://issues.jboss.org/browse/WFCORE-66
> Project: WildFly Core
> Issue Type: Feature Request
> Components: Domain Management
> Reporter: Heiko Braun
> Assignee: ehsavoie Hugonnet
> Labels: console_prio
> Fix For: 1.0.0.Alpha6
>
>
> For proper healthcare it would beneficial if I could verify the server status after it has been started. I.e. to catch these, which currently only appear in the server log:
> [Server:server-three] 17:48:26,636 INFO [org.jboss.as.server] (MSC service thread 1-2) Service status report
> [Server:server-three] New missing/unsatisfied dependencies:
> [Server:server-three] service jboss.binding.http2 (missing)
> [Server:server-three]
> [Server:server-three] 17:48:26,636 ERROR [org.jboss.as] (MSC service thread 1-4) JBoss AS 7.0.0.Beta4-SNAPSHOT "(TBD)" started (with errors) in 3893ms - Started 101 of 149 services (1 services failed or missing dependencies, 47 services are passive or on-demand)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFLY-4696) OutOfMemory DirectByteBuffer XNIO
by Stuart Douglas (JIRA)
[ https://issues.jboss.org/browse/WFLY-4696?page=com.atlassian.jira.plugin.... ]
Stuart Douglas commented on WFLY-4696:
--------------------------------------
There was a bug in that version of Undertow where it would potentially allow any protocol to be used. One cause of this could be a misbehaving or malicious client sending a mangled HTTP request, however that seems unlikely.
> OutOfMemory DirectByteBuffer XNIO
> ---------------------------------
>
> Key: WFLY-4696
> URL: https://issues.jboss.org/browse/WFLY-4696
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 8.1.0.Final, 8.2.0.Final
> Reporter: Carlos Rodríguez Aguado
> Assignee: Stuart Douglas
> Priority: Blocker
>
> I get this errors constantly in my server when a web connection is interrupted from the browser for instance:
> 11:50:45,301 ERROR [stderr] (default task-339) Exception in thread "default task-339" java.nio.BufferOverflowException
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.ByteBuffer.put(ByteBuffer.java:859)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
> 11:50:45,301 ERROR [stderr] (default task-339) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.lang.Thread.run(Thread.java:745)
> And then, I think this errors lead to a OutOfMemory crash:
> 14:23:09,592 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> I also have found that if I perform a full GC manually the server recovers, but it can not recover by itself, by performing other types of GCs.
> This is the trace for version 8.2 of WildFly:
> 17:17:16,957 ERROR [io.undertow.request] (default task-49) Undertow request failed HttpServerExchange{ GET /modulab/servlet/ShowPDFReportServlet}: java.nio.BufferOverflowException
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.put(ByteBuffer.java:859) [rt.jar:1.8.0_20]
> at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166)
> at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:564)
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:106)
> at org.xnio.conduits.Conduits.writeFinalBasic(Conduits.java:132) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.writeFinal(AbstractFixedLengthStreamSinkConduit.java:175)
> at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:194)
> at io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1829)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:565)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:600)
> at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:497)
> at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:581)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:308)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]
> 10:57:12,389 ERROR [org.xnio.listener] (default I/O-4) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:143) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:146) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFLY-4696) OutOfMemory DirectByteBuffer XNIO
by Stuart Douglas (JIRA)
[ https://issues.jboss.org/browse/WFLY-4696?page=com.atlassian.jira.plugin.... ]
Stuart Douglas commented on WFLY-4696:
--------------------------------------
Have you configured a custom very small buffer size? And can you check the output of HttpServletRequest#getProtocol() ?
The error occurs when we attempt to the protocol into the response buffer, which basically means:
- Either the buffer has been configured to be to small for Undertow to use
- The protocol is super long for some reason
- Some other bug that results in a full buffer at that point
> OutOfMemory DirectByteBuffer XNIO
> ---------------------------------
>
> Key: WFLY-4696
> URL: https://issues.jboss.org/browse/WFLY-4696
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 8.1.0.Final, 8.2.0.Final
> Reporter: Carlos Rodríguez Aguado
> Assignee: Stuart Douglas
> Priority: Blocker
>
> I get this errors constantly in my server when a web connection is interrupted from the browser for instance:
> 11:50:45,301 ERROR [stderr] (default task-339) Exception in thread "default task-339" java.nio.BufferOverflowException
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363)
> 11:50:45,301 ERROR [stderr] (default task-339) at java.nio.ByteBuffer.put(ByteBuffer.java:859)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:205)
> 11:50:45,301 ERROR [stderr] (default task-339) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)
> 11:50:45,301 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1489)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1470)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)
> 11:50:45,317 ERROR [stderr] (default task-339) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 11:50:45,317 ERROR [stderr] (default task-339) at java.lang.Thread.run(Thread.java:745)
> And then, I think this errors lead to a OutOfMemory crash:
> 14:23:09,592 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]
> I also have found that if I perform a full GC manually the server recovers, but it can not recover by itself, by performing other types of GCs.
> This is the trace for version 8.2 of WildFly:
> 17:17:16,957 ERROR [io.undertow.request] (default task-49) Undertow request failed HttpServerExchange{ GET /modulab/servlet/ShowPDFReportServlet}: java.nio.BufferOverflowException
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:363) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.put(ByteBuffer.java:859) [rt.jar:1.8.0_20]
> at io.undertow.util.HttpString.appendTo(HttpString.java:204)
> at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166)
> at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:564)
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.write(AbstractFixedLengthStreamSinkConduit.java:106)
> at org.xnio.conduits.Conduits.writeFinalBasic(Conduits.java:132) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.writeFinal(AbstractFixedLengthStreamSinkConduit.java:175)
> at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:194)
> at io.undertow.server.HttpServerExchange$WriteDispatchChannel.writeFinal(HttpServerExchange.java:1829)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:565)
> at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:600)
> at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:497)
> at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:581)
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:308)
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247)
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76)
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166)
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197)
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_20]
> 10:57:12,389 ERROR [org.xnio.listener] (default I/O-4) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError
> at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.8.0_20]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127) [rt.jar:1.8.0_20]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_20]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:143) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:146) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
> at org.xnio.nio.WorkerThread.run(WorkerThread.java:539) [xnio-nio-3.3.0.Final.jar:3.3.0.Final]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFLY-4591) Memory leak inPerThreadTagHandlerPool | Wildfly 8.2.0.Final (Undertow)
by Tomaz Cerar (JIRA)
[ https://issues.jboss.org/browse/WFLY-4591?page=com.atlassian.jira.plugin.... ]
Tomaz Cerar commented on WFLY-4591:
-----------------------------------
Jastow 2.0.0.Beta1 was released today with pull request to include it to WildFly pending to be merged https://github.com/wildfly/wildfly/pull/7557
As it goes for missing class org.apache.jasper.runtime.JspSourceImports it is now present in jastow 2.0
> Memory leak inPerThreadTagHandlerPool | Wildfly 8.2.0.Final (Undertow)
> ----------------------------------------------------------------------
>
> Key: WFLY-4591
> URL: https://issues.jboss.org/browse/WFLY-4591
> Project: WildFly
> Issue Type: Bug
> Components: Web (Undertow)
> Affects Versions: 8.2.0.Final
> Environment: CentOS release 6.5 (Final), java version "1.7.0_71" 64 bit VM, Wildfly 8.2.0.Final, Min heap and Max heap 1536m
>
> Reporter: Srivathsan Agaram Venkatavaradhan
> Assignee: Tomaz Cerar
> Labels: garbage, jastow, undertow, wildfly
> Attachments: jastow-2.0.0-SNAPSHOT.jar
>
>
> We upgraded from Jboss 7.1.1.Final to Wildfly 8.2.0.Final. After the upgrade we are seeing full frequent garbage collection (happening every 2 mins) during load test. This was not the case in Jboss 7.1.1.Final. Our heap dumps suggested that there is huge memory leak with
> org.apache.jasper.runtime.BodyContentImpl. This is part of JSP runtime (jastow). On analysis we found that undertow uses default as PerThreadTagHanlderPool. By changing this to TagHanlderPool, this issue seems to vanish.
> diff --git a/src/main/java/org/apache/jasper/runtime/TagHandlerPool.java b/src/main/java/org/apache/jasper/runtime/TagHandlerPool.java
> index eaa8560..c6c785f 100644
> --- a/src/main/java/org/apache/jasper/runtime/TagHandlerPool.java
> +++ b/src/main/java/org/apache/jasper/runtime/TagHandlerPool.java
> @@ -53,7 +53,7 @@ public class TagHandlerPool {
> result = null;
> }
> }
> - if( result==null ) result=new PerThreadTagHandlerPool();
> + if( result==null ) result=new TagHandlerPool();
> result.init(config);
>
> return result;
> - See more at: https://developer.jboss.org/message/927538#927538
> I think this issue may be related to https://issues.jboss.org/browse/WFLY-4617 as there were huge garbage created and most of the contents were char array (rendered html). May be TagHanlderPool is creating more char instead of reusing from pool. Please provide thoughts. I can share, heapdump or MAT screenshots if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month
[JBoss JIRA] (WFCORE-729) Add smoke test for showing the real value of expression in CLI
by Marek Kopecký (JIRA)
[ https://issues.jboss.org/browse/WFCORE-729?page=com.atlassian.jira.plugin... ]
Marek Kopecký moved WFLY-4702 to WFCORE-729:
--------------------------------------------
Project: WildFly Core (was: WildFly)
Key: WFCORE-729 (was: WFLY-4702)
Component/s: Test Suite
(was: Test Suite)
> Add smoke test for showing the real value of expression in CLI
> --------------------------------------------------------------
>
> Key: WFCORE-729
> URL: https://issues.jboss.org/browse/WFCORE-729
> Project: WildFly Core
> Issue Type: Task
> Components: Test Suite
> Reporter: Marek Kopecký
> Assignee: Marek Kopecký
>
> https://issues.jboss.org/browse/EAP7-275 is adding ability to CLI to show the real value of expression properties in addition to the expression
> This test serves as basic smoke tests to be run within the testsuite.
> RealValueExpressionTestCase:
> * creates new system property "real.value.expression.test.case" with value in this format: $\{oldSystemProperty:defaultValue\}
> * test "read-resource" and "read-attribute" operations, these operations should work correctly with resolve-expressions parametr
> * remove newly created system property
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years, 1 month