]
Mike Zhary commented on WFLY-14233:
-----------------------------------
[~flavia.rainone] Well, I can't provide an application for this now due to
commercials.
Yep, I got that there must be some reallocation, but as you can see at first parts of
graphs it didn't happen at all. but apllying the patch makes good changes to us.
I'll ask our programmers to provide some code that can reproduce it. After that
I'll attach it here
Undertow xnio direct memory buffer leaking
------------------------------------------
Key: WFLY-14233
URL:
https://issues.redhat.com/browse/WFLY-14233
Project: WildFly
Issue Type: Bug
Components: IO, Web (Undertow)
Affects Versions: 19.0.0.Final
Reporter: Mike Zhary
Assignee: Flavia Rainone
Priority: Major
Attachments: 2020-11-10-112753_563x276_scrot.png,
api_src_main_java_org_xnio_ByteBufferSlicePool.java, image_2020_10_30T09_55_47_082Z.png,
Снимок экрана 2020-12-16 в 15.33.53.png, Снимок экрана 2020-12-16 в 15.34.20.png
Wildfly.19.0.1.Final standalone
We'd found an issue with direct memory buffer leaking on one from our servers.
There is rest api application allocated there.
it provides hugh data on some requests.
As per graphics we noticed that there is corrielation between sessions on server and
using of direct memory buffer.
!Снимок экрана 2020-12-16 в 15.33.53.png|width=1199,height=400!!Снимок экрана 2020-12-16
в 15.34.20.png|width=1170,height=392!
Our suggestion was that direct memory buffer is reallocatable with new web requests,
that's we tried change session lifetime to minimum we prefer.
You can see it on graphs with memory. Red line means direct memory buffer.
At first time it takes as much as it can, but after changing session lifetime we faced
the same problem.
There are some drops of redline to zero, but there were restarts of wildfly.
As per our investigations there is the problem in that GC free some part of memory, but
it can't get DMB and that's why it increases.
Something allocates ByteBufferSlicePool (but it marked as depricated, so I think
it'll disappared in sometime, but our bug still reproduceable on WF21)
{code:java}
2020-10-15 18:24:59,569 ERROR (default I/O-4) [listener] [SOMEAPP] XNIO001007: A channel
event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at
org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
at
org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
at
org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
at
org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
at
io.undertow.core@2.0.27.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
at
io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
at
io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
at
io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
at
org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at
org.jboss.xnio@3.7.3.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at
org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at
org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
{code}
!2020-11-10-112753_563x276_scrot.png|thumbnail!
When something needs memory it allocates and save this memory to directBuffers(that was
removed)
Seems there is a point of leaking in that some code allocates direct memory when client
doesn't need it
Another point is at method of cleaning of direct memory. When it called directBuffers
saves in static variable of class as null-buffers and it can be reusable for
ByteBufferSlicePool, that is potential leak point, because of this variable never clean
just accumulate.
So, after removing variable directBuffers we'd found a "method".
We'd disable some cache mechanism of allocated memory areas and now clients of this
lib gets DMB directly and when it free GC clean it.
You can see it on graphics. first part of it shows that DMB grows before maxim get OOM
and another part is when there is no problem with OOM.
Some nearest issue WFCORE-4600
that is our patched library:
[^api_src_main_java_org_xnio_ByteBufferSlicePool.java]