[Red Hat JIRA] (WFLY-14234) EJB bean method direct memory buffer leaking
by Mike Zhary (Jira)
[ https://issues.redhat.com/browse/WFLY-14234?page=com.atlassian.jira.plugi... ]
Mike Zhary updated WFLY-14234:
------------------------------
Description:
Wildfly.19.0.1.Final standalone
We're using haproxy to balance ejb calls between wildfly servers.
That's very useful model, but there are problems with it.
All connections are declared as remote-outbound-connection and it works.
{code}
<remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
</properties>
</remote-outbound-connection>
{code}
But there is the problem when some from servers is offline. haproxy send response by http and send RST by tcp, but WF can't make sense of this response.
Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
But everything is clear, when port closed or server don't response for a calls.
Haproxy doesn't have feature to don't listen port if backed is offline.
At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)
At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
Take a look:
!2020-11-10-112753_563x276_scrot.png|thumbnail!
Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.
We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
As per investigations DMB used for an SSL connection, but we don't it.
So we were to patch jboss-remoting class RemoteConnectionProvider.
after that we'd solved issue with leaking of DMB.
Our patched library: [^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
Please take look how this bug can be fixed.
was:
Wildfly.19.0.1.Final standalone
We're using haproxy to balance ejb calls between wildfly servers.
That's very useful model, but there are problems with it.
All connections are declared as remote-outbound-connection and it works.
{code}
<remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
</properties>
</remote-outbound-connection>
{code}
But there is the problem when some from servers is offline. haproxy send response by http and send RST by tcp, but WF can't make sense of this response it.
Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
But everything is clear, when port closed or server don't response for a calls.
Haproxy doesn't have feature to don't listen port if backed is offline.
At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)
At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
Take a look:
!2020-11-10-112753_563x276_scrot.png|thumbnail!
Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.
We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
As per investigations DMB used for an SSL connection, but we don't it.
So we were to patch jboss-remoting class RemoteConnectionProvider.
after that we'd solved issue with leaking of DMB.
Our patched library: [^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
Please take look how this bug can be fixed.
> EJB bean method direct memory buffer leaking
> --------------------------------------------
>
> Key: WFLY-14234
> URL: https://issues.redhat.com/browse/WFLY-14234
> Project: WildFly
> Issue Type: Bug
> Components: EJB, IO
> Affects Versions: 19.0.0.Final
> Reporter: Mike Zhary
> Assignee: Cheng Fang
> Priority: Major
> Attachments: 2020-11-10-112753_563x276_scrot.png, src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java
>
>
> Wildfly.19.0.1.Final standalone
> We're using haproxy to balance ejb calls between wildfly servers.
> That's very useful model, but there are problems with it.
> All connections are declared as remote-outbound-connection and it works.
> {code}
> <remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
> <properties>
> <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
> <property name="SSL_ENABLED" value="false"/>
> </properties>
> </remote-outbound-connection>
> {code}
> But there is the problem when some from servers is offline. haproxy send response by http and send RST by tcp, but WF can't make sense of this response.
> Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
> But everything is clear, when port closed or server don't response for a calls.
> Haproxy doesn't have feature to don't listen port if backed is offline.
> At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)
> At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
> Take a look:
> !2020-11-10-112753_563x276_scrot.png|thumbnail!
> Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
> The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.
> We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
> As per investigations DMB used for an SSL connection, but we don't it.
> So we were to patch jboss-remoting class RemoteConnectionProvider.
> after that we'd solved issue with leaking of DMB.
> Our patched library: [^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
> Please take look how this bug can be fixed.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 7 months
[Red Hat JIRA] (WFLY-14234) EJB bean method direct memory buffer leaking
by Mike Zhary (Jira)
[ https://issues.redhat.com/browse/WFLY-14234?page=com.atlassian.jira.plugi... ]
Mike Zhary updated WFLY-14234:
------------------------------
Component/s: IO
> EJB bean method direct memory buffer leaking
> --------------------------------------------
>
> Key: WFLY-14234
> URL: https://issues.redhat.com/browse/WFLY-14234
> Project: WildFly
> Issue Type: Bug
> Components: EJB, IO
> Affects Versions: 19.0.0.Final
> Reporter: Mike Zhary
> Assignee: Cheng Fang
> Priority: Major
> Attachments: 2020-11-10-112753_563x276_scrot.png, src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java
>
>
> Wildfly.19.0.1.Final standalone
> We're using haproxy to balance ejb calls between wildfly servers.
> That's very useful model, but there are problems with it.
> All connections are declared as remote-outbound-connection and it works.
> {code}
> <remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
> <properties>
> <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
> <property name="SSL_ENABLED" value="false"/>
> </properties>
> </remote-outbound-connection>
> {code}
> But there is the problem when some from servers is offline. haproxy can't provide data about unavailability and direct memory buffer grows.
> Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
> But everything is clear, when port closed or server don't response for a calls.
> Haproxy doesn't have feature to don't listen port if backed is offline.
> At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)
> At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
> Take a look:
> !2020-11-10-112753_563x276_scrot.png|thumbnail!
> Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
> The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.
> We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
> As per investigations DMB used for an SSL connection, but we don't it.
> So we were to patch jboss-remoting class RemoteConnectionProvider.
> after that we'd solved issue with leaking of DMB.
> Our patched library: [^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
> Please take look how this bug can be fixed.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 7 months
[Red Hat JIRA] (WFLY-14234) EJB bean method direct memory buffer leaking
by Mike Zhary (Jira)
[ https://issues.redhat.com/browse/WFLY-14234?page=com.atlassian.jira.plugi... ]
Mike Zhary updated WFLY-14234:
------------------------------
Description:
Wildfly.19.0.1.Final standalone
We're using haproxy to balance ejb calls between wildfly servers.
That's very useful model, but there are problems with it.
All connections are declared as remote-outbound-connection and it works.
{code}
<remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
</properties>
</remote-outbound-connection>
{code}
But there is the problem when some from servers is offline. haproxy can't provide data about unavailability and direct memory buffer grows.
Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
But everything is clear, when port closed or server don't response for a calls.
Haproxy doesn't have feature to don't listen port if backed is offline.
At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)
At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
Take a look:
!2020-11-10-112753_563x276_scrot.png|thumbnail!
Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.
We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
As per investigations DMB used for an SSL connection, but we don't it.
So we were to patch jboss-remoting class RemoteConnectionProvider.
after that we'd solved issue with leaking of DMB.
Our patched library: [^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
Please take look how this bug can be fixed.
> EJB bean method direct memory buffer leaking
> --------------------------------------------
>
> Key: WFLY-14234
> URL: https://issues.redhat.com/browse/WFLY-14234
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 19.0.0.Final
> Reporter: Mike Zhary
> Assignee: Cheng Fang
> Priority: Major
> Attachments: 2020-11-10-112753_563x276_scrot.png, src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java
>
>
> Wildfly.19.0.1.Final standalone
> We're using haproxy to balance ejb calls between wildfly servers.
> That's very useful model, but there are problems with it.
> All connections are declared as remote-outbound-connection and it works.
> {code}
> <remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
> <properties>
> <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
> <property name="SSL_ENABLED" value="false"/>
> </properties>
> </remote-outbound-connection>
> {code}
> But there is the problem when some from servers is offline. haproxy can't provide data about unavailability and direct memory buffer grows.
> Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
> But everything is clear, when port closed or server don't response for a calls.
> Haproxy doesn't have feature to don't listen port if backed is offline.
> At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)
> At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
> Take a look:
> !2020-11-10-112753_563x276_scrot.png|thumbnail!
> Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
> The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.
> We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
> As per investigations DMB used for an SSL connection, but we don't it.
> So we were to patch jboss-remoting class RemoteConnectionProvider.
> after that we'd solved issue with leaking of DMB.
> Our patched library: [^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
> Please take look how this bug can be fixed.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 7 months
[Red Hat JIRA] (WFLY-14233) Undertow xnio direct memory buffer leaking
by Mike Zhary (Jira)
[ https://issues.redhat.com/browse/WFLY-14233?page=com.atlassian.jira.plugi... ]
Mike Zhary updated WFLY-14233:
------------------------------
Affects Version/s: 19.0.0.Final
> Undertow xnio direct memory buffer leaking
> ------------------------------------------
>
> Key: WFLY-14233
> URL: https://issues.redhat.com/browse/WFLY-14233
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Affects Versions: 19.0.0.Final
> Reporter: Mike Zhary
> Assignee: Flavia Rainone
> Priority: Major
> Attachments: 2020-11-10-112753_563x276_scrot.png, api_src_main_java_org_xnio_ByteBufferSlicePool.java, image_2020_10_30T09_55_47_082Z.png, Снимок экрана 2020-12-16 в 15.33.53.png, Снимок экрана 2020-12-16 в 15.34.20.png
>
>
> Wildfly.19.0.1.Final standalone
> We'd found an issue with direct memory buffer leaking on one from our servers.
> There is rest api application allocated there.
> it provides hugh data on some requests.
> As per graphics we noticed that there is corrielation between sessions on server and using of direct memory buffer.
> !Снимок экрана 2020-12-16 в 15.33.53.png|width=1199,height=400!!Снимок экрана 2020-12-16 в 15.34.20.png|width=1170,height=392!
> Our suggestion was that direct memory buffer is reallocatable with new web requests, that's we tried change session lifetime to minimum we prefer.
> You can see it on graphs with memory. Red line means direct memory buffer.
> At first time it takes as much as it can, but after changing session lifetime we faced the same problem.
> There are some drops of redline to zero, but there were restarts of wildfly.
> As per our investigations there is the problem in that GC free some part of memory, but it can't get DMB and that's why it increases.
> Something allocates ByteBufferSlicePool (but it marked as depricated, so I think it'll disappared in sometime, but our bug still reproduceable on WF21)
> {code:java}
> 2020-10-15 18:24:59,569 ERROR (default I/O-4) [listener] [SOMEAPP] XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
> at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
> at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
> at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
> at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
> at io.undertow.core@2.0.27.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
> at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
> at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.jboss.xnio@3.7.3.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> {code}
> !2020-11-10-112753_563x276_scrot.png|thumbnail!
> When something needs memory it allocates and save this memory to directBuffers(that was removed)
> Seems there is a point of leaking in that some code allocates direct memory when client doesn't need it
> Another point is at method of cleaning of direct memory. When it called directBuffers saves in static variable of class as null-buffers and it can be reusable for ByteBufferSlicePool, that is potential leak point, because of this variable never clean just accumulate.
> So, after removing variable directBuffers we'd found a "method".
> We'd disable some cache mechanism of allocated memory areas and now clients of this lib gets DMB directly and when it free GC clean it.
> You can see it on graphics. first part of it shows that DMB grows before maxim get OOM and another part is when there is no problem with OOM.
> Some nearest issue WFCORE-4600
> that is our patched library:
> [^api_src_main_java_org_xnio_ByteBufferSlicePool.java]
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 7 months
[Red Hat JIRA] (WFLY-14233) Undertow xnio direct memory buffer leaking
by Mike Zhary (Jira)
[ https://issues.redhat.com/browse/WFLY-14233?page=com.atlassian.jira.plugi... ]
Mike Zhary updated WFLY-14233:
------------------------------
Description:
Wildfly.19.0.1.Final standalone
We'd found an issue with direct memory buffer leaking on one from our servers.
There is rest api application allocated there.
it provides hugh data on some requests.
As per graphics we noticed that there is corrielation between sessions on server and using of direct memory buffer.
!Снимок экрана 2020-12-16 в 15.33.53.png|width=1199,height=400!!Снимок экрана 2020-12-16 в 15.34.20.png|width=1170,height=392!
Our suggestion was that direct memory buffer is reallocatable with new web requests, that's we tried change session lifetime to minimum we prefer.
You can see it on graphs with memory. Red line means direct memory buffer.
At first time it takes as much as it can, but after changing session lifetime we faced the same problem.
There are some drops of redline to zero, but there were restarts of wildfly.
As per our investigations there is the problem in that GC free some part of memory, but it can't get DMB and that's why it increases.
Something allocates ByteBufferSlicePool (but it marked as depricated, so I think it'll disappared in sometime, but our bug still reproduceable on WF21)
{code:java}
2020-10-15 18:24:59,569 ERROR (default I/O-4) [listener] [SOMEAPP] XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
at io.undertow.core@2.0.27.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
at org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.jboss.xnio@3.7.3.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
{code}
!2020-11-10-112753_563x276_scrot.png|thumbnail!
When something needs memory it allocates and save this memory to directBuffers(that was removed)
Seems there is a point of leaking in that some code allocates direct memory when client doesn't need it
Another point is at method of cleaning of direct memory. When it called directBuffers saves in static variable of class as null-buffers and it can be reusable for ByteBufferSlicePool, that is potential leak point, because of this variable never clean just accumulate.
So, after removing variable directBuffers we'd found a "method".
We'd disable some cache mechanism of allocated memory areas and now clients of this lib gets DMB directly and when it free GC clean it.
You can see it on graphics. first part of it shows that DMB grows before maxim get OOM and another part is when there is no problem with OOM.
Some nearest issue WFCORE-4600
that is our patched library:
[^api_src_main_java_org_xnio_ByteBufferSlicePool.java]
was:
Wildfly.19.0.1.Final standalone
We'd found an issue with direct memory buffer leaking on one from our servers.
There is rest api application allocated there.
it provides hugh data on some requests.
As per graphics we noticed that there is corrielation between sessions on server and using of direct memory buffer.
!Снимок экрана 2020-12-16 в 15.33.53.png|width=1199,height=400!!Снимок экрана 2020-12-16 в 15.34.20.png|width=1170,height=392!
Our suggestion was that direct memory buffer is reallocatable with new web requests, that's we tried change session lifetime to minimum we prefer.
You can see it on graphs with memory. Red line means direct memory buffer.
At first time it takes as much as it can, but after changing session lifetime we faced the same problem.
There are some drops of redline to zero, but there were restarts of wildfly.
As per our investigations there is the problem in that GC free some part of memory, but it can't get DMB and that's why it increases.
Something allocates ByteBufferSlicePool (but it marked as depricated, so I think it'll disappared in sometime, but our bug still reproduceable on WF21)
{code:java}
2020-10-15 18:24:59,569 ERROR (default I/O-4) [listener] [SOMEAPP] XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
at io.undertow.core@2.0.27.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
at org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.jboss.xnio@3.7.3.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
{code}
!2020-11-10-112753_563x276_scrot.png|thumbnail!
When something needs memory it allocates and save this memory to directBuffers(that was removed)
Seems there is a point of leaking in that some code allocates direct memory when client doesn't need it
Another point is at method of cleaning of direct memory. When it called directBuffers saves in static variable of class as null-buffers and it can be reusable for ByteBufferSlicePool, that is potential leak point, because of this variable never clean just accumulate.
So, after removing variable directBuffers we'd found a "method".
We'd disable some cache mechanism of allocated memory areas and now clients of this lib gets DMB directly and when it free GC clean it.
You can see it on graphics. first part of it shows that DMB grows before maxim get OOM and another part is when there is no problem with OOM.
that is our patched library:
[^api_src_main_java_org_xnio_ByteBufferSlicePool.java]
> Undertow xnio direct memory buffer leaking
> ------------------------------------------
>
> Key: WFLY-14233
> URL: https://issues.redhat.com/browse/WFLY-14233
> Project: WildFly
> Issue Type: Bug
> Components: IO, Web (Undertow)
> Reporter: Mike Zhary
> Assignee: Flavia Rainone
> Priority: Major
> Attachments: 2020-11-10-112753_563x276_scrot.png, api_src_main_java_org_xnio_ByteBufferSlicePool.java, image_2020_10_30T09_55_47_082Z.png, Снимок экрана 2020-12-16 в 15.33.53.png, Снимок экрана 2020-12-16 в 15.34.20.png
>
>
> Wildfly.19.0.1.Final standalone
> We'd found an issue with direct memory buffer leaking on one from our servers.
> There is rest api application allocated there.
> it provides hugh data on some requests.
> As per graphics we noticed that there is corrielation between sessions on server and using of direct memory buffer.
> !Снимок экрана 2020-12-16 в 15.33.53.png|width=1199,height=400!!Снимок экрана 2020-12-16 в 15.34.20.png|width=1170,height=392!
> Our suggestion was that direct memory buffer is reallocatable with new web requests, that's we tried change session lifetime to minimum we prefer.
> You can see it on graphs with memory. Red line means direct memory buffer.
> At first time it takes as much as it can, but after changing session lifetime we faced the same problem.
> There are some drops of redline to zero, but there were restarts of wildfly.
> As per our investigations there is the problem in that GC free some part of memory, but it can't get DMB and that's why it increases.
> Something allocates ByteBufferSlicePool (but it marked as depricated, so I think it'll disappared in sometime, but our bug still reproduceable on WF21)
> {code:java}
> 2020-10-15 18:24:59,569 ERROR (default I/O-4) [listener] [SOMEAPP] XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
> at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
> at org.jboss.xnio@3.7.3.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
> at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
> at org.jboss.xnio@3.7.3.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
> at io.undertow.core@2.0.27.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
> at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
> at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
> at io.undertow.core@2.0.27.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
> at org.jboss.xnio@3.7.3.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
> at org.jboss.xnio@3.7.3.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
> at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
> at org.jboss.xnio.nio@3.7.3.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
> {code}
> !2020-11-10-112753_563x276_scrot.png|thumbnail!
> When something needs memory it allocates and save this memory to directBuffers(that was removed)
> Seems there is a point of leaking in that some code allocates direct memory when client doesn't need it
> Another point is at method of cleaning of direct memory. When it called directBuffers saves in static variable of class as null-buffers and it can be reusable for ByteBufferSlicePool, that is potential leak point, because of this variable never clean just accumulate.
> So, after removing variable directBuffers we'd found a "method".
> We'd disable some cache mechanism of allocated memory areas and now clients of this lib gets DMB directly and when it free GC clean it.
> You can see it on graphics. first part of it shows that DMB grows before maxim get OOM and another part is when there is no problem with OOM.
> Some nearest issue WFCORE-4600
> that is our patched library:
> [^api_src_main_java_org_xnio_ByteBufferSlicePool.java]
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 7 months