[infinispan-issues] [JBoss JIRA] (ISPN-5523) Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"

[Galder Zamarreño (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078077#comment-13078077 ] 

Galder Zamarreño commented on ISPN-5523:
----------------------------------------

A decision has been made to drop eager near caching, details in ISPN-5544. Even if eager near caching is deprecated, similar issues could potentially affect lazy near caching, hence why we've created ISPN-5545 to reduce traffic. Finally, as part of this JIRA, so improvements were made which should still be pushed to master:

1. org.infinispan.server.hotrod.KeyValueVersionConverterFactory.KeyValueVersionConverter should use a simple array instead of a NIO byte buffer to transform key and value information.
2. Use MessageToByteEncoder instead of MessageToMessageEncoder as Hot Rod encoder to avoid byte buffer leaks if there's an exception in the encoding part.
3. Add manual eager near cache stress test.
4. Decide whether MessageToByteEncoder uses a direct buffer or not based on {{io.netty.noPreferDirect}} system property.

If encountering direct memory OOMEs with Netty, using {{io.netty.noPreferDirect}} might help temporarily get around the issue, but sooner or later, the heap memory would also suffer from the pile up.

> Enabling eager caching can lead  to sever throwing "OutOfMemoryError: Direct buffer memory"
> -------------------------------------------------------------------------------------------
>
>                 Key: ISPN-5523
>                 URL: https://issues.jboss.org/browse/ISPN-5523
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Remote Protocols
>    Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
>            Reporter: Galder Zamarreño
>            Assignee: Galder Zamarreño
>             Fix For: 8.0.0.Alpha2, 7.2.3.Final, 8.0.0.Final
>
>
> Some near caching tests are throwing:
> {code}
> 04:11:24,499 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-43) ISPN005009: Unexpected error before any request parameters read: java.lang.OutOfMemoryError: Direct buffer memory
> 	at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_75]
> 	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_75]
> 	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_75]
> 	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:433) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.PoolArena.allocate(PoolArena.java:168) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.PoolArena.allocate(PoolArena.java:98) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:241) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:106) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> 	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
> {code}
> KeyValueVersionConverter allocates a byte buffer but does not release it. It could be the cause...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)