[infinispan-issues] [JBoss JIRA] (ISPN-5930) OOM error when registering continuous query
Adrian Nistor (JIRA)
issues at jboss.org
Tue Dec 1 11:48:00 EST 2015
[ https://issues.jboss.org/browse/ISPN-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135740#comment-13135740 ]
Adrian Nistor edited comment on ISPN-5930 at 12/1/15 11:47 AM:
---------------------------------------------------------------
This OOM happens right when the listener is registered and it starts to receive initial state. This is after the tests runs for a while (minutes) to warm up so I expect there are quite a large number of entries in the cache. All of them are going to be sent to the listener one by one and for each event we get a memory allocation that is going to be released after the event is consumed. So if the client listener is slow at processing these events it will lead to OOM in the server quite easily. To solve this we need to implement a mechanism to cause some backpressure and stop/delay marshalling more events if the client cannot keep up.
was (Author: anistor):
This OOM happens right when the listener is registered and it starts to receive initial state. This is after the tests runs for a while (minutes) to warm up so I expect there are quite a large number of entries in the cache. All of them are going to be sent to the listener one by one and for each event we get a memory allocation that is going to be released after the event is consumed. So if the client listener is slow at processing these events it will lead to OOM in the server quite easily. To solve this we need to implement a mechanism to cause some backpressure and stop marshalling more events if the client cannot keep up.
> OOM error when registering continuous query
> -------------------------------------------
>
> Key: ISPN-5930
> URL: https://issues.jboss.org/browse/ISPN-5930
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 8.1.0.CR1, 8.0.2.Final
> Reporter: Vojtech Juranek
> Assignee: Adrian Nistor
>
> When running CQ perf tests in client-server mode, I hit following exception in HR server.
> The scenario was to store random texts in the cache and CQ which matches all the entries (i.e. query was "%").
> Full logs from server are [here|https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/PERF-CS/job/jdg-perf-cs-continuous-query-simple/25/]. Machines perf01-04 are servers, perf05-07 clients and perf08 is RG master.
> {noformat}
> [0m[31m12:36:37,557 ERROR [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-9-33) ISPN005023: Exception encoding message CustomRawEvent{version=23, messageId=39574, op=CacheEntryCreatedEventResponse, listenerId=([B at 9641785,false), event=([B at 5ecfed02,false), isRetried=false}: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.capacity(UnpooledUnsafeDirectByteBuf.java:157)
> at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:817)
> at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:825)
> at org.infinispan.server.core.transport.ExtendedByteBuf$.writeRangedBytes(ExtendedByteBuf.scala:67)
> at org.infinispan.server.hotrod.Encoder2x$.writeEvent(Encoder2x.scala:47)
> at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:57)
> at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
> at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
> at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:691)
> at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:681)
> at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:716)
> at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:954)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:243)
> at org.infinispan.server.hotrod.ClientListenerRegistry$BaseClientEventSender.sendEvent(ClientListenerRegistry.scala:219)
> at org.infinispan.server.hotrod.ClientListenerRegistry$BaseClientEventSender.onCacheEvent(ClientListenerRegistry.scala:186)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
> at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:21)
> at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1213)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.raiseEventForInitialTransfer(CacheNotifierImpl.java:911)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.addListener(CacheNotifierImpl.java:852)
> at org.infinispan.notifications.cachelistener.CacheNotifierImpl.addListener(CacheNotifierImpl.java:923)
> at org.infinispan.cache.impl.CacheImpl.addListener(CacheImpl.java:722)
> at org.infinispan.cache.impl.AbstractDelegatingCache.addListener(AbstractDelegatingCache.java:347)
> at org.infinispan.server.hotrod.ClientListenerRegistry.addClientListener(ClientListenerRegistry.scala:100)
> at org.infinispan.server.hotrod.Decoder2x$.customReadKey(Decoder2x.scala:384)
> at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:194)
> at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeKey(HotRodDecoder.scala:104)
> at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:48)
> at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:206)
> at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:45)
> at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:370)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)
> at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:31)
> at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32)
> at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:31)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the infinispan-issues
mailing list