[infinispan-issues] [JBoss JIRA] (ISPN-10409) ByteBuf leak in the REST Server

[ https://issues.jboss.org/browse/ISPN-10409?page=com.atlassian.jira.plugin... ] Diego Lovison commented on ISPN-10409: -------------------------------------- {noformat} 20:13:54,032 INFO [org.radargun.RemoteMasterConnection] (sc-main) Message successfully sent to the master [36.029s][info ][gc] GC(6) Pause Young (Concurrent Start) (G1 Evacuation Pause) 813M->105M(16384M) 191.953ms [36.029s][info ][gc] GC(7) Concurrent Cycle [36.177s][info ][gc] GC(7) Pause Remark 129M->129M(16384M) 39.936ms [36.206s][info ][gc] GC(7) Pause Cleanup 130M->130M(16384M) 2.367ms 20:14:01,086 ERROR [io.netty.util.ResourceLeakDetector] (REST-ServerIO-7-3) LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:221) io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:199) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:255) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) org.infinispan.server.rest:ispn-10.0@10.0.0-SNAPSHOT//org.infinispan.rest.cors.CorsHandler.channelRead(CorsHandler.java:75) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) org.infinispan.server:ispn-10.0@10.0.0-SNAPSHOT//org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) 20:14:01,086 ERROR [io.netty.util.ResourceLeakDetector] (REST-ServerIO-7-2) LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331) io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:77) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:784) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) 20:14:01,135 ERROR [io.netty.util.ResourceLeakDetector] (REST-ServerIO-7-6) LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:221) io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:199) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:255) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) org.infinispan.server.rest:ispn-10.0@10.0.0-SNAPSHOT//org.infinispan.rest.cors.CorsHandler.channelRead(CorsHandler.java:75) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) org.infinispan.server:ispn-10.0@10.0.0-SNAPSHOT//org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) [36.376s][info ][gc] GC(7) Concurrent Cycle 346.592ms 20:14:03,643 INFO [org.radargun.service.InfinispanServerClustered] (pool-2-thread-2) Found CacheManager 'jboss.datagrid-infinispan:type=CacheManager,name="clustered",component=CacheManager' MBean. [42.618s][info ][gc] GC(8) Pause Young (Normal) (G1 Evacuation Pause) 817M->207M(16384M) 167.641ms [47.532s][info ][gc] GC(9) Pause Young (Normal) (G1 Evacuation Pause) 919M->307M(16384M) 114.079ms 20:14:13,659 INFO [org.radargun.service.InfinispanServerClustered] (pool-2-thread-2) Found CacheManager 'jboss.datagrid-infinispan:type=CacheManager,name="clustered",component=CacheManager' MBean. [51.530s][info ][gc] GC(10) Pause Young (Normal) (G1 Evacuation Pause) 1019M->413M(16384M) 105.476ms [55.299s][info ][gc] GC(11) Pause Young (Normal) (G1 Evacuation Pause) 1125M->520M(16384M) 107.961ms 20:14:23,638 INFO [org.radargun.service.InfinispanServerClustered] (pool-2-thread-2) Found CacheManager 'jboss.datagrid-infinispan:type=CacheManager,name="clustered",component=CacheManager' MBean. [59.085s][info ][gc] GC(12) Pause Young (Normal) (G1 Evacuation Pause) 1232M->628M(16384M) 102.626ms 20:14:24,031 ERROR [io.netty.util.ResourceLeakDetector] (REST-ServerIO-7-3) LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at: io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:143) io.netty:ispn-10.0@4.1.30.Final//io.netty.buffer.SimpleLeakAwareByteBuf.readRetainedSlice(SimpleLeakAwareByteBuf.java:67) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:305) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.http.HttpServerCodec$HttpServerRequestDecoder.decode(HttpServerCodec.java:101) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) io.netty:ispn-10.0@4.1.30.Final//io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) org.infinispan.server:ispn-10.0@10.0.0-SNAPSHOT//org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421) io.netty:ispn-10.0@4.1.30.Final//io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) io.netty:ispn-10.0@4.1.30.Final//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.base/java.lang.Thread.run(Thread.java:834) [63.131s][info ][gc] GC(13) Pause Young (Normal) (G1 Evacuation Pause) 1340M->740M(16384M) 99.511ms [67.031s][info ][gc] GC(14) Pause Young (Normal) (G1 Evacuation Pause) 1452M->852M(16384M) 102.191ms 20:14:33,641 INFO [org.radargun.service.InfinispanServerClustered] (pool-2-thread-2) Found CacheManager 'jboss.datagrid-infinispan:type=CacheManager,name="clustered",component=CacheManager' MBean. {noformat} -- This message was sent by Atlassian Jira (v7.12.1#712002)