[JBoss JIRA] (ISPN-5033) Lack of EmbeddedCacheManager.undefineConfiguration(String) can cause memory/classloader leaks
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5033?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5033:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Lack of EmbeddedCacheManager.undefineConfiguration(String) can cause memory/classloader leaks
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-5033
> URL: https://issues.jboss.org/browse/ISPN-5033
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 7.1.0.Alpha1
> Reporter: Paul Ferraro
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 8.0.0.Alpha2, 8.0.0.Final
>
>
> EmbeddedCacheManager has defineConfiguration(...) methods for defining a new cache configuration, but has no equivalent "undefine" method. Consequently, once defined, a cache manager will retains a reference to the cache configuration object for the lifetime of the cache manager. While this is normally not a big deal, the configuration objects generated by WildFly contain references to the classloaders of foreign modules.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months
[JBoss JIRA] (ISPN-5315) Tight loop on ClientListenerNotifier
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5315?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-5315:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1231215
> Tight loop on ClientListenerNotifier
> ------------------------------------
>
> Key: ISPN-5315
> URL: https://issues.jboss.org/browse/ISPN-5315
> Project: Infinispan
> Issue Type: Bug
> Components: Listeners
> Affects Versions: 7.2.0.Beta1, 7.1.1.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Priority: Minor
> Fix For: 7.2.0.Beta2, 7.2.0.Final
>
> Attachments: stackTrace
>
>
> Sometimes the {{ClientListenerNotifier}} keeps printing extremely high number of warning and quickly fill GBs of disk space:
> {code}
> 11:28:16,805 WARN [ClientListenerNotifier] (Client-Listener-aacb322ffc60405e) ISPN004039: Unable to complete reading event from server null
> java.nio.channels.IllegalBlockingModeException
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
> at org.infinispan.client.hotrod^C.impl.protocol.Codec20.readEvent(Codec20.java:126)
> at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This can be reproduced when running the server test suite, more specifically the test {{ExampleConfigsIT}}. Right after the server is started the above warning is logged in a loop for the duration of the tests.
> I've started to observe this after commit (1), which changed the behaviour to retry on SocketTimeouts.
> Before the aforementioned commit, log were printing:
> {code}
> 10:49:56,361 ERROR [ClientListenerNotifier] (Client-Listener-b53963d5ab2b4234) ISPN004043: Unrecoverable error reading event from server null, exiting event reader thread
> org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:184)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
> at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
> ... 8 more
> {code}
> And thus the listener thread was exiting,
> (1) https://github.com/infinispan/infinispan/commit/1a03a0a84307ae8b95c06b398...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months
[JBoss JIRA] (ISPN-5523) Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5523?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5523:
-----------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/3542
> Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-5523
> URL: https://issues.jboss.org/browse/ISPN-5523
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 8.0.0.Alpha2, 7.2.3.Final, 8.0.0.Final
>
>
> Some near caching tests are throwing:
> {code}
> [0m[31m04:11:24,499 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-43) ISPN005009: Unexpected error before any request parameters read: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_75]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_75]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_75]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:433) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:98) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:241) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:106) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
> {code}
> KeyValueVersionConverter allocates a byte buffer but does not release it. It could be the cause...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months
[JBoss JIRA] (ISPN-5544) Deprecate eager near caching
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5544?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-5544:
----------------------------------------
As part of the deprecation and removal of eager near caching, lazy near caching should be renamed to something more relevant.
> Deprecate eager near caching
> ----------------------------
>
> Key: ISPN-5544
> URL: https://issues.jboss.org/browse/ISPN-5544
> Project: Infinispan
> Issue Type: Task
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 8.0.0.Alpha2, 8.0.0.Final
>
>
> In it's current form, eager near caching can lead the server to run out of direct memory, see ISPN-5523. The reason that's happening is because we're having to send events containing keys + values for each put() operation, indiscriminately to clients. So, regardless of what the client stores in the near cache, the client receives key+value information for all keys, as each key is created/modified, and this is having a big toll on the capabilities of the server to push those events to clients.
> So, we have decided to deprecate eager near caching in Infinispan 8.0, and we'll remove it completely for Infinispan 9.0.
> Lazy near caching still remains and works well because events are smaller since no value is sent to clients. Lazy near caching is still indiscriminate but we have plans to make it more selective, so that for example, only modified/removed events are sent for those keys that are stored in the client near cache, which would reduce traffic considerably (new jira to be created).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months
[JBoss JIRA] (ISPN-5523) Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5523?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-5523:
----------------------------------------
A decision has been made to drop eager near caching, details in ISPN-5544. Even if eager near caching is deprecated, similar issues could potentially affect lazy near caching, hence why we've created ISPN-5545 to reduce traffic. Finally, as part of this JIRA, so improvements were made which should still be pushed to master:
1. org.infinispan.server.hotrod.KeyValueVersionConverterFactory.KeyValueVersionConverter should use a simple array instead of a NIO byte buffer to transform key and value information.
2. Use MessageToByteEncoder instead of MessageToMessageEncoder as Hot Rod encoder to avoid byte buffer leaks if there's an exception in the encoding part.
3. Add manual eager near cache stress test.
4. Decide whether MessageToByteEncoder uses a direct buffer or not based on {{io.netty.noPreferDirect}} system property.
If encountering direct memory OOMEs with Netty, using {{io.netty.noPreferDirect}} might help temporarily get around the issue, but sooner or later, the heap memory would also suffer from the pile up.
> Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-5523
> URL: https://issues.jboss.org/browse/ISPN-5523
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 8.0.0.Alpha2, 7.2.3.Final, 8.0.0.Final
>
>
> Some near caching tests are throwing:
> {code}
> [0m[31m04:11:24,499 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-43) ISPN005009: Unexpected error before any request parameters read: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_75]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_75]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_75]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:433) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:98) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:241) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:106) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
> {code}
> KeyValueVersionConverter allocates a byte buffer but does not release it. It could be the cause...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months
[JBoss JIRA] (ISPN-5545) Make lazy near caching more selective
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5545?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant commented on ISPN-5545:
---------------------------------------
Thinking about using bloom filters for this ?
> Make lazy near caching more selective
> -------------------------------------
>
> Key: ISPN-5545
> URL: https://issues.jboss.org/browse/ISPN-5545
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
>
> In the current form, when lazy near caching is enabled, the server sends invalidation messages for any key that has been created, modified or removed.
> This is suboptimal for a couple of reasons:
> 1. First of all, a near cache might only interested in receiving invalidation events for those keys that are currently stored in the near cache. If the near cache is small subset of the entire cache, having such option would vastly reduce the number of events sent to clients. So, there needs to be a way to be able to narrow the events sent from the server to this subset of keys.
> 2. Lazy near caches do not care about created events. If an entry is present in the near cache, it has already been created, so it's only interested in modified and removed events. There needs to be a way to narrow this down too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months
[JBoss JIRA] (ISPN-5545) Make lazy near caching more selective
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5545?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5545:
-----------------------------------
Description:
In the current form, when lazy near caching is enabled, the server sends invalidation messages for any key that has been created, modified or removed.
This is suboptimal for a couple of reasons:
1. First of all, a near cache might only interested in receiving invalidation events for those keys that are currently stored in the near cache. If the near cache is small subset of the entire cache, having such option would vastly reduce the number of events sent to clients. So, there needs to be a way to be able to narrow the events sent from the server to this subset of keys.
2. Lazy near caches do not care about created events. If an entry is present in the near cache, it has already been created, so it's only interested in modified and removed events. There needs to be a way to narrow this down too.
was:
In the current form, when lazy near caching is enabled, the server sends invalidation messages for any key that has been created, modified or removed.
This is suboptimal for a couple of reasons:
1. First of all, a near cache is only interested in receiving invalidation events for those keys that are currently stored in the near cache. There needs to be a way to be able to narrow the events sent from the server to this subset of keys.
2. Lazy near caches do not care about created events. If an entry is present in the near cache, it has already been created, so it's only interested in modified and removed events. There needs to be a way to narrow this down too.
> Make lazy near caching more selective
> -------------------------------------
>
> Key: ISPN-5545
> URL: https://issues.jboss.org/browse/ISPN-5545
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
>
> In the current form, when lazy near caching is enabled, the server sends invalidation messages for any key that has been created, modified or removed.
> This is suboptimal for a couple of reasons:
> 1. First of all, a near cache might only interested in receiving invalidation events for those keys that are currently stored in the near cache. If the near cache is small subset of the entire cache, having such option would vastly reduce the number of events sent to clients. So, there needs to be a way to be able to narrow the events sent from the server to this subset of keys.
> 2. Lazy near caches do not care about created events. If an entry is present in the near cache, it has already been created, so it's only interested in modified and removed events. There needs to be a way to narrow this down too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years, 5 months