[JBoss JIRA] (ISPN-1796) Out-of-memory adding a lot of elements in cache with AsyncStore
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-1796?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-1796:
--------------------------------
Issue Type: Task (was: Bug)
> Out-of-memory adding a lot of elements in cache with AsyncStore
> ---------------------------------------------------------------
>
> Key: ISPN-1796
> URL: https://issues.jboss.org/browse/ISPN-1796
> Project: Infinispan
> Issue Type: Task
> Components: Loaders and Stores, Locking and Concurrency
> Affects Versions: 5.1.0.CR3
> Environment: We plan to use Infinispan as a large distributed write-behind cache of terabytes of data, with a little fraction cached in RAM, so OOM is real threat for us.
> Reporter: Andrew Pushkin
> Assignee: Manik Surtani
> Labels: executor, threads
> Original Estimate: 3 days
> Remaining Estimate: 3 days
>
> OOM occure on peaks of putting objects in cache configured to use AsyncStore.
> See Steps to Reproduce.
>
> Profiling shows that the gc path is through AsyncStore.state field.
> The AsyncStore.executor initialized to ThreadPoolExecutor with DiscardPolicy to silently discard tasks if the queue is full, which delays async processing of entries in *state* map, which continues to grow.
> Suggested solution.
> Instead of DiscardPolicy use customized behavior, which is to estimate accumulated state size and (probably comparing it with modificationQueueSize) decide to discard or to block until it is processed.
> The downside of suggested solution is the necessity to lock to estimate state size every time the task is rejected. Possibly it can be alleviated by increasing workingQueue size, so that it survive peaks without rejection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-1876) clustered query is not executed on the right cache
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-1876?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-1876:
-------------------------------------
[~israeldl] hi Israel - any chance for you to take a look at this in the coming weeks? thanks!
> clustered query is not executed on the right cache
> --------------------------------------------------
>
> Key: ISPN-1876
> URL: https://issues.jboss.org/browse/ISPN-1876
> Project: Infinispan
> Issue Type: Bug
> Components: Querying, RPC
> Affects Versions: 5.1.1.FINAL
> Reporter: Mathieu Lachance
> Assignee: Israel Lacerra
>
> when using clustered query, command is not executed on the right cache.
> when debuging, in CommandAwareRpcDispatcher::executeCommand()
> cmd = ClusteredQuery{cache="the-good-cache-name"}
> cmd.cache = "the-bad-cache-name"
> cmd.cacheName = "the-good-cache-name"
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-2789) OutOfMemoryError in case if LuceneCacheLoader.loadAll() is called
by Anna Manukyan (JIRA)
[ https://issues.jboss.org/browse/ISPN-2789?page=com.atlassian.jira.plugin.... ]
Anna Manukyan commented on ISPN-2789:
-------------------------------------
[~sannegrinovero] The thing was that the loadAll() method was calling load(Integer.MAX_VALUE) method passing Integer Max value. The load method was instantiating HashSet for storing the loaded values in it, and the thing was that the HashSet was instantiating by passing the initialCapacity equals to the passed Integer.MAX_VALUE.
The OutOfMemory issue was appearing while instantiating HashSet with the passed initialCapacity value.
So I thought that there is no point in passing these value as an initial capacity to the HashSet, especially if the app works with low number of entities and uses loadAll() method, then a huge amount of memory is allocated but during app lifetime it is not used.
If this fix is not acceptable, then we need to think of another solution, even though the VM parameters extension to high numbers didn't help.
> OutOfMemoryError in case if LuceneCacheLoader.loadAll() is called
> -----------------------------------------------------------------
>
> Key: ISPN-2789
> URL: https://issues.jboss.org/browse/ISPN-2789
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 5.2.0.Final
> Reporter: Anna Manukyan
> Assignee: Anna Manukyan
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> In case when calling LuceneCacheLoader.loadAll() method, the following exception appears:
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.<init>(HashMap.java:204)
> at java.util.HashMap.<init>(HashMap.java:216)
> at java.util.HashSet.<init>(HashSet.java:142)
> at org.infinispan.lucene.cachestore.DirectoryLoaderAdaptor.loadAllEntries(DirectoryLoaderAdaptor.java:87)
> at org.infinispan.lucene.cachestore.LuceneCacheLoader.load(LuceneCacheLoader.java:116)
> at org.infinispan.lucene.cachestore.LuceneCacheLoader.loadAll(LuceneCacheLoader.java:105)
> at org.infinispan.lucene.cacheloader.LuceneCacheLoaderTest.testLoadEntries(LuceneCacheLoaderTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {code}
> I tried to edit VM options for make it to work, but even in case of high numbers for memory usage, this still happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-1855) Accessing a non-distributed cache from a RemoteCacheManager can break topology updates
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-1855?page=com.atlassian.jira.plugin.... ]
Mircea Markus commented on ISPN-1855:
-------------------------------------
Replicated caches are to be implemented as (degenerated) distributed caches (ISPN-2772) as well, so the scope of the bug might narrow post ISPN-2772
> Accessing a non-distributed cache from a RemoteCacheManager can break topology updates
> --------------------------------------------------------------------------------------
>
> Key: ISPN-1855
> URL: https://issues.jboss.org/browse/ISPN-1855
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.1.1.FINAL
> Reporter: Dan Berindei
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Final
>
>
> RemoteCacheManager uses a single consistent hash to map requests to different servers, but caches on the server may have different CHs (or even no CH if the cache is not in distributed mode).
> If the first request goes to a on-distributed cache, the client will never request an updated CH and so it will use a round robin strategy for routing request to all the caches. Obviously this is not optimal for distributed caches.
> Each distributed cache can also have different members since 5.1, so it would be best if we kept a separate CH per cache on the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-2789) OutOfMemoryError in case if LuceneCacheLoader.loadAll() is called
by Anna Manukyan (JIRA)
[ https://issues.jboss.org/browse/ISPN-2789?page=com.atlassian.jira.plugin.... ]
Anna Manukyan edited comment on ISPN-2789 at 2/5/13 11:03 AM:
--------------------------------------------------------------
[~sannegrinovero] The thing was that the loadAll() method was calling load(Integer.MAX_VALUE) method passing Integer Max value. The load method was instantiating HashSet for storing the loaded values in it, and the thing was that the HashSet was instantiated by passing the initialCapacity equals to the passed Integer.MAX_VALUE.
The OutOfMemory issue was appearing while instantiating HashSet with the passed initialCapacity value.
So I thought that there is no point in passing these value as an initial capacity to the HashSet, especially if the app works with low number of entities and uses loadAll() method, then a huge amount of memory is allocated but during app lifetime it is not used.
If this fix is not acceptable, then we need to think of another solution, even though the VM parameters extension to high numbers didn't help.
was (Author: amanukyan):
[~sannegrinovero] The thing was that the loadAll() method was calling load(Integer.MAX_VALUE) method passing Integer Max value. The load method was instantiating HashSet for storing the loaded values in it, and the thing was that the HashSet was instantiating by passing the initialCapacity equals to the passed Integer.MAX_VALUE.
The OutOfMemory issue was appearing while instantiating HashSet with the passed initialCapacity value.
So I thought that there is no point in passing these value as an initial capacity to the HashSet, especially if the app works with low number of entities and uses loadAll() method, then a huge amount of memory is allocated but during app lifetime it is not used.
If this fix is not acceptable, then we need to think of another solution, even though the VM parameters extension to high numbers didn't help.
> OutOfMemoryError in case if LuceneCacheLoader.loadAll() is called
> -----------------------------------------------------------------
>
> Key: ISPN-2789
> URL: https://issues.jboss.org/browse/ISPN-2789
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 5.2.0.Final
> Reporter: Anna Manukyan
> Assignee: Anna Manukyan
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> In case when calling LuceneCacheLoader.loadAll() method, the following exception appears:
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.<init>(HashMap.java:204)
> at java.util.HashMap.<init>(HashMap.java:216)
> at java.util.HashSet.<init>(HashSet.java:142)
> at org.infinispan.lucene.cachestore.DirectoryLoaderAdaptor.loadAllEntries(DirectoryLoaderAdaptor.java:87)
> at org.infinispan.lucene.cachestore.LuceneCacheLoader.load(LuceneCacheLoader.java:116)
> at org.infinispan.lucene.cachestore.LuceneCacheLoader.loadAll(LuceneCacheLoader.java:105)
> at org.infinispan.lucene.cacheloader.LuceneCacheLoaderTest.testLoadEntries(LuceneCacheLoaderTest.java:146)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {code}
> I tried to edit VM options for make it to work, but even in case of high numbers for memory usage, this still happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-2786) ThreadLocal memory leak in Tomcat
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2786?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2786:
--------------------------------
Fix Version/s: 5.2.1
5.3.0.Final
> ThreadLocal memory leak in Tomcat
> ---------------------------------
>
> Key: ISPN-2786
> URL: https://issues.jboss.org/browse/ISPN-2786
> Project: Infinispan
> Issue Type: Bug
> Components: Marshalling, Transactions
> Affects Versions: 5.1.8.Final
> Reporter: Johann Burkard
> Assignee: Galder Zamarreño
> Labels: leak, local, memory, thread, threadlocal
> Fix For: 5.2.1, 5.3.0.Final
>
>
> Just started an app using Infinispan 5.1.8.Final on Tomcat and got a few ThreadLocal problems during un-deployment:
> (Shortened)
> {code}
> key=org.jboss.marshalling.UTFUtils.BytesHolder
> value=org.jboss.marshalling.UTFUtils$BytesHolder@697a1686
> key=java.lang.ThreadLocal@36ed5ba6
> value=org.infinispan.context.SingleKeyNonTxInvocationContext{flags=null}
> key=org.infinispan.marshall.jboss.AbstractJBossMarshaller$1
> value=org.infinispan.marshall.jboss.AbstractJBossMarshaller$1@75f10df7
> value=org.infinispan.marshall.jboss.AbstractJBossMarshaller.PerThreadInstanceHolder
> {code}
> I do call {{DefaultCacheManager#shutdown()}} during un-deployment. :)
> Thanks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-1896) ClusteredGetCommands should never fail with a SuspectException
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-1896?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-1896:
--------------------------------
Fix Version/s: 5.2.1
5.3.0.Final
(was: 6.0.0.Final)
> ClusteredGetCommands should never fail with a SuspectException
> --------------------------------------------------------------
>
> Key: ISPN-1896
> URL: https://issues.jboss.org/browse/ISPN-1896
> Project: Infinispan
> Issue Type: Bug
> Components: RPC
> Affects Versions: 5.1.6.FINAL
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 5.2.1, 5.3.0.Final
>
>
> I have seen this exception in the core test suite logs:
> {noformat}
> 2012-03-02 15:07:19,718 ERROR (testng-VersionedDistStateTransferTest) [org.infinispan.test.fwk.UnitTestTestNGListener] Method testStateTransfer(org.infinispan.container.versioning.VersionedDistStateTransferTest) threw an exceptionorg.infinispan.CacheException: SuspectedException
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:524)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:168)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:478)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:148)
> at org.infinispan.distribution.DistributionManagerImpl.retrieveFromRemoteSource(DistributionManagerImpl.java:169)
> at org.infinispan.interceptors.DistributionInterceptor.realRemoteGet(DistributionInterceptor.java:212)
> at org.infinispan.interceptors.DistributionInterceptor.remoteGetAndStoreInL1(DistributionInterceptor.java:194)
> at org.infinispan.interceptors.DistributionInterceptor.remoteGetBeforeWrite(DistributionInterceptor.java:440)
> at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:455)
> at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:274)
> ...
> Caused by: SuspectedException
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:349)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:263)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:163)
> ... 60 more
> {noformat}
> The remote get command should return null instead of failing, even if it had a single target node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months
[JBoss JIRA] (ISPN-2259) Async store is too slow
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2259?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2259:
--------------------------------
Issue Type: Enhancement (was: Bug)
> Async store is too slow
> -----------------------
>
> Key: ISPN-2259
> URL: https://issues.jboss.org/browse/ISPN-2259
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Affects Versions: 5.1.6.FINAL
> Environment: Windows XP, Java 7
> Reporter: Radim Kolář
> Assignee: Galder Zamarreño
> Labels: performance
> Fix For: 6.0.0.Final
>
> Attachments: profiler.png, testcase.zip
>
>
> Async support for loaders is too slow. I am writing 1000 items into cache backed by jdbm with modification queue size 10000 and it takes 6 seconds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
13 years, 2 months