<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'>
I did exactly that last night. I built a test that allocated HashEntry<K,V>[] just like it does in the Segment constructor. I don't have the numbers in front of me right now, but at the maximum size (2^30) and without specifying concurrency it was trying to allocate 32 Segment objects each having a HashEntry<K,V>[] with 2^25 elements. Each one of these HashEntry<K,V>[] takes up around 128MB of memory.<BR> <BR>
> From: galder@redhat.com<BR>> Date: Thu, 20 Jan 2011 09:38:43 +0100<BR>> To: infinispan-dev@lists.jboss.org<BR>> Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions<BR>> <BR>> <BR>> On Jan 19, 2011, at 1:47 PM, david marion wrote:<BR>> <BR>> > <BR>> > It was my problem. If -1 is set for maxEntries, then I am creating a BoundedConcurrentHashMap with a size of Integer.MAX_VALUE. Then when I ran the tests all of them ran out of memory. I should have looked at the stack trace a little more closely. I am going to have to put logic in so that the maximum capacity is determined by the amount of free memory when the cache is created. Previously, if -1 was set for maxEntries, a ConcurrentHashMap was created with maximum capacity of Integer.MAX_VALUE. I have not looked at the code, but my guess is that it works because it grows over time and does not pre-allocate all structures ahead of time.<BR>> <BR>> Hmmm, that sounds odd. Maybe you wanna build a test to show the exact differences and see if too much is being allocated on startup?<BR>> <BR>> > Any thoughts on logic for determining maximum capacity at creation time? <BR>> <BR>> A percentage of the available free memory? 40% by default? You have to accomodate for the other apps running on the same JVM, so we shouldn't be too aggressive by default.<BR>> <BR>> > <BR>> > > From: galder@redhat.com<BR>> > > Date: Wed, 19 Jan 2011 10:32:42 +0100<BR>> > > To: infinispan-dev@lists.jboss.org<BR>> > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions<BR>> > > <BR>> > > Out of curiosity, what is a code problem on your side? Or a set up issue?<BR>> > > <BR>> > > On Jan 19, 2011, at 4:29 AM, david marion wrote:<BR>> > > <BR>> > > > Disregard, I found the issue. Sorry for the spam.<BR>> > > > <BR>> > > > > From: dlmarion@hotmail.com<BR>> > > > > To: infinispan-dev@lists.jboss.org<BR>> > > > > Date: Tue, 18 Jan 2011 20:39:21 -0500<BR>> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions<BR>> > > > > <BR>> > > > > Hah, might be because the POM is configured to run tests in parallel with a<BR>> > > > > max heap size of 1GB. Can someone update<BR>> > > > > http://community.jboss.org/wiki/ParallelTestSuite or<BR>> > > > > http://community.jboss.org/wiki/InfinispanQuickStartGuideforDevelopers#Testi<BR>> > > > > ng with instructions on how to run test serially (I didnt find anything wrt<BR>> > > > > Maven surefire plugin)? Thanks<BR>> > > > > <BR>> > > > > -----Original Message-----<BR>> > > > > From: infinispan-dev-bounces@lists.jboss.org<BR>> > > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Galder<BR>> > > > > Zamarreņo<BR>> > > > > Sent: Tuesday, January 18, 2011 6:12 AM<BR>> > > > > To: infinispan -Dev List<BR>> > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions<BR>> > > > > <BR>> > > > > Hmmm, try increasing -Xmx? Start with -XX:+HeapDumpOnOutOfMemoryError<BR>> > > > > -XX:HeapDumpPath=/tmp/java_heap and inspect the dump with Eclipse MAT?...<BR>> > > > > <BR>> > > > > On Jan 17, 2011, at 4:51 PM, david marion wrote:<BR>> > > > > <BR>> > > > > > same result.<BR>> > > > > > <BR>> > > > > > Date: Mon, 17 Jan 2011 11:54:15 -0300<BR>> > > > > > From: vblagoje@redhat.com<BR>> > > > > > To: infinispan-dev@lists.jboss.org<BR>> > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions<BR>> > > > > > <BR>> > > > > > Its easier to run everything from command line for two reason I can think<BR>> > > > > of:<BR>> > > > > > - everyone has same env to compare results with you<BR>> > > > > > - it is all ready to go<BR>> > > > > > <BR>> > > > > > See http://community.jboss.org/wiki/InfinispanandMaven<BR>> > > > > > <BR>> > > > > > On 11-01-17 11:38 AM, Dave wrote:<BR>> > > > > > All,<BR>> > > > > > <BR>> > > > > > I am trying to test some changes that I made using Eclipse and the TestNG<BR>> > > > > plugin. I am not familiar with TestNG. Are there any settings that I should<BR>> > > > > make? Any ideas?<BR>> > > > > > <BR>> > > > > > java.lang.OutOfMemoryError: Java heap space<BR>> > > > > > at<BR>> > > > > org.infinispan.util.concurrent.BoundedConcurrentHashMap$HashEntry.newArray(B<BR>> > > > > oundedConcurrentHashMap.java:295)<BR>> > > > > > at<BR>> > > > > org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.<init>(Bound<BR>> > > > > edConcurrentHashMap.java:898)<BR>> > > > > > at<BR>> > > > > org.infinispan.util.concurrent.BoundedConcurrentHashMap.<init>(BoundedConcur<BR>> > > > > rentHashMap.java:1367)<BR>> > > > > > at<BR>> > > > > org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja<BR>> > > > > va:78)<BR>> > > > > > at<BR>> > > > > org.infinispan.container.DefaultDataContainer.<init>(DefaultDataContainer.ja<BR>> > > > > va:47)<BR>> > > > > > at<BR>> > > > > org.infinispan.container.DefaultDataContainer.unBoundedDataContainer(Default<BR>> > > > > DataContainer.java:93)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.DataContainerFactory.construct(DataContainerFactory<BR>> > > > > .java:53)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst<BR>> > > > > ractComponentRegistry.java:315)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs<BR>> > > > > tractComponentRegistry.java:251)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc<BR>> > > > > ies(AbstractComponentRegistry.java:840)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac<BR>> > > > > tComponentRegistry.java:225)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist<BR>> > > > > ry.java:120)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac<BR>> > > > > tComponentRegistry.java:192)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst<BR>> > > > > ractComponentRegistry.java:323)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs<BR>> > > > > tractComponentRegistry.java:251)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc<BR>> > > > > ies(AbstractComponentRegistry.java:840)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac<BR>> > > > > tComponentRegistry.java:225)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist<BR>> > > > > ry.java:120)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac<BR>> > > > > tComponentRegistry.java:192)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(Abst<BR>> > > > > ractComponentRegistry.java:323)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(Abs<BR>> > > > > tractComponentRegistry.java:251)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry$Component.injectDependenc<BR>> > > > > ies(AbstractComponentRegistry.java:840)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac<BR>> > > > > tComponentRegistry.java:225)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.ComponentRegistry.registerComponent(ComponentRegist<BR>> > > > > ry.java:120)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.AbstractComponentRegistry.registerComponent(Abstrac<BR>> > > > > tComponentRegistry.java:192)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory<BR>> > > > > .java:92)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFac<BR>> > > > > tory.java:78)<BR>> > > > > > at<BR>> > > > > org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFacto<BR>> > > > > ry.java:62)<BR>> > > > > > at<BR>> > > > > org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.j<BR>> > > > > ava:510)<BR>> > > > > > at<BR>> > > > > org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java<BR>> > > > > :440)<BR>> > > > > > at<BR>> > > > > org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java<BR>> > > > > :409)<BR>> > > > > > at<BR>> > > > > org.infinispan.config.ConfigurationValidationTest.testDefaultMemoryGuardConf<BR>> > > > > iguration(ConfigurationValidationTest.java:100)<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > <BR>> > > > > > -----Original Message-----<BR>> > > > > > From: infinispan-dev-bounces@lists.jboss.org<BR>> > > > > [mailto:infinispan-dev-bounces@lists.jboss.org] On Behalf Of Vladimir<BR>> > > > > Blagojevic<BR>> > > > > > Sent: Friday, January 14, 2011 11:05 AM<BR>> > > > > > To: infinispan -Dev List<BR>> > > > > > Subject: Re: [infinispan-dev] ISPN-863 - Thoughts / Questions<BR>> > > > > > <BR>> > > > > > Hey David,<BR>> > > > > > <BR>> > > > > > First of all kudos for taking on such a non-trivial task!<BR>> > > > > > <BR>> > > > > > On 11-01-14 12:22 AM, david marion wrote:<BR>> > > > > > 1. Modify configuration in some way so that the following can be<BR>> > > > > specified:<BR>> > > > > > a. The percentage value of used JVM memory (i.e. 95) at which<BR>> > > > > entries should be evicted to try and avoid an OOM error.<BR>> > > > > > b. The number of items that should be evicted when memory reaches<BR>> > > > > this threshold<BR>> > > > > > 2. Modify LRU and LIRS Eviction class so that the accessQueue member<BR>> > > > > can be accessed by the new Eviction class so that two access queues dont<BR>> > > > > have to be maintained.<BR>> > > > > > 3. Create a new Eviction class, a subclass of LIRS, where the<BR>> > > > > accessQueue is used from the Eviction strategy the user specifies and the<BR>> > > > > for loop in the execute method is exited when the evicted set equals value<BR>> > > > > from 1.b above.<BR>> > > > > > 4. Modify DataContainerFactory.construct() to call<BR>> > > > > DefaultDataContainer.boundedDataContainer() regardless of eviction policy.<BR>> > > > > This will always create a BoundedConcurrentHashMap<BR>> > > > > > <BR>> > > > > > I don't think you have to extend LRU and LIRS, as long as you implement<BR>> > > > > EvictionPolicy you are fullfilling the contract. What extending LRU and LIRS<BR>> > > > > can do is give you some order in selecting proper elements for eviction. I<BR>> > > > > think the precise technical term in research literature is "eviction<BR>> > > > > precision". <BR>> > > > > > <BR>> > > > > > <BR>> > > > > > 1. <BR>> > > > > > 2. Create an instance of the new Eviction class in each segment.<BR>> > > > > > 3. Modify BoundedConcurrentHashMap.Segment put and replace methods<BR>> > > > > such that when new values are going to be put into the Segment, the memory<BR>> > > > > usage is checked and the execute method is called on the new Eviction class.<BR>> > > > > > <BR>> > > > > > Sounds right!<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > 1. <BR>> > > > > > <BR>> > > > > > Questions:<BR>> > > > > > <BR>> > > > > > 1. What are the implications of using a BoundedConcurrentHashMap<BR>> > > > > instead of a ConcurrentHashMap when maxEntries is set to -1?<BR>> > > > > > <BR>> > > > > > I think none except you turn on eviction by using<BR>> > > > > BoundedConcurrentHashMap.<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > 1. <BR>> > > > > > <BR>> > > > > > Thoughts<BR>> > > > > > <BR>> > > > > > 1. This will not guarantee that an OOM error does not occur. It will<BR>> > > > > attempt to guard against an OOM caused by putting new values into the cache.<BR>> > > > > This will probably be more effective when the cache is being used in<BR>> > > > > client/server mode, and less effective when used in embedded mode as to<BR>> > > > > other code running in the JVM.<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > -- Dave Marion<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > Cheers,<BR>> > > > > > Vladimir<BR>> > > > > > <BR>> > > > > > _______________________________________________<BR>> > > > > > infinispan-dev mailing list<BR>> > > > > > <BR>> > > > > > infinispan-dev@lists.jboss.org<BR>> > > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> > > > > > <BR>> > > > > > <BR>> > > > > > _______________________________________________ infinispan-dev mailing<BR>> > > > > list infinispan-dev@lists.jboss.org<BR>> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev_____________________<BR>> > > > > __________________________<BR>> > > > > > infinispan-dev mailing list<BR>> > > > > > infinispan-dev@lists.jboss.org<BR>> > > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> > > > > <BR>> > > > > --<BR>> > > > > Galder Zamarreņo<BR>> > > > > Sr. Software Engineer<BR>> > > > > Infinispan, JBoss Cache<BR>> > > > > <BR>> > > > > <BR>> > > > > _______________________________________________<BR>> > > > > infinispan-dev mailing list<BR>> > > > > infinispan-dev@lists.jboss.org<BR>> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> > > > > <BR>> > > > > <BR>> > > > > _______________________________________________<BR>> > > > > infinispan-dev mailing list<BR>> > > > > infinispan-dev@lists.jboss.org<BR>> > > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> > > > _______________________________________________<BR>> > > > infinispan-dev mailing list<BR>> > > > infinispan-dev@lists.jboss.org<BR>> > > > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> > > <BR>> > > --<BR>> > > Galder Zamarreņo<BR>> > > Sr. Software Engineer<BR>> > > Infinispan, JBoss Cache<BR>> > > <BR>> > > <BR>> > > _______________________________________________<BR>> > > infinispan-dev mailing list<BR>> > > infinispan-dev@lists.jboss.org<BR>> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> > _______________________________________________<BR>> > infinispan-dev mailing list<BR>> > infinispan-dev@lists.jboss.org<BR>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR>> <BR>> --<BR>> Galder Zamarreņo<BR>> Sr. Software Engineer<BR>> Infinispan, JBoss Cache<BR>> <BR>> <BR>> _______________________________________________<BR>> infinispan-dev mailing list<BR>> infinispan-dev@lists.jboss.org<BR>> https://lists.jboss.org/mailman/listinfo/infinispan-dev<BR> </body>
</html>